Thread by @mominmbutt

Predictive Analytics and Metrics

Notes of first half of Chapter 4 from Sports Analytics: A Guide for Managers, Coaches, and Other Decision Makers

Book by @BenCAlamar

#sportsanalytics #SportsDataBookClub

Perhaps the most important attribute for a decision maker in aiding the development of an analytics program is the ability and willingness to ask questions.

Their ability to provide clear and usable analysis is greatly enhanced when decision makers ask
questions not about the analysis but about the decisions that they have to make.

Analysts bring a set of skills but rarely will the analyst understand the sport as deeply as the top decision makers. Some questions, those that usually prove to be the most interesting, never get asked because the decision maker does not believe that the answer can be quantified

In the area of player evaluation, these are often referred to as the player’s intangibles and come in a variety of forms in a nonquantitative scouting report:

• Makes his teammates better
• Great leader
• Hustles on every play
• Coachable

Comments such as these are often viewed as squarely in the domain of unquantifiable player attributes, so questions about measuring these attributes and how they affect a player and his team’s performance go
largely unasked.

If decision makers instead begin to ask the questions and probe on the meaning and effect of these attributes, the analyst can often devise methods to measure what was previously unmeasured—not
immeasurable.

One example of this is the effect of teammates on one another. Some teams seem to play above what the sum of their parts suggest, and this ability not only to play well but to play well together is often referred to as team chemistry.

The theory goes that some teams have good chemistry and thus teammates raise one another’s games, and others do not and so underperform. The concept of team chemistry is regularly discussed as an important but immeasurable trait.

Dean Oliver, analyst for the Seattle Supersonics,
Denver Nuggets, and ESPN) started to ask sports executives and coaches what they meant when they referred to “chemistry” in an effort to measure it.

Several themes emerged, so Oliver approached this question with the idea that athletes have specific skill sets and that some skill sets fit together better than others.

Simply by starting to ask the questions and building basic models around how teammates might actually make one another better, he was able to develop an approach to quantifying how well teammates fit.

Oliver’s work on teammate fit was not a comprehensive answer to the question of team chemistry, but it is a starting point that helps measure and explain an important concept in sports that was previously unmeasured.

In order to fully embrace asking questions, it is important for decision makers to have a clear idea of what it means to measure or quantify something. Putting a number on a skill, for example, often denotes a level of precision that is simply false.

The goal of quantifying something, hitting ability in baseball, for example, is not to know beyond a shadow of a doubt exactly how good a hitter a particular player is, but rather to reduce the uncertainty around the decision maker’s evaluation of the player’s hitting ability.

The evolution of batting statistics is a good example of the idea that we are not measuring anything exactly but rather are using the information we have to get as clear a picture as possible about a player’s abilities.

Another example of the value of quantifying skills and attributes is the analysis that is done around amateur drafts. Start by considering the extreme case in which no information is known about any potential draft
pick.

In this scenario, the decision maker can do no better
than to randomly select a player and hope for the best. Here we have minimized measurement and maximized risk. As we start to add information such as scouting reports, we reduce the risk in the decision-making process.

Scouting reports are not exact and are not always correct, but they provide information that reduces the risk in making a selection on draft day. Now we add the ability to interview players before the draft. The
interview process adds more information about the player.

Finally, we add the ability to analyze the quantitative information from the player’s amateur performances. Here the statistical analysis of the player’s skills and
how those skills project to the professional level provides an additional piece of information.

The analysis is not an exact measurement of how well
the athlete performs in different aspects, nor does it provide an exact projection of how the athlete will perform at the professional level. It
does, however, provide more information and reduces the risk of a draft pick

Risk v Information trade-off

ANALYTICS AND HIRING A COACH

NFL teams do not generally use much quantitative analysis in the hiring of a head coach.

The argument against the use of quantitative analysis has been that since we can point to examples of successful and unsuccessful coaches from a variety of different backgrounds, there are too many intangibles involved in what leads to head-coaching success in the NFL.

This is an instance of a narrow idea of what quantitative analysis can provide. Clearly, there are successful head coaches from a variety of backgrounds.

Hiring a head coach has proven to be a risky process, and, just as with the draft, asking questions and adding new information to the process can help reduce the inherent risk. There are few decisions that have more impact on an NFL franchise than the selection of the head coach.

One NFL franchise went through the process of hiring a head coach and made what turned out to be a poor decision. The team performed well below expectations, and ownership felt it had to move on to a different head coach.

Instead of using the same process that led to the previous choice, the top decision makers at the team started to ask questions. They asked what elements of a candidate’s background are most likely to produce a successful head coach.

Once the decision makers started to ask these questions, the decision maker and the analysts could discuss what elements might be important:

Years coaching in the NFL, previous head-coaching experience at any level, previous NFL playing experience, Super Bowl wins as a coordinator, winning percentage as a college coach, etc. A long list emerged of potential pieces of the head-coaching puzzle.

The analyst was then able take that list of potential elements and assemble the relevant data on potential head coaches in the NFL over the previous twenty seasons.

Before the analysis could move forward, however, the decision makers had to define and establish what it was to be a successful head coach. This required the decision makers to set the bar.

This questioning process allowed the decision makers to firmly establish in their minds what they were trying to find in a head coach and allowed the analyst to understand clearly what it was they were trying to measure.

The decision makers accepted from the beginning that there was going to be risk in the decision and that the quantitative analysis could help them reduce but not eliminate that risk. They used the analysis to give themselves the highest probability of success.

The result of the analysis was a grading scale that gave a score to each element of a candidate’s background that was found to have a significant effect on success. This allowed the decision makers to be more fully informed about the risk they were assuming with each candidate.

The decision makers could see that hiring a coach who scored poorly meant accepting more risk, and so they would need to have a clear rationale as to why this particular candidate would succeed despite twenty years of data suggesting he is unlikely to.

Even with this analysis there is no guarantee that the coach the team hired would be successful. The success of the analysis was not dependent on the outcome of the hire but on the process the team went through.