Lessons from "What the Geeks Don't Get: The Limits of Moneyball"

In less than two weeks over two thousand sports geeks, statisticians, econometricians, journalists, and staff from professional sports teams will converge on Boston for the 2012 Sloan Sports Analytics Conference. What started as a small gathering of a few hundred people five years ago has now blossomed into what many consider the premier sports analytics conference. With the ever increasing sophistication of sports analytics models and the popularity of the field of study, it's easy to get wrapped up in the idea that the games we watch and play are being revolutionized by a bunch of motivated number jockeys. The reality is that change comes slowly, especially when trying to change a business model in highly successful leagues built upon an old boys (or girls) club of practitioners-turned-managers.

The ability to change sports in an evolutionary manner comes when one puts themselves in the shoes of the skeptics whose minds they're trying to sway. Practitioners of sports analytics should ask themselves, "where could I/we be wrong?" "What are the limits of my/our statistical approaches to the game?" "How sensitive are my/our models to variation or unpredicted inputs?" One must ask themselves these questions, because any inquisitor worth their salt will ask those questions themselves. The analyst should use such introspection to be better prepared when the inevitable questions are asked.  It's often not the fact that someone has one potential theory for success in sports that makes it powerful. It's that it is better at explaining sports success than other competing theories, especially when probed for breakdown or failure.

Last year's conference tried to take on the broader question of the limits of sports analytics in a panel entitled, "What the Geeks Don't Get: The Limits of Moneyball." The panel was moderated by Michael Lewis, who wrote the book for which the panel was named. Bill Simmons from ESPN provided some broader commentary, Bill Polian (former president of theIndianapolis Colts) and Jonathan Kraft (President of the Kraft Group, owners of the New England Patriots) provided commentary from the NFL, and Mark Cuban (owner of the Dallas Mavericks) and Daryl Morey (General Manager of the Houston Rockets) gave views from the NBA. While none of the participants was from the soccer community, the broad discussion of many of the topics over the hour and twenty minutes led to several lessons that can apply within soccer.

The panel got of to a very direct start with Lewis asking each of the panelists to list the one thing they dislike the most about the Moneyball movement. Bill Simmons provided the oft-heard critique of analysts difficulty in making their complex analyses understandable to the wider public and senior team management. From Simmons' perspective, the impact of analytics in the wider sports consuming public will be limited until more analysts can master the skill of simplified communication. Bill Polian came back to this theme when asked a question about how much the management of the game had changed with computers.  He's not a computer guy, and needs information to be communicated in an actionable manner.

I must admit that I often fall into the trap that Simmons mentions. A recent conversation with a fellow blogger could have been paraphrased as, "Dude, I am reasonably intelligent when it comes to stats and I can't follow what the heck m-pound-whatever means. Simplify it, and you will get more people to understand it." And likely more clicks was the insinuation. It is easy to forget that the numerically inclined mind that builds such complex models is not how the rest of society is wired. At the end of the day, the reader or club executive digesting the information is simply looking how to "buy wins" to paraphrase Peter Brand. They aren't looking for the most detailed description of the model. In fact, they're probably just looking for enough of description to understand if it makes sense and then make decisions based upon it if it does. This is especially true in the more statically-skeptical sport of soccer.

One way to check if a presentation method to a non-technical audience may be successful is to use the 10 second rule.  The concept is very simple: a statistic and how it's calculated must be communicated to a non-technical audience in ten seconds or less.  This rule comes from ESPN, who must tailor their infographics to a wide range of viewers.  Those tuning into SportsCenter may have a wide variety of sports backgrounds with depth in only one of the five or six sports being covered in the show.  The infographics and sports-specific terms are toned down for such a viewing audience.  Sports-specific shows like Baseball Tonight get the more dense graphics and detailed terms like VORP and WAR.  Tailoring the message to the knowledge level of the audience given the brief time one has with them is absolutely critical.

While the decision making in a sports club has no where near the potential downside of an investment decision on Wall Street that involves billions of dollars, there was a parallel to Wall Street drawn by the panel. The collapse of the financial industry in 2008 was largely blamed on the sophisticated models that formed the underpinnings of the credit market that were highly unstable when confronted with challenges like sub-prime mortgage defaults. The panelists actually debated whether or not the Wall Street CEO's knew of the financial system's susceptibility - the insinuation by Michael Lewis is that they knew. Whether they knew or not is not the point. The point is that leaders and analysts having blind faith in their models, rather than questioning their limitations, can lead to far worse failures that not having the models in the first place.

Daryl Morey outlined his method for preventing such massive failures. He desires to create a culture of "no bad ideas" inside the Rockets organization, which encourages open and honest debate about player and team evaluation. Taking such an approach allows him to question and challenge every theory, and then see how the person being questioned reacts. It's not a challenge to the person's theory, but rather a test of whether or not they've thought of the failure modes associated with it. Responding to Morey's questions with an intelligent answer inspires confidence in the robustness of the model and encourages further questions. Slinking away or simply saying, "Yeah, you're right..." lets Morey know the analyst doesn't actually believe enough in their model or theory to defend it. So why should Morey?

In general, the NBA representatives like Morey were far more likely to embrace analytics in the public sphere than the NFL representatives. Jonathan Kraft made a puzzling statement about the definitive nature of the limitations of public analytics in football, especially in comparison to other sports (yet in the next breath he admits to knowing nothing about them). Polian goes as far to state his disagreement with Kraft, but then then makes declarations that public models on management tactics are "worthless". In general, both men feel the NFL's tactics change from play to play and the fact that they're completely dependent upon the exact match ups across the line of scrimmage make the applicability of any general theory on game management very low. Similar to football, the fluidity and specificity of events in the game of soccer also make useful, publicly-available models few and far between.

Given the difficulties of in-game analytics, the NFL representatives did agree with the NBA participants that analytics that apply to player psychology will pay dividends in the coming years. Polian and Kraft outlined how they're currently scouting player's psychologically by going all the way back to their junior high coaches for information. Cuban talks of continuing the evaluation throughout the player's career at the club, as only the stress of the regular season and playoffs can bring out the player's true behavior in the team environment. As Polian explains, a player at 22 is "single and has more money than he knows what to do with, while a player at 28 is likely married, maybe has a kid, and is staring at the back end of his career." How they react to the stress of the game will be very different. Cuban concludes that profiles in the NBA can be used to play the game of "Protect the Moron" - the guy who doesn't have the game intelligence to be on your team, but who the team wants to drop unsuspectingly on another team.

Soccer's focus on youth academies and long term investment in players emphasizes just such psychological profiling.  With the market for soccer players being truly global and transfer fees for good players being in the tens of millions of dollars, the ability to develop a great youth system is a way to contain player costs.  The most recent example of advanced psychological profiling and mental development came in Issue 3 of The Blizzard via John Sinnott's profile of Standard Liege's Michel Bruyninckx and his efforts to improve the development of the club's young players. Standard Liege's approach is one they call "overload", which emphasizes the development of fast, multiple input decision making that is required on the soccer pitch.  The skills the players learn are not only useful on the pitch, but also in other areas of life to the benefit of the majority of players who never make a living in the professional game.  Such skills are extremely difficult to capture in an analytical model, especially when it's built upon limited public data.

What all of the participants in the panel emphasized was that statistics are meaningless without a culture to go with them.  This was a point also made during the soccer-specific panel at the 2011 conference.  Without the consistency of a culture that flows from the top leadership down to the player lowest on the depth chart, the use of statistical analysis is subject to the relative success or failure of the latest manager.  Bill Polian gave the example of the Boston Red Sox in baseball, whose principal owner is analytical by nature, the front office is staffed by marketing and legal executives who buy into an analytical approach, and whose GM until the end of the 2011 season had perhaps the deepest analytical outlook of any of the Red Sox leadership team.  The key, according to Polian, is to meld everything they do into a fan friendly product that succeeds on the playing field.  So when one sees in-game statistics that indicate success on the field that should be replicated by other teams, one of the key missing ingredients in the model is the rigorous cultural reinforcement of the approach being taken by the successful club.  Thus, Barcelona's tiki-taka approach may be universally loved and often imitated, but without the Barcelona culture it will never be as successful.  The results can be far worse if the system is imposed on a club with a culture diametrically opposed to Barcelona's.

This panel not only provided a few reminders as to the limitations of an analytics approach to understanding sports, but it confirmed why I am looking forward to a few of the panels that may be light on analytics at the 2012 conference.  "The Business of Sports: Winning off the Field" will provide a full range of business executives, including Manchester United CEO David Gill, exploring topics like media rights, stadium development, and sponsorships.  Such recipes for business success are critical in a debt-constrained UEFA operating under Financial Fair Play rules that include a break-even-or-better requirement.  Like it or not, soccer clubs must operate more like businesses and less like recreational distractions.  "In the Interest of the Game: The Evolution of Sport Leagues" will bring together commissioners and league executives from various sports to talk about how leagues attempt to manage growth and success for all teams involved.  Finally, I am looking forward to the closing event - a live BS Report with Bill Simmons interviewing Mark Cuban.  Cuban will be his lively, opinionated best and was the executive who most embraced analytics on the 2011 panel.  It should make for a wonderful interview.

Tickets for the 2012 conference are now sold out.  If you're in the Boston area that weekend, you can sign up at the conference website to be wait listed in case any late tickets are released.  Otherwise, I hope to bring you some good coverage of what looks to be a fun 2012 conference.