Note: This is the second post of a several part series that I will be writing about MLS’ playoff structure and how it penalizes successful teams. Part one is located here.
In my first post in this series, I explained how MLS’s current playoff structure and generous standards to determine which teams make the playoffs has lead to a number of eighth seeds with losing records that have made it deep into the playoffs (including appearing in two of the last three MLS Cups). In this post, I will explain how the disparity in the number of games played throughout the regular season contributes to the predictable outcome of a first round playoff match, and why it seems to uniquely penalize the higher seeded teams.
Factors In Predicting MLS First Round Playoff Outcomes
Climbing the Ladder had a great post right after the conclusion of the first round of the 2010 playoffs were different factors that had contributed to first round playoff success since 2003 were compared to each other. A snapshot of the table that was developed is shown below.
What was interesting is that fewer games, out of all the factors studied, stands out as the most significant determinant of first round playoff success. This seems a bit unexpected to those versed in how the US game operates, and I will explain why in a bit. The next most significant factor – regular season goal difference – does make a good bit more sense. The less significant coach field is much like the fewer games category - the results are a bit opposed to what our first guess might suggest. As the comments in the original post evolved, it became clear that everyone wanted to focus on the “fewer games played” category. Here’s why.
While MLS plays a balanced schedule in their regular season, their teams have three other types of matches they can play.
- Friendlies: The number of friendlies a team plays is under the direct control of the team management, and varies from team-to-team. Teams may use them as pre-season conditioning or as mid-season revenue generators. I’ve seen both approaches from my hometown team, the Seattle Sounders. Regardless, they demand match preparation and player conditioning, even though the starting XI may get changed from the side that participates in a typical MLS match. While teams somewhat enjoy them, a growing number of fans aren’t. Seattle’s supporters have been so vocal as to create a Twitter hashtag to emphasize their displeasure, which was subsequently picked up by Real Salt Lake. Perhaps the supporters are on to something – if you’re going to lower the chances of the team making it deep into the MLS Cup Playoffs, don’t do it with a meaningless friendly.
- CONCACAF Champions League (CCL): Like every other continental governing body, CONCACAF has a tournament where the best teams from each league play each other to determine which club is the best from that governing body. Winners of the CCL move on to the Club World Cup. While the format for the CCL has changed over the years, it has followed a 4 team-per-group, 6 game balanced-schedule-within-group format since 2008. While the standards for MLS CCL qualification has also changed over time, currently the MLS Cup champion and Supporter’s Shield winner are guaranteed a spot in group play, while the MLS Cup runner up and US Open Cup champion must play a two-legged play-in via the preliminary round. This means MLS CCL participants can play an additional two to eight matches during their MLS regular season, depending on qualification path. As the CCL follows the FIFA calendar, the six games a team may play in the Championship Round of the tournament occur at the beginning of the next MLS season. Thus, if a team qualifies for the Championship Round in one CCL tournament and qualifies for the Preliminary/Group Play stage in another, they may play up to fourteen additional matches in a season.
- US Open Cup: The longest running soccer cup competition in the United States has been contested since 1914. Thirty-two teams from the lower divisions of the American soccer pyramid begin the tournament in June, with eight MLS teams being added to the third round to compete against the eight remaining teams from the lower levels of the pyramid. MLS teams must play four matches in the tournament to win the US Open Cup. MLS assigns its berths in the tournament by giving its top six teams (by points) automatic berths and the remaining teams compete for a berth via a playoff format. Thus, MLS teams can play between four and eight matches in qualification and participation in the US Open Cup.
All of this makes for a pretty compelling case that game differential may play a significant roll in the first round. This seems intuitive – teams that make it deeper into competitions outside of MLS will have played more matches, and fatigue at the end of the long season may take a bigger toll on those teams. Combine fatigue with the wild nature of a playoff, and you have a recipe for the team who has played fewer games to win. This seems a bit unfair, as the only way these teams have played more games is that they finished well the season prior (qualifying for the CCL) and/or have played well in the US Open Cup (in trying to qualify for next season’s CCL). The US soccer landscape already has two knockout style championships, so why does MLS feel the need to add the unpredictability of a third?
Testing the Data Statistically: How Playing Fewer Games Leads To Upsets
All of those observations are just idle speculation if they can’t be backed up by statistics showing the difference in games played actually upending expected results. Interestingly, the statistics present a mixed answer that suggests MLS may just be proceeding down the path of unintended upsets in how their playoff structure interacts with the teams’ other competitions.
The first comparison that was studied was Climbing the Ladder's observation regarding the overall impact o fewer games. In this case, it must be determined if the sample size is large enough (enough playoff games played since 2003) to determine if the observed effect is statistically significant. This entails comparing the expected percentage for overall first round playoff matches (50% chance of winning the series) to the observed percentage for teams that win and have played fewer games throughout the season (67.2%). As my comment in the original Climbing the Ladder post suggests:
Sadly, even the "fewer games" category is not statistically significant when compared to the 50/50 chance most teams have of advancing. Using this site and entering the following values:
p0 = 0.5
p1 = 0.672
1 sided Test
Default alpha and power levels
Yields a sample size of 51. This means this proportion of win percentage would have to be viewed over 51 matches to determine that fewer games was a good predictor vs. a coin flip.
This means that the 32 opening round series played in the last eight seasons and the resultant 67.2% win percentage for teams that played fewer games during the regular season are not significant enough to declare an advantage for such teams. Statistically, such a percentage is technically no better than flipping a coin to determine who moves on to the next round.However, interesting conclusions can be drawn when the focus turns from the overall "fewer/greater number of games" metric and instead look at the impact of the difference in the number of games played. One way to measure this is to look at each series where there was a difference in the total number of games played by each team, and note if the team with fewer games won the series or not. Teams that had fewer games and won are binned as a "correct prediction", while teams that lost while playing fewer games during the regular season are binned as "incorrect predictions". If each bin of predictions is then sorted by ascending goal difference and a (1-CDF) plot is generated (a concept introduced here), the effect of increasing goal difference in predicting the outcome of the first round playoff series is understood. A graph representing this relationship is shown below (click to enlarge).
What becomes immediately clear is that by the time the game differential gets to four, the team with fewer games has nearly a six in seven chance of winning the series. No team who has played six or more games during the regular season has won a first round playoff series. Clearly, three matches or fewer appear to be a statistical toss-up, but a match difference of four or greater greatly favors the team that played those precious fewer matches.
Recall that the MLS Suppoters' Shield and MLS Cup winners receive automatic berths into the CCL. The odds seen in the (1-CDF) plot also play out when we look at these two types of teams that also tend to play a much greater number of games.
The table below summarizes the performance of each Supporter's Shield Winner once they reached the playoffs. Five of the teams played no more or less than two games when compared to their opponents, and this resulted in mixed results for the team that played fewer games. The three series where a team had played four or more games less than their opponent saw that team win the series. The latest occurrence of this is especially illustrative. The 2010 playoffs saw the Los Angeles Galaxy defeat the Seattle Sounders after the Sounders had played two more games than LA in the US Open Cup (in fact, Seattle eliminated LA in that competition) and Seattle played six more games in a poor CCL group play campaign (due to LA failing to win their preliminary round series).
What's even more interesting is studying the performance of MLS Cup winners in their first round playoff matches when defending their title (see table below). Save for LA in 2005, each MLS Cup winner since 2002 has qualified for the playoffs the season after they won the championship. Nearly every one of them lost in the first round, with the Houston Dynamo being the only exception when they successfully defended their championship in 2007. Ironically, Houston would follow up that 2007 campaign with a 2008 campaign that bridged the prior and current CCL format, which meant they set a record for first round game differential (14) that will likely never be equaled. They subsequently lost to New York in the 2008 playoffs.
Ultimately the table confirms what the graph showed. The two first round series that had a game differential of less than 4 were a toss up - one win and one loss. The remaining series all involved game differentials of four or more, and all five ended in favor of the team with fewer games played. With the latest format for the CCL, where MLS Cup winners automatically qualify for six matches of group play rather than a home-away elimination format as in years past, we've likely seen the end of any dynastic teams like those in the first decade of MLS.
The predictability of outcome of first round series with a game differential of four or more does not stop there. Of greater concern is the alignment of such predictability with the likelihood of an upset. Each of the thirty-two first round matches was categorized as an upset if the lower seeded team won (as indicated by conference seeds one through four, regardless of whether or not a team was moved from one conference to the other under MLS's odd playoff seeding rules). The table below shows the results of each first round playoff series, categorized by whether or not an upset occurred and binned according to the difference in the number of games played.
Clearly, as the game differential increases the likelihood of an upset increases. If a comparison is made between the ratio of upsets in the series with a game differential of four or greater versus the rest of the series, one finds that there is insufficient sample size to determine that the difference is statistically significant. A further breakdown of the matches with a game differential of four or more also yields the table below.
What is interesting is that matches with a game differential of four or more have picked up significantly since the change in the CCL group play format in 2008. In fact, using this calculator to make a comparison of the proportions of the table above shows that a statistically significant difference in the two eras exists if one more series were played in each era and the proportions held. The proportion of matches with a game differential of four or more has certainly increased since 2008.
The proportion of series that ends in an upset has also gone up with era, but not by a statistically significant amount.
In general, it appears as if there is a greater number of first round series with a game differential of four or more games since 2008. Recognizing the challenge such a game differential presents to a team is key in setting realistic expectations for a season. Several key conclusions can be drawn.
- If a team wins the Supporter’s Shield and/or the MLS Cup one season, they are unlikely to make it out of the first round of the next year’s playoffs. Sorry LA Galaxy and Colorado Rapids supporters.
- Fans have to be realistic when it comes to setting expectations for success in MLS and other competitions. My fellow Sounders Supporters have the desire to pull off the domestic double and possibly the CCL treble. This is completely unrealistic in MLS – supporters need to pick which competition they want their team to win and not expect the impossible.
- Sounders supporters, who desire a US Open Cup threepeat, should think twice if they want an MLS Cup. Such a win would put the team back into the CCL and likely cause them cross the crucial four match threshold.
Also likely is a good bit of correlation with overall parity for which MLS is notorious. As the Sounder at Heart blog suggests, they only measure how much parity there is and not whether or not parity is good. Interestingly enough, the English Premier League scores at the opposite end of the parity index. This aligns with less numerical expectations, and illustrates the conundrum I have. I want a meritocratic league where teams are allowed to reward players with the best contracts possible, and I want players to be rewarded financially for their skills. In the same vein, I don’t want to turn the league into such a financial boon that Russian oligarchs or oil sheiks decide to purchase championships for their their clubs with funds from outside their soccer operations and thus drown out other clubs from competing for available talent. In the case of MLS, the seemingly random nature of the league and the increasingly predictable success of the lower seeds in the playoffs suggests that the system needs to be corrected.
The exact nature of that correction can take many forms, but I will attempt to make some suggestions in my next post in this series.