In what has become a tradition for most SBNation hockey blogs following their teams withdrawal from the NHL season, FTF has started individual player season reviews for the 2011-2012 campaign. In an obvious swindle to woo your anticipation we’ve started with the bottom guys, the role players, the unsung heros, the guys that bring their lunch pails to the rink, the guys that play with stick/smash/heart/chip on their shoulder. There are about a billion ways these players have been described by journalist throughout mainstream media. They capture a fan’s vision of these players as invaluable to a team’s success. For without Jody Shelly, the Flyerswould have been run over and intimidated this year, and could not have conceivably won as many games as they did. An objective, albeit less romantic synonym, would be replacement.
In many ways this simple word is what baseball analyses has over many other sports analyse. Due to the frequency of games and length of season, Keith Woolner in 2001 then writing for Baseball Prospectus, posted a simple article detailing a theoretical idea that would inevitably find it’s way into the front office of every major league team. In it’s most simplest form Woolner explains replacement level as;
The talent distribution in baseball can be summed up as follows: there are very few “superstar” level players, a somewhat larger number of “average” producers, and a practically unlimited number of “scrubs”. This is usually represented as the tail end of a bell curve or normal distribution, with the vast majority of the overall population already weeded out through other factors prior to reaching professional ball.
Average players have value. Measures that use average production as a reference point, such as TPR, incorrectly estimate the contribution of average players by failing to recognize the value of average playing time.
Replacement level is a less concrete mathematical concept, but it is an important economic one. In particular, it more correctly values durability and playing time versus rates of production.
Essentially replacement level is the value of a player that is freely available to any team, usually slightly above the baseline salary. At the NHL level,although impossible to draw a distinct line in the sand, replacement level is somewhere between a 4th line skater and a career AHLer that gets called up from time to time. For Sharks fans, think of Andrew Murray, Brad Winchester, or Jim Vandermeer. You might be wondering, why in the hell would we care about how these players have performed? The key to understanding the importance of replacement value is in its utility. By defining a base for talent and/or value, we can then compare players to that base. We otherwise end up comparing all players to peers without an ability to truly understand what that difference means. Replacement level gives us the floor from which our player valuation systems can stand.
Unfortunately hockey is no where near baseball, and we don’t have conclusive measures of offensive and defensive talent, let alone an arsenal of holistic measures that can be used for player valuation. After spending considerable time thinking about this, and mining through data, I’ve reached a tentative conclusion why hockey is much more difficult to parse than other sports.
- Skaters in the NHL play both offense and defense simultaneously. This point may be the most important of all the bullets that follow. For example when a team in the offensive zone cycles the puck they achieve 2 goals. 1) A forward is always high for protection against a turnover leading to an odd man rush (ie. the defensive part of the cycle) and 2) The rotation of players opens passing lanes for scoring opportunities. (ie. the offensive part). Unlike batting and fielding, we can’t separate those 2 events because they happen simultaneously.
- Hockey is played without frequent discrete events. There are faceoffs from time to time, but a majority of the action doesn’t come immediately after a faceoffs, unlike baseball, football, or virtually any other sport.
- Skaters are not highly specialized in their role. Anomalies (Malhotra) aside, all teams feature players that play a considerable amount of even strength, power-play, and penalty kill. It’s not hard to separate these events, but it forces teams to select players that excel at at least 2 of the three. The only player that is specialized is the goaltender. We can make a detailed account of their contribution (mainly, Sv%).
- In many other sports there are objective ways to measure if a team is advancing towards it’s goal. Football features down and distance and field position (in yards), baseball has players on base, no such event exists in hockey. At the end of enough games (I place it around 17), we can make an argument for which teams have possessed the puck more, but within a game, it’s very difficult to objectively document if a team is advancing toward scoring a goal.
- Hockey goals are random chaotic events. Underneath my post about Plank and Taylor’s prediction probability lay a golden stat egg. The fact that hockey goal scoring within a game follows a Poisson distribution extremely well should be absolutely alarming to you. Ie. The number of goals scored by a team in a game can be predicted at the same probability as cars lining up at an intersection. As difficult as that may be to accept, a non-hockey fan might tell you that 10 beast-men skating at speeds vehicles travel, on the earth’s most slippery substance while trying to command a tiny peice of rubber into a narrow hole between a pipe and the largest human you’ve seen may result in a lot of chaos.
But in an effort to keep pushing the envelope ever so slightly, I’ll try to break player talent into discrete partitions in an attempt to help define NHL replacement level. Personally I feel this is an absolute necessity. It’s great that new stats and measures have added considerable information to how we value players and teams, but I’ll keep beating the same drum I’ve been beating all season. Context is the most important characteristic of a stat. I’ve spent a lot of time talking about reliability (# of events usually expressed in games, before a stat is more talent than noise), which is very important, but we now must also consider value over replacement level for individual stats. As I mentioned above, we need that floor from which all other values can be compared.
A brief discussion of methods before I present the data. I began this journey in an initial attempt to parse out the contribution each line from each team has made toward their team’s success this year. I came to realize that I wasn’t really getting anywhere comparing lines, they were essentially floating stats. I realized the necessity for something to compare all the lines to, which resulted in my hunt for replacement level. Replacement level changes from year to year, and if I was getting paid I would have went back and looked at historic replacement level players, but I’m not so you get stuck with 1 year, 2011-2012.
For a majority of the stats (the rate stats) I took players with at least 30 games played, divided them up by time on ice per game (TOI/G). I define top line and top 6 as forwards with the 3 and 6 greatest TOI/G, respectively. Bottom 6 and 4th line represent 7-12 and 10-12 TOI/G respectively. The same logic is applied to defensemen.
Replacement level is a bit fuzzier, but I tried to focus on player’s with no potential to improve, (ie UFA, not RFAs), and earned less TOI than all the players mentioned above. For forwards that means <10 min TOI/G on their team. TOI/G isn’t the greatest measure of replacement level for defense, so I just used all the players that earned less TOI/G than the 6th highest defensemen for their team. I also took out players under 22 and 25 for forwards and defensemen respectively, to further eliminate the possibility of young talent on tryout creeping into the sample.
I’ll first present what’s already been developed by Tom Awad (Goals Vs. Threshold). Alan Ryder (Player Contribution, not shown below), and somebody over at hockey-reference (Point Shares). These are the current holistic metrics that attempt to assign a value to players above a theoretical replacement level. They’ve done a tremendous amount of work, and should be applauded for their contribution to the hockey community. What I found interesting is that (atleast to my knowledge, and if you know I’ll love to hear about it) not one of these authors have spent time defining replacement level. In fact, Ryder says that it’s virtually impossible to define (for many of the same reasons I mentioned above), and pegs it somewhere around 58% of average.
By definition these measures have to put replacement level near or equal to 0, and work backward from there. What I found really interesting is that there is some clear blurring between replacement level and 4th line talent. The key difference is that replacement level actually costs more! The real difference is that in the NHL, the lower lines are often be filled by young talent on RFA and EL contract’s. Take Demers and Braun, who make up the Sharks bottom pairing defense. Their Cap hit was (1.25 + 0.875) = 2.125 M, both on RFA contracts. A lot of GMs (thankfully, not DW) end up burying their over-zelous UFA acquisitions on the lower lines, at a significant cost. Also, replacement level often includes more than 2 or 3 players also contributing to an increased cap burden.
The biggest issue I have with the current holistic metrics is the noise in the signal. It’s true that they do a very good job accounting for all the available value in the NHL. I agree with them that there must be some currency for wins in the NHL, and by observation, goals makes the most since. But for precisely the reason I mentioned earlier (goals scored follows a Poisson distribution) assigning contribution or value to a player based on random events makes that value or contribution extremely variable year to year. Last year Demers finished in the top 25 for PC and top 50 for GVT for defensmen, and this year finishes with near replacement level. This happens all the time with these metrics because there is no attempt to separate talent from noise. Demers scored a lot of big goals last year, which vaulted his GVT/PC, but he didn’t this year.
I also think these metrics greatly over-value goaltending talent. I agree that goaltenders have a huge impact on the game, but we get back to the same noise vs. signal, or reliability problem. Small changes in Sv% result in large swings in goal differential for teams But those small movements in Sv% are due to bounces and variation, not a real repeatable talent. Career EV Sv% is the only reliable measure for goaltending, and trying to assign less than a season’s worth a data to goaltending talent greatly over-estimates the goaltenders actual value going forward.
In summary, I think the current holistic measures are fantastic, and a great step in hockey analysis. My biggest issue has been the stubbornness to adhere to accounting principles, rather than statistical reliability. Ultimately I think GVT/PC/PS tell us what has happened to a player, rather that what they will do.
So keeping these metrics in mind, we’ll now move to the more widely used advanced metrics, breaking them down by line to gain further understanding of elite play through replacement level. What comes next is what I’ve dubbed contextual stats.
The 2 tables above give us the most important distinction between lines. There is a clear trend for increased or decreased ice time, zone start%, and competition. This will frame everything else talked about in the remainder of the article, and is the biggest difference seen between the various lines. This also separates hockey from other sports in that elite players tend to play other elite players more. Unlike baseball, where all batters tend to face the range of pitcher talent, or football in which quaterbacks face (nearly) all levels of defensive talent, hockey players are on the ice with like competition, further confounding traditional metrics.
The first set of columns get a lot of attention, especially at the beginning of the year. What strikes me here is the decline in Sh%, increase in Sv%, for forwards. Perhaps a year’s worth of data isn’t enough to draw this out, but I’m not totally convinced that we should say that PDO should be 1 (or 1000) for all players. In this context it certainly seems to me that it’s acceptable to demand a higher Sh% out of your top line. The other interesting thing is that they are not mirror images of each other. Top forwards Shoot well, and defend well enough to slightly inflate their PDO, while replacement and 4th liners shoot horribly, so as to increase their peers Sv%, while winding up with a sub-par PDO.
In terms of points, I have to agree with Alan and Tom, that goals are probably worth more than assists. Last year I looked at weighting points, and I’m still in favor of this. the weighted Points per 60 min TOI (WP/60) is Goals/60*1.44 + A1/60*1.32 + A2/60*0.24. This keeps the total equal to 3, but weights goals and first assists as much more valuable than second assists. This also brings out the value of replacement level, which is close to nothing.
Outside of penalties taken vs drawn, there isn’t a huge difference in these numbers. That’s not to say they don’t mean anything. The variation within lines is quite high (hopefully I’ll be able to post those 2011-2012 numbers soon). Across the lines the variation isn’t that high because we’re averaging all these players competition against each other, and thus must come close to either 0 for Corsi Rel, and Corsi On, or 0.50 for Corsi%.
As many of you know, Fear the Fin is an independent site run by Sharks fans for Sharks fans. Help keep Fear the Fin independent by contributing to our