It goes without saying that strong goaltending is critical to any team hoping to contend for the Stanley Cup, and the quality of the two netminders currently playing in the Finals is a testament to that. What's not always clear, though, is how to know which goalies give a team the best chance to win. Popular measures of goalie performance have obvious flaws — a bad goalie on a juggernaut team can pile up wins without playing particularly well, and a good goalie can have a poor GAA playing on a bad team that gives up a ton of shots — so many analytically-minded hockey fans have turned to save percentage as their metric of choice. Intuitively, Sv% makes sense: it measures the frequency with which a keeper stops pucks, irrespective of the number of shots or how many goals are being scored at the other end. As such, it offers a handy way to quickly compare goalies on their ability alone.
Trouble is, the single data point of career Sv% doesn't really tell you what you'd like to know. On one hand, the average keeper in today's NHL has a Sv% of 0.912; assuming the NHL average of 30 SOG/game, a league-average goalie will give up 2.64 goals per game. If your goalie's Sv% is higher than that, you'd expect fewer; if it's lower, you'd expect more. Except that this isn't really how it works. A Sv% is an average over all the goalie's appearances: sometimes he's better than that (i.e., shutouts), and sometimes he's much worse (example: on March 23, this year's Vezina winner, Sergei Bobrovsky, allowed 4 goals on 11 shots in just over 10 minutes against the Predators). And what we really care about is in the variation around that average: how often can we expect our goalie to steal (or cost) us games? And how likely is a bad goalie to outplay a good one?
Fortunately, the player profiles at nhl.com include a log of game appearances for each season of a player's career. This makes it easy to extract data on shots faced, goals allowed, and save percentage for every game of a goalie's career. Rather than build a full database across NHL goalies, I decided to focus on a handful of cases that would illustrate the differences between goalies of varying quality. I decided to focus on goalies with plenty of career appearances, so goalies like Tuukka Rask and Bobrovsky weren't included; I also focused on career starters, to minimize confounding from partial-game "relief" appearances. Finally, I wasn't picky about the game situation, since all we care about is how often goalies stop pucks; so, my database includes partial appearances and playoff games as well as regular season.
First, I wanted four elite goalies to set the bar. My (maybe subjective) choices were Tim Thomas (career Sv% 0.923), Henrik Lundqvist (0.920), Pekka Rinne (0.919) and Roberto Luongo (0.919). Next, I picked three goalies that I see as capable, if not elite: Antti Niemi (0.915), Ryan Miller (0.915), and Ilya Bryzgalov (0.913). Finally, I added three bad goalies. You might think it'd be hard to find bad goalies with a few hundred career NHL appearances, but the league isn't always quick to catch on, so come on down, Jose Theodore (0.909), Marc-Andre Fleury (0.909), and Steve Mason (0.904).
The first thing we want to know is the probability that each of these goalies has a strong game, a poor game, or a disastrous game. I tend to think of such things in terms of goals allowed. Assuming an average of 2.64 goals against the other goalie, I classified a game allowing 2 or fewer goals as a strong game, 3 or 4 as a poor game, and 5 or more as a disaster. If we assume 30 shots against per game, a good game is a save percentage over 0.900; a poor game corresponds to a save percentage between 0.833 and 0.900; and a disaster is anything under 0.833. The table below shows the observed probability of each type of game for each goalie in our analysis:
A few things jump out at me right away:
1. Tim Thomas and Henrik Lundqvist are really, really good, and Steve Mason is really bad.
2. This might be the only time you ever read that Roberto Luongo is a better goalie than Pekka Rinne, but it appears to be true, at least in terms of the expectation of strong games versus disastrous games. Rinne is a great illustration of variance in action: the overall Sv% is strong, but it looks to be comprised of a lot of great games and a lot of terrible ones as well.
3. Sharks fans, take heart: Antti Niemi is almost Lundqvist-like in his ability to avoid disastrous games.
What this doesn't tell us, of course, is what to expect when different classes of goalie go head-to-head. Even bad goalies, obviously, have great games, and great goalies can be awful. Fortunately, data on variation in Sv% can help us here as well. Using all the game data from each goalie, I calculated standard errors for their career Sv% numbers (without getting too technical: standard error is just a way of describing variation around an average), and constructed 95% confidence intervals around each Sv%. See the table below:
|95% CI - Low||Career Sv%||95% CI High|
The width of the interval reflects the extent to which we'd expect Sv% to vary from game to game, but it's important to remember that the number of appearances is used in calculating the standard error: so, goalies with fewer appearances are going to have wider CI's, just because math. But, still, what I can't help noticing is this: the best Sv% we can reasonably expect from a bad goalie is still worse than the worst Sv% we'd expect from a good goalie.
To explore this further, I used t-tests to compare save percentages statistically between good goalies, capable goalies, and bad goalies. To keep things simple, I didn't compare goalies within their class (i.e., Thomas vs. Lundqvist). The p-values from these analyses are as follows:
As you'd expect, the difference in Sv% between any of the bad goalies and Thomas, Lundqvist or Luongo was statistically significant, or greater than you'd expect based on chance alone. Thomas and Lundqvist are both significantly better than Bryzgalov as well. The interpretation here: it's extremely unlikely, based on Sv%, that a bad goalie will outplay any of these elite goalies. Interestingly, the variability in Rinne's performance really hurts him here; his career Sv% is higher than that of any bad goalie, but variance means the difference isn't statistically significant. So, you'd expect an elite goalie to outplay a bad goalie, but variation in performance does matter. Basically, if you're a good but inconsistent goalie, you can be outplayed by a bad goalie far more often than you'd like.