## Your Goalie May Vary, Part II: Looking at Single-Game Save Percentages

A few months ago, I played around with some goaltender data in order to explore different ways of evaluating goalie performance. Measures like wins and GAA are problematic insofar as they reflect team quality as much as goaltender quality, but save percentage doesn't necessarily tell you what you'd like to know either. For one thing, you need to be capable of a very high Sv% to end up playing regularly in the NHL, so the netminders we care about don't differ a lot in Sv% (i.e., how much better is a 0.915 goalie compared to a 0.912 goalie?). But more to the point, the raw proportion of shots stopped doesn't tell you how often a goalie plays well enough to steal games (or, conversely, how often he plays badly enough to inspire beach-ball memes from his team's fans).

And that variation does matter. After grabbing game log data on team Sv% going all the way back to the last2005 lockout; I calculated the conditional probabilities of winning associated with various save percentages. Long story short: below about 0.825, win probability hovers below 10%; above 0.825, it increases monotonically and steadily until it peaks at 97.6% when the goalie pitches a shutout (a handful of scoreless ties going to shootouts explains the other 2.4%). The single-game save percentage associated with a win probability of 50%? 0.9231. (Interesting trivia note: the single worst team goaltending performance in these data belongs to the Ducks, who gave up 7 goals on 16 shots against the Flyers in November 2006. That Jean-Sebastien Giguere was riding a 0.922 Sv% to the Stanley Cup a few months later only underscores how much single-game performance can vary.)

Rather than focus on a handful of goalies as before, I decided to focus on the last six seasons (beginning with 2007-2008), and grabbed all goaltender game log data in that period from nhl.com, encompassing 15956 appearances from 159 goalies. Partial game appearances and playoffs were included. I used the full database to estimate the probability distribution of single-game save percentages among all NHL goalies, grouping them using 8 bins (<0.825, 0.825-0.850, 0.850-0.875, 0.875-0.900, 0.900-0.925, 0.925-0.950, 0.950-0.975, 0.975-1.000). I also calculated what I define as "Quality Appearances": the proportion of goalie appearances with a save percentage of 0.9231 or better (i.e., the proportion of appearances in which the goalie plays well enough to give his team a greater than even chance of winning). In addition to presenting the overall results, I've calculated them for every active goaltender with at least 50 NHL appearances since 2007-2008. These results are below.

 Goaltender No. Appearances % Quality Appearances Mean Sv% Sv% Probability <0.825 0.825-0.850 0.850-0.875 0.875-0.900 0.900-0.925 0.925-0.950 0.950-0.975 0.975-1.000 League Average 15956 43.4% 0.912 0.11 0.06 0.10 0.13 0.16 0.18 0.16 0.10 Anaheim Jonas Hiller 296 48.0% 0.919 0.05 0.09 0.08 0.15 0.16 0.22 0.16 0.10 Boston Tuukka Rask 173 52.0% 0.928 0.06 0.05 0.09 0.10 0.19 0.19 0.18 0.14 Buffalo Ryan Miller 384 44.8% 0.917 0.08 0.05 0.10 0.13 0.19 0.19 0.17 0.09 Jhonas Enroth 54 44.4% 0.914 0.15 0.07 0.07 0.13 0.15 0.15 0.19 0.09 Calgary Joey MacDonald 107 35.5% 0.904 0.07 0.07 0.15 0.18 0.18 0.14 0.12 0.09 Carolina Cam Ward 361 42.9% 0.915 0.09 0.06 0.09 0.15 0.19 0.17 0.17 0.08 Chicago Corey Crawford 186 45.2% 0.916 0.11 0.04 0.08 0.09 0.23 0.17 0.21 0.07 Nikolai Khabibulin 224 42.0% 0.907 0.11 0.05 0.14 0.16 0.13 0.23 0.12 0.06 Colorado Jean-Sebastien Giguere 229 41.5% 0.910 0.10 0.06 0.09 0.15 0.17 0.19 0.14 0.09 Semyon Varlamov 166 45.8% 0.912 0.08 0.08 0.10 0.12 0.15 0.22 0.14 0.10 Columbus Sergei Bobrovsky 128 48.4% 0.915 0.13 0.04 0.08 0.11 0.16 0.23 0.20 0.06 Curtis McElhinney 70 36.8% 0.899 0.18 0.07 0.09 0.21 0.09 0.15 0.10 0.12 Dallas Dan Ellis 190 41.6% 0.909 0.16 0.07 0.09 0.14 0.13 0.16 0.15 0.11 Kari Lehtonen 270 47.8% 0.916 0.08 0.06 0.09 0.14 0.17 0.21 0.16 0.09 Detroit Jonas Gustavsson 114 32.5% 0.899 0.11 0.05 0.18 0.23 0.11 0.16 0.09 0.07 Jimmy Howard 272 47.1% 0.918 0.07 0.05 0.10 0.13 0.18 0.21 0.18 0.07 Edmonton Devan Dubnyk 139 46.0% 0.913 0.09 0.09 0.09 0.14 0.14 0.22 0.15 0.08 Jason LaBarbera 141 48.2% 0.912 0.13 0.05 0.06 0.13 0.15 0.20 0.17 0.11 Florida Scott Clemmensen 149 46.6% 0.909 0.11 0.08 0.08 0.16 0.10 0.22 0.13 0.11 Jose Theodore 276 40.9% 0.909 0.11 0.09 0.09 0.11 0.19 0.18 0.15 0.08 Los Angeles Jonathan Quick 336 45.2% 0.917 0.09 0.07 0.09 0.13 0.18 0.17 0.18 0.11 Minnesota Niklas Backstrom 334 44.0% 0.915 0.11 0.06 0.10 0.12 0.17 0.18 0.18 0.08 Josh Harding 118 46.6% 0.913 0.09 0.08 0.12 0.11 0.14 0.18 0.13 0.16 Montreal Peter Budaj 187 37.1% 0.901 0.14 0.09 0.13 0.16 0.12 0.15 0.16 0.06 Carey Price 340 45.9% 0.914 0.09 0.06 0.12 0.12 0.16 0.19 0.19 0.08 Nashville Pekka Rinne 319 48.6% 0.919 0.11 0.05 0.09 0.11 0.15 0.17 0.18 0.13 New Jersey Martin Brodeur 370 43.8% 0.912 0.12 0.05 0.11 0.12 0.17 0.19 0.15 0.10 Cory Schneider 108 60.7% 0.927 0.07 0.06 0.07 0.07 0.12 0.22 0.21 0.18 NY Islanders Evgeni Nabokov 333 44.7% 0.911 0.12 0.06 0.10 0.12 0.16 0.18 0.18 0.08 NY Rangers Martin Biron 213 41.8% 0.912 0.10 0.04 0.10 0.17 0.17 0.17 0.16 0.08 Henrik Lundqvist 442 50.9% 0.921 0.07 0.05 0.08 0.15 0.15 0.21 0.17 0.12 Ottawa Craig Anderson 280 48.6% 0.920 0.08 0.08 0.10 0.09 0.18 0.20 0.16 0.12 Philadelphia Ray Emery 131 35.1% 0.904 0.11 0.06 0.11 0.18 0.18 0.11 0.15 0.10 Steve Mason 243 42.4% 0.904 0.15 0.07 0.09 0.16 0.11 0.16 0.15 0.11 Phoenix Mike Smith 259 47.1% 0.916 0.13 0.05 0.10 0.12 0.13 0.15 0.21 0.11 Pittsburgh Marc-Andre Fleury 404 42.9% 0.912 0.10 0.07 0.10 0.16 0.14 0.17 0.19 0.07 Tomas Vokoun 327 49.4% 0.922 0.09 0.03 0.09 0.13 0.17 0.21 0.13 0.13 San Jose Antti Niemi 269 43.5% 0.915 0.08 0.06 0.12 0.14 0.17 0.18 0.16 0.10 St. Louis Brian Elliott 222 43.2% 0.908 0.15 0.07 0.09 0.14 0.12 0.16 0.16 0.12 Jaroslav Halak 227 48.5% 0.918 0.10 0.03 0.14 0.11 0.15 0.15 0.17 0.15 Tampa Bay Mathieu Garon 204 35.5% 0.904 0.14 0.09 0.10 0.13 0.18 0.16 0.11 0.09 Anders Lindback 63 41.3% 0.909 0.06 0.10 0.10 0.14 0.19 0.19 0.10 0.13 Toronto Jonathan Bernier 63 47.6% 0.913 0.14 0.03 0.10 0.10 0.16 0.17 0.16 0.14 James Reimer 111 43.2% 0.916 0.06 0.07 0.11 0.15 0.17 0.17 0.13 0.14 Vancouver Roberto Luongo 382 49.2% 0.917 0.11 0.04 0.08 0.13 0.16 0.20 0.17 0.12 Washington Braden Holtby 78 51.3% 0.925 0.10 0.04 0.10 0.12 0.13 0.10 0.23 0.18 Michal Neuvirth 130 40.6% 0.910 0.09 0.05 0.18 0.12 0.16 0.16 0.15 0.09 Winnipeg Al Montoya 63 39.7% 0.906 0.14 0.08 0.10 0.11 0.17 0.13 0.17 0.10 Ondrej Pavelec 231 39.1% 0.907 0.12 0.07 0.12 0.20 0.12 0.16 0.15 0.07 Free Agent/Non-NHL Brian Boucher 130 39.5% 0.908 0.12 0.07 0.10 0.15 0.18 0.13 0.21 0.05 Ilya Bryzgalov 387 44.4% 0.913 0.11 0.06 0.08 0.14 0.17 0.19 0.16 0.09 Rick DiPietro 113 28.6% 0.895 0.13 0.10 0.15 0.21 0.14 0.16 0.07 0.04 Jeff Deslauriers 62 38.7% 0.901 0.15 0.11 0.08 0.21 0.08 0.15 0.15 0.08 Erik Ersberg 54 37.0% 0.909 0.17 0.04 0.07 0.11 0.24 0.17 0.11 0.09 Martin Gerber 90 45.6% 0.910 0.11 0.07 0.12 0.11 0.16 0.20 0.17 0.07 Johan Hedberg 197 36.2% 0.903 0.13 0.07 0.14 0.13 0.16 0.15 0.11 0.10 Cristobal Huet 152 46.7% 0.909 0.13 0.08 0.13 0.06 0.14 0.20 0.17 0.10 Brent Johnson 105 42.9% 0.906 0.15 0.06 0.09 0.10 0.17 0.17 0.18 0.08 Michael Leighton 74 35.1% 0.904 0.15 0.03 0.12 0.16 0.22 0.14 0.12 0.07 Chris Mason 237 39.8% 0.905 0.17 0.05 0.12 0.10 0.15 0.14 0.16 0.10 Andrew Raycroft 101 34.0% 0.896 0.15 0.08 0.16 0.16 0.11 0.10 0.12 0.12 Curtis Sanford 71 46.5% 0.908 0.14 0.07 0.07 0.11 0.14 0.17 0.13 0.17 Tim Thomas 320 57.1% 0.927 0.06 0.03 0.09 0.11 0.13 0.22 0.23 0.13

A few things to note here:

• Disastrously bad goaltending is more common than one might think. 11% of NHL appearances are associated with a save percentage below 0.825, which (assuming the league average of 30 shots against per game) translates to over 5 goals allowed per game. In contrast, about 10% of all appearances have a Sv% of 0.975 or higher (in practice, these are all shutouts). In other words, single-game Sv% is frequently pretty far from a goalie's average.
• Slightly technical statistical note: this is why looking at the standard deviation of Sv% can be misleading and confusing. When you calculate the SD of Sv%, you're not looking at the variation of single-game Sv%s from the mean Sv%: you're looking at how all the 1's and 0's (representing saves and goals allowed) vary around the average. Once a goalie's faced a few thousand shots, Sv% becomes harder to move, and the SD gets smaller*. At that point, however, the SD tells you nothing about the probability of a bad (or great) game. To drive it home: the SD for Sv% in my all-goalies dataset is 0.000428; almost 100% of the single-game save percentages in the data fall more than 2 SDs from the average.
• The "Quality Appearances" percentage suggests how tough it is to be an elite NHL goaltender. Only 5 goalies have given their teams a win probability over 50% in more than half their starts; they include Henrik Lundqvist (widely considered the best goalie in hockey), multiple Vezina winner and (bafflingly) current free agent Tim Thomas, and three of the best young goalies in the sport in Tuukka Rask, Cory Schneider, and Braden Holtby. Just missing that 50% threshold are two more goaltenders generally regarded as outstanding: Tomas Vokoun (49.4%) and Roberto Luongo (49.2%).
• Andrew Raycroft and Rick DiPietro are as bad as you remember. Though Jonas Gustavsson still collects an NHL paycheck despite being just as bad.

The first obvious question to ask about the table above is how stable these save percentage distributions are. I'll give the short answer here and leave the long answer as a footnote. Short answer: they're reasonably stable, though the threshold for "reasonably" is entirely subjective. Long answer:** Which shouldn't be all that surprising: again, the margin separating NHL goalies in terms of Sv% is pretty thin, so even randomly-generated distributions of Sv% aren't going to vary that much.

So, wrapping up:

• Although game-to-game variation in Sv% is limited by the quality of NHL goaltenders, it still varies a lot more than the standard deviation might suggest.
• Goalies that actually help a team win are rare.
• The distribution of single-game save percentages is fairly stable, though a prospective study of the subject is clearly superior to anything one can do with game log information.

* Quick thought experiment: say we've got a league-average goalie (Sv% = 0.912) who's faced 10,000 career shots. Now say he gets lit up and gives up 5 goals on 15 shots one night before being pulled. His new Sv% is . . . still 0.912.

** My general approach was to bootstrap single-game Sv%s from the full dataset, and see whether the distribution of one set of games predicted that of another set. Specifically, I generated two sets of 1000 25-game samples, 1000 50-game samples, and 1000 100-game samples, and looked at whether one set's distribution predicted the other. One tricky question is how you define "prediction"; what's the threshold for determining whether one distribution is significantly different from another? The problem gets even worse when you're bootstrapping the samples: by construction, the hypothesis you're testing is false (i.e., you're testing whether two samples from the same distribution are from the same distribution). So I took a different approach: I assumed each sample's distribution would predict that of its comparator in aggregate, and instead counted up how frequently they differed according to some test statistic. In other words, you expect a goalie's performance to have some consistency over time, but how often is it not consistent? My second problem: distributions of single-game Sv% are heavily skewed to the right, meaning that traditional statistics assuming symmetry (e.g., arithmetic means and standard deviations, and related tests like t-tests) will be misleading if applied to them. So I used Mann-Whitney U tests (a non-parametric method not reliant on symmetry) rather than t-tests. In the 25-game samples, the U statistics suggested some difference (z-score <-1) in 32% of samples, and a big difference (z<-2) in 4% of samples. Results in the 50-game samples (28% and 4%, respectively) and the 100-game samples (34% and 4%, respectively) were similar.

This item was created by a member of this blog's community and is not necessarily endorsed by Fear The Fin or members of its editorial staff.