Predicting The Starting Goalie

The days of rest that a team is expecting also matters. A team that will be have 2, 3, or 4+ nights until the game after their current one is substantially more likely to use their main goalie than otherwise (log OR= 0.288, 0.255, and 0.293 respectively). Since teams never play three games in three consecutive nights, at least one of the ‘current rest’ and ‘expected rest’ effects will come into play for each game.

On top of all of this is the back-to-back game effect. If there was a game the previous night and the main goalie started, then the log-odds of the main goalie starting again are -2.704 lower than otherwise. This effect is specific to the combination of those two conditions, and is in addition to the individual effects of having only one day rest (reference coefficient) and having started in the previous game (log OR = 0.897).

Time of Year Matters

As the season gets later, the main goalie is less likely to start, all else being equal. The log-odds of the main goalie starting the last game of the season is 0.401 less than the log-odds of them starting the first season. If load management is related to this effect, it could be an indication the main goalie is being saved for more and more critical games as the season progresses. Alternatively, it could reflect the increasing cumulative chance of the main goal being unavailable due to injury.

  Finally, the performance of the goalie in the last game matters. For each goal that the team won or lost (or reached the end of overtime) by in the last game, the log-odds of the main goalie starting the next game is -0.120 higher if the secondary goalie was in net last game, and is (-0.120 + 0.225 = 0.105) higher if the main goalie was in net last game.

In other words, if the team won their last game, they are more likely to continue using whatever goalie was in net for the win. If the team lost, they are more likely to change goalies between one game and the next. Furthermore, the strength of this tendency is proportional to the size of the win or loss. Here, a game that goes to shootout is treated as if it has a goal differential of zero.

Model performance

Let’s start with some null model data:

As shown in Table 1, in the training set, 3148 of 5084 (61.9%) of starts were given to the main goalie. In the test set, this rate is a little lower (59.6%) due to season-to- season drift. As such, there are more false positives (predicting a main will start when someone else does) than false negatives. This figure, 59.6%, also gives us a null accuracy to compare to.

Using the full dataset, we find that the model reduces the deviance from 9574 (on 7169 degrees of freedom) to 7961 (on 7138 degrees of freedom), which translates to a Nagelkerke R2 of 0.2733. It doesn’t make sense to speak of variance when using a binary response like “will the main goalie start”, but Nagelkerke’s R2 is a close analogue to reduction in variance from the model. Here it suggests that with the model we can be about 27% less uncertain about the starting goalie than simply assigning the global mean probability.

Table 3 has the confusion matrix of the test set and, for assessing the degree of overfitting, the training set.

Table 3: Confusion matrices of the test and training sets

Test Set:

Sensitivity = TP / (TP + FN) = 1093 / 1243 = 0.879

Specificity = TN / (TN + FP) = 350 / 843 = 0.439

Accuracy = (TP + TN) / (TP + TN +FP + FN) = 1451 / 2086 = 0.701

Training set:

Sensitivity = 2736 / 3148 = 0.869

Specificity = 976 /  1936 = 0.504

Accuracy = 3712 / 5084 = 0.730

From the test set and training set accuracy, we can say there are 24.7% and 31.9% fewer misclassifications respectively, as compared to the null. Using the model is better than just guessing the main goalie every time, but it’s far from prophetic. 

Breaking down the predictions by the fitted probability of the main being selected doesn’t improve the story. Figure 2 is a stacked histogram of the number of times each goalie was used, binned by the fitted probability from our model. Unfortunately, there is not a strong gradient in the actual results along the fitted probability, meaning that we cannot interpret a fitted value of “90% main” as there being an actual 90% chance of the main goalie being selected.

Figure 2

We hope future models improve on these results, but until then, the ‘dead ends’ section gives a detailed record of everything else that was tried that either didn’t improve the model, or was equally good as something already being done.

Dead ends

Weighing games by recency didn’t help

As with any aspect of the sport, the decisions surrounding goalies are evolving. In our three-season window, we see some evidence of this in the drifting proportion of main goalies used from 64% to 60%. Just in case this was a real effect and not a statistical artifact, we tried some weighted models where the games from each season counted for 5-40% less than the games from the next season. For each decay rate, the model either performed the same or worse, possibly owing to the loss in effective sample size.

If a window of more than three seasons were being used, then any long-term changes in strategy might be more apparent, and some long-term adjustment might be more useful then.

Treating home and away starting goalies as correlated didn’t help

There is only limited evidence that one team’s starting goalie choice carries information about the other team’s choice. Over the last three seasons, the home team selects their main goalie to start 65.1% of the time. When the visiting team also selects their main, that increases to 66.8%. When the visiting team uses someone else, the home team goes with their main 62.9% of the time. Exact counts are in Table 4.

Alternatively, the visiting team uses their main 57.4% of all games, which breaks down to 58.8% and 54.6% when the home team uses their main or someone else, respectively. However, it makes more sense to describe the home team’s selection conditional on the visiting team’s selection because the home team has the privilege to make roster selections second.

Nevertheless, this conditional selection is barely statistically significant against independence (chi-squared Yates = 5.934, p = 0.015); it’s not a large enough difference to warrant modelling ‘home starter’ and ‘away starter’ as anything but independent variables.

Table 4: Starts by home-away situation

More detailed goalie rest information didn’t help

An earlier version of this model used the exact number of days of rest of both the secondary and the main goalie as predictor variables, up to 10. For the purpose of fitting the data and making predictions, many of these variables became redundant, especially after the “last three games’ selections” variables were included. However, the model coefficients for these variables were still useful describing a few trends in goalie selection, so the explanation of these coefficients is included here instead. The complete model is included in the appendix.

The amount of rest of the main and backup goalies has a separate set of effects from those of the team’s rest. For short periods, the more days of rest that the main goalie has had, the more likely the main is to appear in the next game (log OR = 0.967. 0.940, 0.653, and 0.513 for 2,3,4, and 5 days of rest respectively), but for longer periods of time this effect disappears and then reverses (-0.394, -0.378, and =-0.499 for 8,9, and 10+ days of rest respectively), possibly because longer rests are an indication that the main goalie is injured or otherwise scratched from the roster.

For the secondary goalie, the story is simpler: for any amount of time more than 1 night that the secondary goalie has started, the main is less likely to start. There doesn’t seem to be any clear pattern as to the amount of rest the secondary has had, and the effect ranges from -0.530 for 2 days of rest on the secondary, to -1.133 for exactly 9 days of rest on the secondary.

Figure 3 shows the log odds-ratio of the main goalie being selected compared to a ‘one day rest each’ baseline.

More Detailed Goalie Performance didn’t help

The model uses the goal differential for the most recent game, but other than that, no other direct indicators of performance are used. Early versions of the model included moving averages of wins, shots against, goals against, and save percentage for both the main and secondary goalies at 5-, 10-, and 40-game windows. None of these variables explained a significant amount of variance that wasn’t already being explained by other variables. 

Some common transformations were also considered, including the difference between the main and secondary goalies’ stats, as well as replacing the actual numbers with their quantiles.

Game Context didn’t help

We suspected that there could be some effects related to the context of the game, such as the relative importance of the game for playoff qualification, whether the opposing team was in the same division, the relative position in the standings of the teams. None of these variables improved the model. 

We also considered the possible effect of rivalries (e.g. “we want Schneider in net against Vancouver because they traded him away” or specialty effects (e.g. “every time Quick is in net against Anaheim we win, so keep doing that”). We did this by including the proportion of games that the main goalie is played against an opponent using the same formula as we used for the proportion of games played in a season; it didn’t improve the model.

The number of starts out of the last 10 games, and the last 40 games, were considered for the model, but the proportion of starts in a season explained more variation than both measures together.

Other Discussion / Future Work

Predictions 2-3 games ahead

The current model offers some predictive ability for the next game to be played by a team, but this information is only useful from the end of one game to the morning skate of the next game. For media, fan, and strategic purposes, it would be much more valuable to be able to predict the starting goalies multiple games into the future.

A simple approach would be to apply the current model to a Monte Carlo simulation of goalie selections and game outcomes. However, given the current model’s predictive strength, those predictions would quickly decay towards the global mean for the home and away teams respectively.

In order to offer something like the “likely starting pitchers” that media routinely does for Major League Baseball, a much stronger model is required. 

More Seasons, Coaching Data

We only considered 3 seasons of information from the Sportlogiq event database because we wanted to take advantage of the most detailed information available. However, the only features that mattered in the end were those already available to the public. With some additional preparation, a database with more longitudinal depth and coaching information could be used instead.

With a dataset of 10 or even 20 seasons, we could better account for changes in goalie selection strategy over time, ages and level of experience of individual goalies, and effects from head coaches which do occasionally change teams. Some other features like rivalry effects which were too noisy or sparse to contribute to the model might become useful after all.

Game Theoretic Considerations

All of this speculation about improved models assumes that coaches and teams are indifferent to others’ knowledge about their starting goalie selection. If there is any strategic advantage, or even the perception of an advantage, that could be gained from an opposing team knowing this before morning skate, then coaches may take additional steps to make their selection harder to predict.

Alternatively, we could assume that the value gained in making one’s goalie selection unpredictable is too small compared to the value lost from making an otherwise suboptimal decision.

There’s still a lot of work to be done in predicting goalie selections, but we hope this manuscript starts some conversations about strategy.

Appendix

The following is an alternative model that looks at main goalie and secondary goalie rest at a one-day resolution up to 10 days instead of a three-day resolution up to 12 days. It also doesn’t explicitly include the choices of goalie for two and three games before the upcoming one. This model performs worse than the final model (e.g. Nagelkirke R2 = 0.273 instead of 0.280), but provides additional insights into the effects of rest on goalie selection, as shown in Figure 3.  

Table 5: Alternative goalie starts model with detailed rest

Null deviance: 9574.2, 7169 DF

Model deviance: 7961.3, 7138 DF

Nagelkerke R-squared: 0.2733

Test Set:

Sensitivity = TP / (TP + FN) = 1101 / 1243 = 0.886

Specificity = TN / (TN + FP) = 350 / 843 = 0.415

Accuracy = (TP + TN) / (TP + TN +FP + FN) = 1451 / 2086 = 0.696

Training set:

Sensitivity = 2709 / 3148 = 0.861

Specificity = 979 /  1936 = 0.506

Accuracy = 3688 / 5084 = 0.725

[1] Measured in log-odds instead of log-odds ratio

[2] In the case of 2019-20, an 82-game season was expected, so ‘number of games played divided by 82’ is used.

[3] ”This team score” minus “opposing team score” at the end of overtime. Shootout goals are not counted.

Follow us on Twitter @Sportlogiq to stay up-to-date with our latest news.