Phil Birnbaum, who is one of the great non-hockey analytics writers out there, has taken a number of stabs at the shot quality question in hockey over the last couple of years, and today weighed in on the Tom Tango controversy.

The issue – whether to weight goals significantly higher than other shots in corsi analysis – is one I’ve stayed relatively quiet on, and that’s simply because like any good jury, I want to see all the evidence presented before coming to a decision. We’re still not at the point where I’m totally confident evaluating the worth of Tango’s statistic, or the merits of shot quality overall, but I think that’s partially because the answer depends on the question we’re trying to answer. Does shot quality matter? Absolutely. Does it render large sample shot differential metrics useless? Nope. Can it be used to improve on what we have? I think it can. As Birnbaum has often pointed out, we know that shot quality impacts shooting percentages because we see it in score effects. Is it possible a team could play a system in which they more resembled a team down a goal than a team in a tied state, thus impacting shot differentials and shooting percentages? It’s possible, although it’s important to note that there are psychological factors involved in score effects, as well as the other team playing a certain way. It’s not just one team that impacts it. It also seems quite plausible that teams make the conscious choice to forego shot attempts in order to try for better shots. I think the Ducks are a team that have done this the past couple of years, and the Leafs may be as well. Those changes aren’t enough to impact the idea league-wide that more shots = more goals, but on a team level it could. This is where the sniff test comes into play. We may not have the statistics to prove such decision-making exists, but that just means we have to try harder to find them.

So where does this leave us? Ultimately, dealing with goal data is still very difficult because of the variance involved in goalie play and shooting combining into something that tells us very little reliably. But it’s time that the discussion shifts from the dismissive “we’ve done a regression and we’ve shown that shot quality doesn’t impact the numbers much so it’s not worth pursuing” to “what can we do to limit the variance involved and get meaningful data out of teams’ ability to convert shots into goals”. I don’t think the Tango statistic adds all that much – although I’m waiting to see a version that accounts for score effects – but I also think that’s because it’s so basic and doesn’t really do anything to account for the variance involved in goal scoring.

What’s next? Without tracking data, I’m not really sure. But I definitely wouldn’t want to be somebody staking their reputation or career prospects on how good a team is or isn’t based on corsi in cases where there’s the possibility that system effects or even a changing environment based on shots and carry-ins becoming more of a policy target (see Goodhart’s law) are skewing those numbers and failing to give a truly accurate representation of a team’s even-strength play.

Birnbaum is somebody I will be following closely through all this. As I will everybody else who is refusing to accept “shot quality doesn’t make much difference” as proven fact. After all, absence of evidence does not equal evidence of absence.