I don’t have a ton of time to blog at the moment with finals coming to an end, but just wanted to throw this up quickly with Ray Shero becoming the New Jersey Devils’ new General Manager and the questions about his seemingly poor draft record. Corey Pronman wrote a nice piece a while back about why Shero’s record in particular is underrated, but I wanted to more briefly examine a few more general reasons why I would be weary about being too reliant on such a history or lack of history of success.

1. Small Sample Size.

One of the central themes with regards to analytics in hockey is that we’re trying to maximize sample size in order to get the most accurate possible view of a player or team’s talent. This is no different with regards to drafting. The fact is, a GM can only draft on average seven players per season, meaning that over the course of, say, a five year tenure, that’s only 35 picks. Some may get hurt, some might lose their love for the game, some might develop better than others simply as a result of random variation. It’s very difficult to isolate real success based on 35 or so picks – which is one of the big reasons why drafting also appears to be so random based on studies in just about every sport.

2. Delegation

General Managers have a lot on their plate. The fact is, it’s not all trades and draft picks. There’s a lot of paperwork, travel, scouting, meetings, staff and other decision-making that the public doesn’t see on a daily basis. Therefore, one of the big jobs of a GM is delegation. They delegate coaching to a coaching staff, legal and salary cap matter often to a lawyer or staff of cap experts, and they delegate scouting to a scouting staff. Now the GM still has final say generally, and they are held responsible for the actions of their delegates, but that doesn’t mean the decisions of the employees reflect perfectly on the capabilities of the executive. So if you combine that with the small sample size issue we’re confronting – and the fact that it can be difficult for a GM to realize that a scouting staff is underperforming in time to do anything about it – it’s a difficult thing to evaluate.

So How Can We Evaluate GMs or Scouting Directors?

There are a couple of keys here. The first thing a GM must ensure is that his scouting staff embodies his ideology on drafting. In the 21st century, that should include an analytic approach which features a combination of game viewership, statistics tracking, hindsight analysis on past drafts, and a formulaic approach to see which prospects, even based off qualitative characteristics, are most likely to succeed. But even if it’s not all that, in order to evaluate a GM based on their drafting, the GM should be sure they’re getting the information THEY want.

How would I evaluate a scouting staff if I was a GM? Or a GM based on scouting if I was a President or Owner? Simple. Beyond simply observing their process, I would take all of their draft lists and create a scale to evaluate those based on an objective ranking of players, say five years into the future. It’s not perfect, since there’s still luck involved and you need time for it to become relevant – which often an owner doesn’t have in evaluating a GM – but at least it increases the sample size, and uses information and data the public doesn’t have to come to the best possible decision on whether a change needs to be made in terms of scouting and drafting.