Questions Posed by Andrea Kremer of  HBO’s Real Sports with Bryant Gumbel.   For this conference,  some big brass in the NFL came to play.GMS, COO’s, GM’s. Also, Aaron Schatz,  Founder of Football Outsiders. 

Kevin Demoff, COO, St. Louis Rams.

Scott Pioli former NFL GM, KC Chiefs.

Paraag Marathe, COO, SF 49ers

Aaron Schatz, Founder, Football Outsiders.

Q:  Do analytics play a role in player evaluation?  

PM:  It’s one tool, and it’s a delicate balance between scouting and data. WRT analytics, there are reams of data about all sorts of things.  Lots of opportunity to augment the scouting reports. The best teams marry both in a good relationships.

AS:   Statistics see every play, including the unmemorable ones. They also allow you to compare an individual to the group.

SP:  “gut”  is based on previous information and experience. That’s analytics to a degree.

Q:  what NFL Combine tests are most “telling?”  

KD:  We graded our players ahead of time  before they did the combine. No single test.  OL coaches look at first 10 yards of the 40.  Ultimately, you are trying to show something different. Most perform as we predict pretty closely.

SP:  I’m not a big fan of the combine.  There’s physical speed, and playing speed. Manti T’eo , in playing speed, is remarkable, even though his pure speed is not so good. we had a 4.3 guy who was not fast on the field. Measurement do have value:  QB hand size . Tom Brady has huge hands.  Others are misleading to me.

PM:  40 yard dash and WRs. THere are a few predictors.  The flying 20:  the 20-40 speed separation speed seems to be a good correlate.  Correlates to separation speed for a receiver.

AS:  We’ve seen the opposite:  only the first 10 seconds seem to correlate. … weird. Also the Vertical jump correlates.  You have to look at it all in the big picture. Speed Score: based on 40 time with to weight. (F = ma, kids!) .  Mark Ingraham is not fast enough for his size.

SP: So much of this game is reaction and them moving the speed.

Q: Do you do other drills/ measurements not in the NFL combine?  

KD:  We

….  lots of individual observations, very little about process, using data wisely. Everybody sounds like they are looking for needles in haystacks.

Q:  Are there key measurable predictors for certain positions?   

PM:  No big theory that I’ve seen  There’ so much covariance between predictors.  There’s no hard/fast linear model.  So many unique systems that become clear.

SP:  Football is  based on so many interdependent relationships, it makes it extremely difficult to value analytics.

PM: That’s why trading players is frought with risk.

SP:  On the other hand,  Wes Welker – a success story after

AS: I don’t know what analytics can do to fit your system, but you can compare him to a “peer group,”  I suppose.

SP:  Not all players are the same in different systems

AS:   Jason David.   on the

KD:  You’re looking what 3-4 OL’s can also be 4-3 OLs? It would be great to know who

SP: You must learn your players very much – that’s essential for a successful free agency system.  The only people who know the players that well are the coaches.

Wow… this is starting to sound like the challenge of applying data to a classroom, no? 

Q    What’s the value of draft picks?

SP:  HUGE.  Having picks are currency.  Many are going to blow up.  Also trades super helpful.

AS:  It’s like the Patriots have mind control over other teams giving up their pick.  We’ve looked at the Jimmy Johnson draft chart.  If you go back in time.  THe best GMs amass picks and trade up the quality and quantity of their picks.

KD:   Great in theory, but often not in practice.  Everybody has a different vantage point.  If the time frame for decision making keeps shrinking,  it’s harder for them to make decisions in the abstract. We’ve all made bad decisions, but depends on the situation.

Q:   Your philosophy between economics and what you’re trying to do.  

NOTE:  It appears clear from this session that data analytics to measure good predictors for success in football is very very HARD.  So many decisions change once a circumstance changes.  Also, decisions create circumstances.  

The same questions keeps coming up:  “How do you quantify that?”   The panelists are left dumfounded.  Every time.

Even the football sabermetrician  can’t do this.  The answers are coming from the experience of the GMs.  THey are using their experience and instinct.

Q:  The Wonderlich Test.  Other psychological Tests. A new test:  Player Assessment Tool. What do we want to try to quantify in terms of evaluating players’ non-athletic (mental, psychological) abilities?  

SP:   Lots of biases in the Wonderlich test.  ( blah blah GET TO THE POINT).  I want to find out more about the “makeup” of the individual.  You can teach someone work habits.You can’t teach work ethic.  Selflessness:  are  they more team / self oriented?

KD :  Coaching matters…. “who do you remind me of?”  What can be done to give your coaches the most information?  The personal aspect is an untapped area.

SP:  With so much change/ turnover in organizations,  having a standardized idea of what you’re looking for.  As you age, you’re referencing people others don’t know: especially with a lot of turnover.  I think it’s a huge advantage in SF – longevity and continuity.

PM:  Consistency and longevity matter greatly.  changing systems (especially with the presence of a salary cap) is not good. Consistency breeds success.

Q: How can you convince a coach to listen to analytics?  

SP:  Mutual respect.  We have lots of competing egos and nonsense.  You need an open minded head coach, and the presentation matters. I encourage you to work on your delivery, your ability to communicate.  It impact how your information is received. Don’t be smug, smarmy, etc.  For football people, understand the role that analysts can have on your success.

PM:  Negotiations and Organizational Behavior.  Solve something first, but a 5/10 with good communication is better than a 10/10.

AS  Watch film with your football folks. See how the coaches  / scouts see things. 

Q how do you use analytics for coaching?   

PM:  Aggressiveness on 4th down,s time management tendencies on 4th downs. Presence of esoteric

Q: Most overrated / underrated in NFL?  

SP  Not answering.

KD  Every potential FA from the Rams.

AS:  I’m the outsider: I’ll say it.  In general the most overrated – those whose athletic abiity does not translate.   Boom/bust rushers,  D’Angelo Hall is an example.  10th and 20th is no a difference.  89th is a problem.  Underrated:  rookies who play well in smaller markets, and those who fail on TV.  LeVanTe David and Tony Romo.  He’s one of the best 6 or 7 QB’s in the league.  Bad Luck with national TV.