## MIT SSAC 2013: True Performance and the Science of Randomness

Time for another all-star panel.   Plus, hopefully, some real mathematics/statistics.   Our Moderator is Daryl Morey, GM of the Houston Rockets. He reminds me of Chris Farley on SNL.  Except he’s a GM of a major sports team.   Stars are:

Alec Scheiner, Cleveland Browns

Phil Birnbaum, By the numbers  author, and sabermetrician

Benjamin Alamar, Menlo College

Jeff Ma, tenXer,  one of the MIT Blackjack Team.

and our returning Champion, Nate Silver.

p.s. .. LOVE that the students at the MIT Sloan School are introducing the speakers, and running the conference.

Q:  Speaking of serendipity and randomness, what got you to this spot?

Alec:  I worked at a big law firm at DC, doing private equity work.  Jerry Jones serendipitously called a colleague of his at work.

Nate:  My route is also serendipitous.  We courted controversy when I bet Joe Scarborough \$1000 on the outcome.  As much as one can say I used good methods in election prediction, a bit of luck along the way helped.

Jeff:  I was part of the MIT BlackJack team.  We wrote the book, it blew up, “21” blew up,  and then Moneyball  blew up – my chance to look into sports.  We sold a sports business to Yahoo.

Ben:  Jeff hired me as a consultant.  They needed an economist to build a stock market for sports. at ProTrade we developed a model.  With that article,

Phil:  I just blog about sabermetrics form home.  I am just as lucky to be here as the rest:  Every little step you take in life is random.

Question: To find the signal in the noise, you need data. Circumstances change in each sport. One main issue by the time you have the signal, the environment changes.  What’s the best way to attack this?

NS:  On the baseball panel,  getting to the root causes is important.  Probably same in all sports.

JM:  Being right is sometimes bad. It’s important to separate the process from the outcome. When something doesn’t make sense in the data, find the root cause. 2012 NBA:  scoring down.  Why?  What was the root cause?  Focusing on process helps.

Alec:  Agreed. We’re trying to rule out predictors, things rather than find predictors in haystacks.  There are only 16 games per season, so a  lot of noise. I think we have a good group.  Others think we have a lot of losers.  Are the teams winning Super Bowls doing everything right?  I don’t think so.

Darryl:  Misattributing signal for noise.  What are some great examples of this?

JM:   Odom/Durant Decision.  The numbers said pick Durant over Odom,  but the reality was that Odom was injured when data was compared.

Alec: We have a bias for 7 foot, athletic players. We err around those mistakes. Understanding the variance in those players as part of the process helps you be more realistic / objective.

Phil:  You are evaluating a player on many different dimensions, each of which has randomness. 250 w/o power  vs .250 with power:  a .290 hitter can hit .325 reasonably often.

Nate Silver:  We think of “luck”  and skill as polar opposites. In the short run, luck dominates. In the long run, skill does. How much short term interference do you have?

Alec:  in NFL,  we have very little time to prove oneself.  16 games.

Darryl:  The Odom /Durant decision reassures me. Nearly every team would have preferred Durant.   We have our own judgment, and computers:  which can’t actually produce random numbers.  What are humans good at? Computers? Balance?

NS:  Humans find patterns well. We sometimes have overactive imaginations.   We see false patterns in noise all the time. Humans, however need to make hypotheses. Computers need to be used carefully/ thoughtfully/purposefully.

JM:  Humask ask, computers answer.  Start partnering with those in industries –  to understand what are the questions that need to be answered. The “non-math”  folks know the questions to ask.  But the mathy folks need to be approachable, willing to listen.

Ben:  Computers help up narrow the range of what is plausible with randomness. If we let data reduce the risk around the risky decisions we make, that’s helpful. Data / computers identify the degree of risk in our moves/ decisions.

Darryl:  When do you overrule what the data suggest?

Alec:  I thought last year was interesting with RJ III..  Was it the right move? Most people say no.  We worked hard to see beyond the surface to see if its a good/ bad move.  We son’t know for 5 more years.

NS:  Depends how successful your algorithm is.  Yes, Use your horse sense. Soem things are really hard to solve no matter We forecast Senate ones. There are special cases, but we have to separate those from the one you’re personally invested in calling some cases.  I think if your formula is good, then the model should usually be trusted.

Darryl:  Jeremy Lin. There’s no data for players like Jeremy Lin,  but little confidence in his outcome.

Darryl:  Jeff’s Book and Nate’s Book are very transparent in shifting the odds, not being sure about their answers. How do you sell yourself?  It’s not always popular.

Alec:  It’s hard. We find the things we’re sure about first, to gain their confidence. It’s a constant sales process. They’ve had tons of success in the past. So to convince them,  it’s a challenge.

JM: Let’s look at Paraag: He has been deferential and advanced. What if he were more aggressive? Would they have been stronger?

Alec:  You pick your battles.  You need to influence people.  It’s very delicate.

NS:  You still need a good business, good management, good culture.  Analytics can’t help with that.

Ben: One of the dangers analysts face:  your sureness, in the face of others who are certain you are wrong. “Why is Burt Lopez kicking our ass right now?”

JM:  In you create this culture around being data driven,  that probably relps, right? It’s just a way of making better decisions.

Phil:  Do you ever talk about that uncertainty?  Does it actually happen?

Ben:  We try to.  If it’s about a single argument, you miss the boat. The process involves risk.  It’s a tough sell.

JM:  Back to the idea: process over outcome.  Think about hitting on 15, dealer has 9.  THe process is clear, but you win 60% of the time (you’re a genius)  lose 40% of the time, (and you’re an ahole).

Darryl:  Let’s talk about weather forecasters…

NS:  Forecasters cheat, because their incentives are asymmetric.  They did figure out forecasting because of chaos theory. There’s a limited degree of possible precision. How do you convey uncertainty?  Hurricane predictions is a success.

Darryl:  Q:  OK smart guys.  How do you measure good process?

Phil: You identify a specfic question you have to answer. Measure talent. That’s unmeasurable. What correlates? the broad outcome?  You can try to measure something that correlates better with talent.  In baseball, pitching statistics, WL record has lots of noise. ERA is better.  From there, Expected ERA even better.  The process is trying to figure out relationships between performance and talent that is less variable.

Alec: If you ignore the process often, the process is bad.

JM:  Outcome is important, but this will emerge in the long term.  Hard to use outcome in the short term.

NS: ID those who use it well:  what are their characteristics?  Learn about this indirectly.  Over the long run,  what metrics correlate with good business decisions?

Darryl:  How can you get good training for this…. is gambling?

JM:  Yes, gambling (poker, blackjack) when you can get an edge. Losing \$100K

NS: Poker players are best at estimating on the fly. Also Backgammon.  Interesting: Your mistakes cost you a lot.  But analytics help you filter out  the really stupid decisions.

Phil:  Play a baseball simulation game:  Strato-Matic.  It’s great to use dice rolls/ spinners to observe the randomness. You get a feel for what happens during he game:  It’s very hard to predict.

Ben:  Always estimate the likelihood you’re right.  And then keep track.  Anything.  We should be good about getting a sense of this

Darryl:  Gambling is a good way to assess your level of sureness.

Darryl:  I wonder if you can “smooth your heart away”  until you lose the “heart” of the data… comments?

Alec – What a terrible question. Pass.

Q:  Can we change the results-oriented culture?  Is this changeable?

Alec:  almost impossible in sports.  Imagine explaining that to your fans.  Little things: at the Cowboys:  look at how the Rockets are.  Look how good they are. That was a good framing.

JM: Education. Explain why you do what you do.

Darryl:  Hire people you respect.  Some teams measure scouting staff,  which makes sense, until you hammer them over their decisions. (SOUNDS FAMILIAR, TEACHERS?)  In sports, informaiton is important. Lots of teams don’t have a collaborative process because info gets out.

JM:  Kevin Compton 1st rule:  Cancel your local newspapers.

Darryl: That won’t work.

Q:  There’s only one winner usually.  Does that incentivize riskier decisions?

JM:  Darryl, you replied, when I asked you Does it scare you that BB is so hard/ random?  You said no, because the noisiness can help me leverage that. #2, you said, yes I am riskier.

Ben: It’s not 1 of 32.  What does that risk add to your situation? What’s the added value?  …

Alec:  But is risky going “against the data,”  or “against the grain?”

Darryl:   “Variance?”

NS:   In Baseball, we (in certain cases)  like a high variance strategy, hope for a hit.

Q:  dichotomy between  process and outcome.  Don’t outcomes influence process?

Darryl:  I don’t get it.  Sorry.  PASS.

Q:  As players play to adapt to a measurement, at the expense of innovations  how do you fight the observer effect?  (SOUNDS FAMILIAR, TEACHERs?  )

JM:  That’s Bad metrics. then you fix your metics.  If it’s not right, then it’s not right. If it is, then it is.

Ben:  In the draft situation, we don’t have NBA data.  we have to use proxies.  We do protect it.  People respond to incentives to have good numbers.

Darryl:  Black box metrics are a BAD idea.   As a manager I have to understand everything.