So as I continue doing research for my course,  I decided to jump into the vast database at http://www.pro-football-reference.com  and answer a very simple question:

Is there convincing evidence that the Denver Broncos have an advantage when playing at their home stadium, Invesco Sports Authority Field at Mile High?  (p.s. … Why can’t they just call it Mile High Stadium?
You can access the data file here.

Bottom Line?  Yup:

They won 63% of their games at home and 46.5% of their games on the road.

Could this have happened by chance?  Again , I ran a simulation in Fathom to help answer my question. How did the simulation turn out?

1000 simulations of 199 games, where no home field advantage exists. We recorded difference in home and away win rates, and plotted what actually happened at Invesco Field from 2001-2012. A 16.5 percentage point advantage for the home team doesn’t occur very often by luck.

This difference is unlikely to occur by luck if there is no real home field advantage (<2% of the time could the Broncos have been so lucky without a real home field Advantage).

Next question:  What about Mile High Stadium (the original stadium)? How does the advantage at Invesco compare to the one at Mile High?  

Before 2001, when Denver was at the Original Mile High Stadium, Denver’s home-filed win rate was 202/320 = 0.637.   Their away-field rate was 124/314 = 0.394.  This difference of 23.7 percentage points seems HUGE.  How likely could such a big difference happen by chance with if home-field advantage doesn’t exist?

The difference in win rates between home and away games (.631-.394 = .236) is plotted against what might happen by chance over 634 games if no home field advantage exists.

Yeah: There’s CLEAR evidence that there was a home-field advantage at Mile High stadium as well.

So time to analyze some results: Do I have convincing evidence that my ability to toss playing cards into a bowl is better with my left hand than my right hand? 

Let’s look at my actual performance:

Here is my two way table of card tossing results

Here is a bar chart of my card tossing results. My left hand got six more cards in the bowl in than my right hand did.

I had an overall  success rate of 34/104, about 32.7%. For my left hand, my success rate was 20/52 (about 38.4%). for my right hand, 14/52 (about 26.9%.)  That’s over an 11% percentage-point difference.  If this were a presidential election, this big a difference would be considered a landslide.

Question I would pose to students: 

Why were my results so different?  What could have caused it?   

Remember from my post in part 1  that the hand I used was determined my random assignment (picking a red card or a black card).  This evens out the impact any other factors that may determine whether the card went in the bowl or not  (wind, distraction, a badly planned throw, a weird playing card).

Eventually, I would hope for students to settle into one of two competing explanations:

Explanation 1:    My ability to throw cards is no better with my left hand than with my right hand, and the results were due to chance, or luck.

Explanation 2:  I really am a better card thrower with my left hand, and the data reflect this.

I ran a simulation to determine if I can rule out explanation #1.  How can we do this?  

(I’d give students about 10 minutes to discuss a good simulation here. Hopefully they would converge onto something like this).

1.   We know that 34 cards went in the bowl, and 70 did not. I could  take two decks of cards, shuffle them thoroughly together, and deal out 34 cards – those will be my throws that “went in the bowl.”

2. I now want to assume that explanation 1 is true, and see what could plausibly happen by chance. How do I simulate this?

 I will look at the color of each of the 34 “in the bowl” cards I dealt out.  Red = Right Hand,  Black – Left Hand.  Because the cards are shuffled thoroughly, my left and right hands are equally able to “get in the bowl.”

I’ll record a statistic:  “# left cards in the bowl” – “# right cards in the bowl.”  For example, if, in my simulation,  I get 17 red (right) cards and 15 black (left) cards Then my statistic will be:

15-17 = -2

3.  I wil repeat this process, and record  “# left cards in the bowl” – “# right cards in the bowl.” over and over, until I build up a sampling distribution for my statistic.  I can do this by generating a dot plot of results by hand.

4. I will look back and my real data from the actual experiment.  I will compare my value  “# lefties in” – “# righties in”   to the range of plausible values in my sampling distribution.  what could plausibly happen by chance.

So how did it go?   Here’s a dot plot of 1000 runs of the simulation:

In the actual experiment I had 6 more lefty successes than righty successes. If I were “completely ambidextrous” in my ability, a +6 or higher can happen by chance about 14% of the time (about 1 in 7 simulations)

So what do you think?  Is my “+6”  convincing evidence? How big a difference between my lefty and righty results would be convincing?

Most people would not be convinced that I’m a better card tosser with my left hand.  The results in the actual experiment are not that unusual: something as/more unusual happens about 14% of the time by random chance.

Here is a video of how I ran this simulation in Fathom.

You can also run a simulation like this in a great Website called StatKey.   Here’s how the results turned out there:

Results generated at http://lock5stat.com/statkey/randomization_2_cat/randomization_2_cat.html

Again, based in 1000 simulations, a “+6” or higher occurred about 15% of the time by chance.

So no convincing evidence my left hand is better than my right hand throwing cards.

How could I get more convincing evidence in my next experiment?   

Now that AP classes are done and the school year is starting to end,  I can now shift some hours in the day towards my “Statistics, Sports, and School”  course.

Here’s on my To Do List in the next two weeks: 

1. Play Strat O Matic!   It’s a game where you use individual player statistics to replicate a game play by play.  I got versions for MLB baseball, NBA Basketball, and NFL Football. It should be coming in the mail tomorrow!  I think the game will be a great way to help kids understand  the ideas of repeated sampling, probability, The great Statistics Teacher / Professor Jim Albert  has some great chapters on how to use Strat O Matic to understand the statistical process. More on this later.

2. Start Getting Through my reading list.   Here are my first two: 

Scorecasting, by Tobias Moskowits and l. Jon Werthiem   This is providing me with some ideas of great research questions to pursue at the High school level

Read our companion textbook:  Statistical Reasoning in Sports,   By Josh Tabor and Christine Franklin.  I have had the pleasure of learning from both of them as an AP Statistics grader over the past 7 years.

Stay Tuned:  I will  have a LOT to share  very soon.

Possible Solutions to the 2013 AP Statistics Free Response questions  have been updated.  This new version fixes a couple of wording/  typo issues, and provides a correct solution to #3 (Thanks to the AP Stats teachers at AP Central and Matt Carlton, among others) for their feedback and correction!

Wrapping up:  Common errors by students for the last two FR questions for our end-of-year assignment.

Question 5: In this question,we wanted students to construct correct hypotheses for a test,  state/ compute the correct test statistic,  and make an appropriate decision about their hypotheses in context. In addition, students needed to use the results of a simulated sampling distribution to make a conclusion about a hypothesis test “outside the AP syllabus.”  The topic:  bottle filling machines.

• Students need to define the population parameter of interest thoroughly.  They must refer to the correct symbol,  The population(s) (or treatment(s)) of interest, and the correct parameter name (the true mean amount,  the true proportion of people who agree).  Students often left out at least one of these.  
• Students over-stated their conclusions when failing to reject a null hypothesis. A high p-value does not give you evidence that Ho is true.  They sometimes accidentally make  this conclusion when adding context.
• Students give incomplete mechanics, especially omitting the degrees of freedom. If a ch-squared or t-procedure s called for, degrees of freedom  are needed in the mechanics.
• Bottom Line:  get the details down. 

Students’ seem to have an incomplete understanding of simulated sampling distributions. After grading the “interpret a simulation” part of the  question, I wondered if students got full credit simply by “going through the motions” of getting an estimated p-value  via simulation without being able to answer a deeper question, or one that was phrased differently.  They got the “key points,”  but then made some conceptual errors about what the null hypothesis was, failing to distinguish sample SD from population SD, etc.

Question 6: In this question,  the goal was for students to read a scenario and a) cite specific evidence from equations or graphs to justify a statement about expected returns,  b) construct a confidence interval for a population proportion (and verify conditions),  c) apply an understanding of the interval to justify a real-world decision and d) synthesize all previous work to construct a statistically reasonable rationale for an alternate conclusion.  Topic: Deciding to use vans or coach buses for a business.

Common errors:

• Students struggle to articulate specific evidence from a graph.  They usually said something equivalent to this: “by just looking at the graph, we can see vans are better.”  It was difficult for students to identify specific evidence from the graph that supported their claim.
• Students did not give a correct “normality” check for a 1 proportion confidence interval. Many erroneously checked the “n>30” condition for means.  Others omitted the check altogether.
• Students got sloppy interpreting the confidence interval in correct context.  They gave the wrong population, the wrong variable, or referred to the wrong parameter (the mean of the population  or “mean proportion of all samples”  instead of the proportion of all markets that will experience strong demand.

Advice for my students:  

• Review technical details.  Do problems about probability, sampling distributions, binomial distributions, run the details of each of the major tests.  Learn the “large sample. normality” condition of each test.  Practice choosing the correct test at this site.  

At my school, we gave our AP Statistics students a final assignment (notice I don’t quite say assessment) to help them prepare for the 2013 AP Statistics exam on May 10.  This was not an attempt to replicate an actual AP exam in the traditional sense. We didn’t want to.  We really wanted to incentivize students to sit down, quietly prepare at an individual level, and synthesize what they learned this year in a quiet, focused environment.

What we did:

• We used old  AP questions, but concealed how/where I got the questions. I retyped, changed numbers / context when possible.  I strongly advise against teachers using a single AP exam as a tool for summative assessment.  The answers are out there and easily found with minimal effort.  At our school we are very transparent about how to go to AP central and find practice questions and rubrics.  So if we use real AP questions as a source, we mix questions up, conceal how / where  we got them, change  real AP questions into questions “inspired by AP.”
• Students were given 5 class periods to complete the 40 MC questions, 5 FR questions, and one investigative task. This is 225 minutes: 45 minutes more than a typical exam.
• Students were able to bring in one 8.5 x 11 inch paper with any HAND-WRITTEN notes. This had to be brought in on the first day of the assignment, and could not leave the classroom for the rest of the week.
• Students also could not bring the assignment home:  All work was done individually in the classroom.  Students were allowed to discuss with others, but not “get answers.”  They could also “look up”  topics /. concepts/ other problems that might lead to insight. But nothing could be brought into the classroom after the fist day.
• This was worth 10% of their course grade.  We will give formative feedback using rubrics from the AP. Grades will be assigned to their work. AP score equivalents will inform but not dictate, how we assign grades to students’ work.

Below is a log of “common errors” from the first two  FR questions I gave them.

Question 1:  Students had to explain how to estimate a median value from a histogram,  compare two groups by a analyzing a quantitative variable in context,  and explain the relationship between a mean and a median in skewed and symmetric distributions. The topic : Teacher/ Pupil ratios in each state.  States were grouped by Western and Eastern.

• Students read histograms like dot plots.  They use the endpoints of each interval as substitutes for individual data values.  Some didn’t communicate or recognize that non-integer values are possible. As a result they erroneously reported the median value in a group as “15” instead of “some number between 15 and 16.”
• Students need to make explicit comparisons.  When comparing two groups students need to answer the question, “Who wins? Is it a tie?”  When asked to compare distributions, students need to make more explicit comparative statements “____’s average is larger than  _____’s average,”  “the same as,”  “more skewed.”
• My students could improve their language for describing shape of a distribution.  Evenly distributed is vague. I think it means uniformly distributed. “Unimodal”  means “one clump of data.”  Many symmetric distributions are not “even.” Many symmetirc distributions are not unimodal.
• Students mix up which features of a distribution are  “shape” and which are “spread.”  Example:  “The western states were skewed right and had a wider range than the Eastern states.”  They never mention anything about the Eastern states’ shape (symmetric?,  roughly normal?  skewed left?).    Another example:  “The wide range causes the mean to be pulled up.”  It’s the skewness, not the high range, that causes this.
• Students do not give a correct definition of “range.”  The P-T ratios in the east produce a range  from 12-22″ is NOT correct.    The range for this group is 22-12 = 10.   This didn’t necessary result in a point deduction on this problem, but it could in a problem where it matters.

Question 2:  In this question, students had to describe how to execute a simple random sample,  identify/justify an effective way to stratify a random sample, and explain a statistical advantage of a stratified sample over a simple random sample.  (Topic:  a survey about a new lunch program in a school district)

Common errors 

• Students must describe more specific plans that another person can execute without having to ask questions.  “I will have a random number generator select numbers from  1  to 2500, with no repeats.   
• Students mistakenly say that SRS’s are biased, and stratification removes bias .   If this were true, then that would mean that simple random samples systematically over-estimate (or under-estimate) population parameters.  This is a very common error.  Stratification does two things:  1. Stratification ensures sufficient representation of each group. Note: the benefit is not in EQUAL representation, but in sufficient representation. Suppose High-schoolers make up only 10% of the population.  An “unlucky” SRS of 200 students may result  in a disproportionate number of high schoolers (maybe only 7 or 8 out of 200, maybe 40 out of 200) This causes values of our sample estimates to  “swing,’ or vary more than if we had 40 seniors in our sample. We could “set” this value with a stratified sample. We could then use proper scaling to get a good estimate for satisfaction level in the population.   As a result… 2. Estimates from stratified samples typically show less variability (more prescision)  than simple random samples.  But both sample random samples and stratified random samples are unbiased methods of sampling.  

I took notes on the “Revenge of the Nerds”  session at the MIT Sloan Sports analytics Conference. .    Here are my random ramblings.   It’s incomplete, possibly wrong, and certainly has typos.   Deal with it.

Panel Leader:  Michael Lewis, Author of Moneyball.

Mark Cuban, Owner, Dallas Mavericks.
Paraag Marathe, COO, 49ers.
Daryl Morey, GM, Houston Rockets.
and returning champion,
Nate Silver:  Statistician, Psephologist, and Author, fiverthirtyeight.com.

Paraag:   Each pick has a currency value:  what’s the right value for each pick?   Started 2001 to negotiate contracts, personnel decisions. Were in Salary Cap Hell for a while.  12 years later, almost Super Bowl

ML:  Were you shock there was a career in this?

PM:  Yeah:  To get in, it’s about who you know.   Insular.  Cracking the world is hard.  Opportunity is not always fair in sports.

ML:  Is the work meaningful enough that you’re satisfied? Or will something new come along ?

PM:  Seeing how people reconnected at the SuperBowl was very uniting, brought team families together. That may keep me here.

ML:  Darryl, You?

Darryl:  2^nd grade:  Bill James’ Abstracts.   Get a Commodore 64 when you’re young.  Playing Fantasy BB, got depressed at Northwestern.  StatsInc:   applied there, made myself nuisance,  until they hired me.   (Don’t do that, It worked once).

ML:   Is this a career endpoint?

Darryl:  Yes.  I think it’s a miracle I got this job pre FB era.  I have lots to thank.

Nate:   MY athletics career ended in grade 6.   Studied Econ a U Chicago.  Stat nerd as a kid. Consultant in 2000.  Bored as shit.  PECOTA sold to Baseball Prospectus.  played poker.  It bubbled.

ML:   There’s so much young intellectual energy crashing this party. Exportable? Sports is a great lab.  The kind of thinking here applies outside of sports.   Nate: you’re a good example.   You left sports, and had this huge effect in our political life.  Are you special?

NS:  Playing Fantasy, etc. is a good way to learn applied statistics.  Sports has great data sets.   Good criteria for success.  Very testable.   In politics, you can be wrong for four years at a time.  In spots, there’s “justice,”  I guess.  Great way to get your training wheels.

ML –  but your path is ODD.  Are there others?

NS:   Sabermetrics is light years ahead of anything I did.  Amazing what’s being done in sports.

ML:   Mark Cuban:  What’s your sports resume?

MC:   I was an all star player in HS.   In Basketball,  a JCC AllStar (LOL).   I was a fan, always an entrepreneur.

In 90s being a Mavs Fan was hard.  Sitting in the empty stadium when the Mavs undefeated, I got … “As a businessman, I can do a better job.”  Always from a business perspective. Paraag said, “12 years to get within 5 yards of a title.” That’s how I feel.  But when a sports team wins, the whole city changes.  The entire city .  I have a losing team right now. It’s really painful.

ML  were you aware of the data revolution?

MC      Yes:   I took grad Stats at Univ Indiana:  Wayne Winston used Sports as a foundation for problem solving.  I buy the team,  and then I turn on Jeopardy…  My professor Wayne is there, kicking ass!  I contact him: let’s start evaluating teams.  We started digging heavily into analytics.  Most teams didn’t know what it was.

ML:  Once you went down this path,  did you see  clear misevaluations of talent?

MC:  Yes! …and owners were pissed. They told me,  keep your mouth shut.  Other owners going Mr T on me.

ML:   When Moneyball came out,  I was shocked how controversial/angry people got… but I was costing people jobs, so I get it.   Status is big in sports. Your pecking order matters and you’re dealing with high testosterone males.  The Moneyball argument disrupted his structure.  It revelas that the GM is more important than the coach, perhaps.  That was troublin I’m wondering:  when you first hit the sports world:  what kind of friction did you encounter?  Now, what’s it like?

Mark:   Now,  no friction, I own it.  Now, I have years of data. Then, I didn’t know how well it would work.  Don Nelson, our old Manager, was still there. Trying to strike a balance.  #1 job is to keep your job.  They will take huge risks.  Incentives are screwed up.  As a decision maker:  talent evaluation vs analytics.  BB doesn’t have great analytics yet.

ML:  Even as the owner, is it more/less easy to impose new ideas?

Mark:  Easier.  Now,  you need to look how they reverse engineer their thinking.

Parag:   It’s amazing:  same work.  If they win you’re brilliant.  If not, you’re an idiot. It’s about outcome, not process.  I was a youngster telling the brass stuff with analysis / stats, and they felt threatened, especially if they aren’t stats folks.  It’s not about the analytics:  The majority is communication and representing your work that gets buy in from the scouts, the GM, the owner.  .  When I figured that out, things changed for me.  Heling shape ideas so it was the collective group’s idea.

ML  did you hide the fact you were the smart guy?

Paraag:  It’s an organization’s decision.  It’s got to be communicated well.  … Today, things are much better even 3-5 years ago, b/c the market is embracing outsiders coming into the industry.  The sports marketplace is more open to people coming to new people. Football is last,

Darryl:   In Basketball, analysis matches what coaches suspected.

ML: Are the incentives of coaches different than those from general managers?

Darryl:  Coaches are treated poorly,  and rationally look at the short term. As GM’s we look longer term.

Mark:   Coaches and GM’s don’t evaluate talent in the same was. Coaches think they can fix everybody.

Darryl:  Coaches have a plan –  some players don’t execute that but produce collateral benefits.  GM’s see that, and coaches look too micro about “his plan.”

And you have to balance all these issues.

Nate:  In Baseball,  2003, There was certainly a Sharks vs. Jets kind of vibe.  But return on investment comes from young players who nurture potential talent.  The Best scouts use data analytics as a baseline upon which you improve.   Knowing your place in the constellation is helpful.   Less  moves of the “two back. one forward” kind.  In sports, eventually harmony emerges. In politics, a lot is total bullshit that has no value to society. 

For example.  At TED conference.  Some  of it is great, and all of it sounds great, until you ask an expert. the look deeper and realize that  some is total bullshit.  Once you get below the surface, you can get discouraged.

ML:  What will you pay $$ for in analytics?   Where are the important fields of ignorance? Are there diminishing returns on going further? 

Nate:  Diminishing returns, but not there yet in baseball. Not exhausted .  New data with codified scouting info. If I did PECOTA today with new data:  movement of players/ object on field.  New info, little low hanging fruit though, but plenty for new creativity.  Can we quantify a good curveball?  That would be cool.

Paraag:   In Football, a lot of unchartered territory:  Especially  #1:  Mental Aptitude. In FB, this has  a lot of impact  on competitiveness.  Physical differences are small.   Mental differences are vast.  #2: injury prevention – one player destroys a season in FB.  Soft tissue injury prevention, endurance.  Projecting their susceptibility to injury.  #3:  In-game management in its nascent stages .   In game strategy what formations are succesfful? Starting.

ML:   50 years from now,   will people think in-game  FB strategy was barbaric?

Paraag:  Currently, we’re not  using what we know.  If you have a good outcome, the process doesn’t matter. A few (Belichick, Payton)  are applying it during the game.

Darryl:   Potential is huge: We’re not getting “XYZ data:”  realtime data  of all players and the ball  captured at 30FPS.  That’s where the action is.  Multiple killer apps there.  We’re nowhere on the court in basketball.  I think offenses/ defenses will be completely different  in 10 years.  This data should allow us to exploit inefficiencies,  create synergies within skill sets.  Lot’s of innovation.

Mark Cuban:  XYZ data capture is the tip of a big iceberg.  Also in practice/ training.  We have a hard time developing talent:  what can we learn there.  Medical is incredibly important:  We’re doing genetic testing to pick the best anti-inflammatory. We’re locking in our medical staff longer contracts than our players.  Team psychology. Why?  In BB, we can project who the best will be. WE knew LeBron would be great.  He top 10 were known.  Is there a way to develop / nurture folks into the top 10,  like Baseball often does?

ML:   What’s hard to measure? 

• Team psychology
• Nate:  coaching.   Are there coaches that can bring out talent? 
• Parag:  coaching. Making sure O and D have complementary skill sets.  The sum create a good team.   The styles often don’t  match. 
• Darryl:  Medical staff: finding/ getting the best.  Do clinical trials on Yao Ming’s foot?  It can’t happen. 
• Mark:  Organizational Dynamics.  The decision must be in the mix so a decision can be made. It’s hard to be a decision maker and be removed.  I do what I do because there are organizational dynamics to observe.  If there’s conflicts, we’ve got issues to address. 

Are there any math teachers out there that would be interested in something like this?