Should I be Surprised When Someone Predicts Perfectly?

I run prediction markets within Target, predominantly focused on supply chain & operations metrics. For example, will market channel X achieve Y% of fulfillment volume in month Z or will distribution center X exceed 90% capacity before date Y? To date, we’ve closed 6 markets and a single individual (call them Genius) has correctly predicted all of them. It got me wondering; should I be surprised that someone predicted 100% of markets correctly?

All markets have been binary outcomes; either yes or no. Approximately 20 individuals have participated in each market. What is the probability that at least one person predicts all 6 correctly? I have always struggled with plugging the numbers into probability equations… so to answer this for myself I opt for a quick toy simulation. 

My tool of choice is R (Python would be fine, too, I just happened to have a session of RStudio running at the time). If you aren’t interested in the code, skip below for the results and insights. Notice, too, that I over-index on commentary in the code so it’s easy to understand my thought process..

people <- seq(1, 20) # assume same 20 people in each time market
markets <- seq(1,  6) # 6 prediction markets

# define a plus equals function for incrementation
/`%+=%/` = function(e1, e2) eval.parent(substitute(e1 <- e1 + e2))

# create function for single instance where 6 markets are run
# and the outcome for each person is recorded in vector outcome
single_instance <- function() {
  # example output: c(6,2,3,2,3,4,3,3,0,3,3,2,2,3,4,6,2,2,3,2)
  # first person got all 6 markets correct and
  # second person got 2 markets correct, etc.

  # create holder outcome vector with zeros
  # each spot represents outcome for a single individual
  outcomes <- c(rep(0, length(people)))
  # loop through markets and people
  # since 6 markets, we'll do this part 6 times
  for (m in markets) {
    # since 20 people, we'll do this part 20 times
    for (p in people) {
      # assume prob of correct prediction is 50%
      rand_num <- runif(1)
      if (rand_num < 0.5) {
        outcomes[p] %+=% 1
      }
    }
  }
  # return vector where each observation is number of correct 
  # prediction by individual representing that observation
  return(outcomes)
}

# run single_instance x times
# if at least 1 individual predicts all 6 markets correctly, record 'win"
sims <- 10000
wins <- 0
for (i in 1:sims) {
  hold <- single_instance()
  if (max(hold) == max(markets)) {
    wins %+=% 1
  }
}
 
# observe the "win rate" aka the probability that at least 1 person
# predicted all 6 markets correctly
print(wins/sims)

It shows me that the probability that at least 1 individual predicted all 6 markets correctly is ~27%. This means that it’s not likely, but it is not rare. It is easy to confuse this question with another one; what is the probability that Genius, specifically, would predict all 6 markets correctly? There is a major difference between will someone get it and will this specific person get it? To calculate the probability for that second question, we use the equation that everyone knows by heart. You do 0.5 ^ 6 = 0.015 = 1.5% chance that Genius, specifically, predicts all 6 correctly. 

Wait. If I’m not everyone, can I use the sim to verify that 1.5% probability? Yes! At the top of the script instead of having 20 people… make it only 1. Then run it again and, presto, 1.5% probability. The reason I love this approach so much is that it forces me to think through the process I’m simulating. When I capture that, I can model all kinds of things.

people <- seq(1) # assume only a single participant

So, should I be surprised? The big caveat above is that I assumed each participant had a 50% chance of guessing each market correctly. In reality, these individuals probably have good domain expertise on the topics. Despite the fact that the markets are structured near a threshold, they still likely have better than fifty-fifty odds.

As a final exercise, what if that probability of guessing a market correctly rises to 0.9? We modify the random number threshold from 0.5 to 0.9 and reveal that, now, the probability of at least 1 individual predicting all 6 markets correctly is 99.9%.

if (rand_num < 0.9) {
  outcomes[p] %+=% 1
}

And the probability of Genius predicting all 6 markets correctly rises from 1.5% to a 54% probability. Even Genius, despite her wonderful judgment, is still prone to imperfection arising from uncertainty. As with most everything, a perfect track record is part good judgment, but also part luck. So, Genius… if you get the next market wrong, shrug it off, stick with your methodology, and keep your eye on the prize!

Leave a Reply

Your email address will not be published. Required fields are marked *