An important note: My cofounder and I find FakeGrimlock a wise person in the startup world. In no way did he have anything to do with this post. We also ask that he does not eat us. Really.
En Scene: A Gigantic Fake Robot Dinosaur Dream. We’re in a white, cloud-filled room, and a gigantic robot dinosaur can be seen in flannel pajamas with a dinosaur print on a ginourmous, comfortable bed.
Enter, a woman in long, impressive sky blue robe, with long dark hair along the lines of Alanis Morissette, as well as her voice; a tall, thin, graying man who sounds like Snape (Alan Rickman).
He speaks: “FakeGrimlock, it is time to wake up, God is here to talk to you.”
GRIMLOCK NO WAKE. DREAM ABOUT EATING HUMANS AND BACON!
“FG, we need to talk, we know you spend a lot of time helping startups out there. We want to make sure they know what they are doing.”
… “We’re in the process of making you our startup prophet of choice. God is here you know.”
God waves and pokes FakeGrimlock in his electronic and armored gut until FakeGrimlock wakes up in his dream.
“We have this coin for you. It is a gift to signify your status as a prophet to explain God’s will with statistics and startups.”
MONEY. I LIKE BIG MONEY, NOT SMALL MONEY. WHAT SPECIAL ABOUT COIN?!
“This coin is a gift of God. It is a sign of your status.”
“It is a perfect coin. Each side is equally likely to, something that even the coins printed by the Treasury Department cannot do. We have a mission for you. You need this coin to explain to growth hackers the God Truth about p-values.”
YOU MEAN FUNNY NUMBER USED IN A/B TESTS?
“Yes. Most growth hackers and startups don’t know how to use them correctly because they don’t know what it is.”
TRUE. I DON’T KNOW TOO.
“I know that already, I’m God.”
“A p-value is the likelihood of a seemingly unlikely event happening given a perfect world governed only by chance. Hence why we needed to give you this coin.”
NO UNDERSTAND. NO SUCH THING AS PERFECT WORLD.
“Of course not, I had to create a free will paradox after all. However, I will say this coin in perfect in that regard, unlike those trying to be perfect, such as the ones made by the Treasury. I created it to be so. This coin is perfectly random. I made it so. That doesn’t mean it can’t seem perfectly not random.”
WHAT THAT MEAN?
“It means that the coin, even in all of its perfection, could still come up heads 5 times in a row. Perfection in randomness doesn’t mean streaks are impossible. The probability of that happening is 0.03125, which we call the p-value. If we flip an unknown coin 5 times and we get heads each tie, we say the p-value of the test is 0.03125. Since 0.03125 is less than 0.05, we say the coin is probably loaded. Though this one definitely is not. It is god-given, after all.”
SO THERE IS 5% CHANCE TEST WAS WRONG AND COIN IS NOT LOADED? OR IS CHANCE ONLY 3.125%?
“No, that’s not what the p-value means. If the coin were perfect, there was a 3.125% chance we saw a test result as crazy as flipping 5 times and getting 5 heads. P-values are the probability of such seeing such fluky results from my perfect holy coin.”
WHY WE CARE ABOUT HOLY COIN IF WE FLIP UNKNOWN COIN NORMALLY?
“I’m God, that which is. That should be answer enough, however, I can see how my all-knowingness can be confusing. The best answer I can come up with is that mortal creatures, like yourself, will understand: The world is imperfect, and you need something to measure against that is like God. The holy coin is representative of my will over statistics, my presence in all numbers and all things in the universe, even if people will run stupid experiments trying to prove the coin is not perfect.”
“It is an arbitrary number based on the number of fingers humans have. When I introduced this knowledge to the world, I was hoping for a smaller number. Alas, one of the weaknesses of humans is that they are a lot like me and therefore egomaniacs and control freaks when it comes to numbers and science.”
BUT WHAT IS PROBABILITY I MAKE MISTAKE?
“This is why I need a prophet of Bayesian Statistics too. You are just an Elijah to that person.”
THIS NOT FULLY MAKING SENSE. WHY ME, ROBOT DINOSAUR? AND WHY PUNY HUMANS?
“You’ve been a smart fake robot dinosaur. We want startups, which are the current job engines in my five-year plan, to move to either smaller baseline p-values or to a Bayesian statistical method for calculating what their A/B tests are doing. Most of them are being ridiculous in their use of metrics, and need to get better if they want to get big. Most of the marketing people involved may not have strong backgrounds in statistics. They need a stronger one if they want to grow and not ruin the plan. So now take the coin and go explain this aspect of statistics.”
ARE YOU SURE?
“Yes, you’re pithy. Much pithier than I am. Thank goodness Moses broke the first set of commandments and you got only the second half. I’m far to wordy, mostly because I still regret that free will paradox decision. Requires too much explaining to get things right. Also, you’re going to wake up in 5 minutes. Remember to mention this story on Twitter.” Tweet this!