A d20 Versus the Chi Squared Test
Posted by Doug on February 10, 2009
So I was trawling through the official Dungeons & Dragons forums, and came across one thread where a DM claimed that he had a player who had rolled a 20 on a twenty-sided die a total of twenty-three times in a row. Of course, given that the odds of this occuring are 838,860,800,000,000,000,000,000,000,000 to 1, the consensus of the community seems to be that either the “lucky” player is cheating, or that the DM is full of it.
Of course, the player may not be consciously cheating, and the DM may not be deliberately lying, but just consider: assume that the odds of a player cheating (consciously or not) are 1 in 1000 (generous) and the odds of a poster in a forum repeating nonfactual information (consciously or not) are also 1 in 1000 (veeeeeerrrrrry generous). This means that either a cheating player or a nontruthful poster are 838,860,800,000,000,000,000,000,000 times more likely than a non-cheating player actually rolling twenty-three 20s in a row.
The other forum-goers who decided to chime in brought up loaded dice (a possibility that the DM vehemently rejected), and this got me to thinking about my own dice: were they fair? Does, for example, my green-and-white Chessex d20 have exactly the same chance of landing on any one of its twenty faces?
There is a way to test this: a chi-squared test. I rolled my green-and-white d20 a total of one thousand times (each time rolling it around in my hand for a second before rolling it about thirty centimeters across my padded card table so that it bounced off of my keyboard before coming to a rest), recording each roll in an Excel spreadsheet. Here’s what I got:
Right away some things start looking a little fishy: 8 and 20 don’t seem to show up as often as they should (1000 rolls evenly divided among 20 results means each result should be expected to occur 50 times). 1 and 13 don’t show up that often, either. But 12 and 15 show up a lot. Looking at the die, I can see that 12 and 15 are adjacent to each other, and if you hold the die so that those two are the highest, then 1 and 13 are adjacent on one side of the die, and 8 and 20 are adjacent on the exact opposite side of the die. Simply put, it looks as though the die is unbalanced such that it is more likely to end up on a 12 or 15.
It looks a little fishy…but “looks” is not equal to “is.” Here’s where the chi-squared test comes in. I take my results to the online chi-squared calculator I linked to above, plug them in, and it tells me…well, it tells me a bunch of stuff I don’t understand, but the one thing I do understand is that the “two-tailed P value” is 0.0100 or 1%, and that is the odds that my die is fair (if it was perfectly fair and each result occurred an equal number of times, the two-tailed P value would be 1.0000 or 100%).
Still, the die may not be perfectly fair, but I don’t think it has been hurting me during the D&D sessions. The average result of the rolls is 10.44, just slightly less than the expected 10.5. A result of 16 to 20 (“rolling high” in my book) happens about 24.5% of the time, only slightly less frequently than a perfect die. A roll of 10 or higher (a success in D&D 4th edition’s saving throw system) occurs 54.8% of the time, just slightly less that the perfect die. A natural 20 (an automatic hit) occurs only 3.6% of the time, but a natural 1 (an automatic failure) only occurs 3.7% of the time, so in the end, I don’t see any reason to stop using my green-and-white d20.
Of course, I have ten other d20s, and almost a hundred dice with varying numbers of sides…and I feel compelled to test them all.