**Last updated:** 2017-04-17

**Code version:** 335d169

You should know what a \(p\) value is.

A key problem with \(p\) values, when testing null hypotheses, is that they can be difficult to calibrate. That is, it is hard to answer the question “If I get a \(p\)-value of 0.01 (or any other number) how strong is the evidence against the null hypothesis?”

Here we just give a simple (but artificial) example of an test in which a \(p\) value of 0.01 actually corresponds to evidence *for* the null, even though 0.01 is usually considered to be strong evidence *against* the null. (This example is modified from the book Bayesian Analysis, by J Berger, p25.)

Suppose \(x \in \{1,2,3\}\) and \(\theta \in \{0,1\}\) with

x | 1 | 2 | 3 |
---|---|---|---|

\(p(x | \theta=0)\) | 0.005 | 0.005 | 0.99 |

\(p(x | \theta=1)\) | 0.009 | 0.001 | 0 |

Note that the likelihood ratios for \(H_1\) vs \(H_0\) for \(x=1,2,3\) are \(9/5, 1/5\) and \(0\) respectively. So as \(x\) increases the evidence against \(H_0\) decreases.

Now, let us suppose that we observe \(x=2\). Then by definition the \(p\) value for this observation is \[p:= \Pr(\text{we would see evidence as strong or stronger against $H_0$ than $x=2$} | \theta=0).\]

Here “evidence as strong or stronger against \(H_0\) than \(x=2\)” is \(x \in \{1,2\}\). And the probability of this under \(H_0\) is \[\Pr(x \in \{1,2\} | H_0) = 0.005+0.005 = 0.01.\]

So the \(p\) value for \(x=2\) is 0.01.

And yet, the observation \(x=2\) is 5 times more probable under \(H_0\) than under \(H_1\)! So \(x=2\) has \(p\) value 0.01 but is actually evidence *for* \(H_0\).

This example is obviously contrived to make a point: so it only demonstrates that it is possible to contrive a situation where \(p=0.01\) corresponds to evidence *for* \(H_0\).

However, given this it seems natural to ask: in “typical” situations, does \(p=0.01\) correspond to evidence for or against \(H_0\)? Of course, the answer to this depends on what one views as “typical”. For a start towards answering this question see here.

`sessionInfo()`

```
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X El Capitan 10.11.6
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] workflowr_0.4.0 rmarkdown_1.4
loaded via a namespace (and not attached):
[1] Rcpp_0.12.10 digest_0.6.12 rprojroot_1.2 backports_1.0.5
[5] git2r_0.18.0 magrittr_1.5 evaluate_0.10 highr_0.6
[9] stringi_1.1.2 rstudioapi_0.6 tools_3.3.2 stringr_1.2.0
[13] yaml_2.1.14 rsconnect_0.7 htmltools_0.3.5 knitr_1.15.1
```

This site was created with R Markdown