Jaekeun Lee's Space     Projects     About Me     Blog     Writings     Tags

P value fallacy and misconceptions

Tags:

Definition of p-value

 

  • P value is frequently used as a metric to evaluate significance of hypothesis testings
  • It is the probability of observing the same or greater statistical summary than actual observed results, given that the null hypothesis is true
  • It is used as a determinator to reject the null hypothesis; the less the p-value, the less the probability of getting the observed results is. This is why null hypothesis is rejected if p-value is low.

   

     

Misconceptions

 

P value is often used as a metric to show how significant the results of experiments are. However, the misunderstandings of p-values are prevalent many scientific research and scientific educations. This made the American Statistical Association release statement on p-values and statistical significance.

The following is the definition of p-value in the statement of ASA:

“Informally, p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.”

 

These are some common misconceptions regarding p-values:

  1. P-value is not a probability that the null hypothesis is true or the probability that the alternative hypothesis is false

  2. P-value is not the probability that the observed effects were produced by random selection

  3. The 0.05 significance level is merely a convention

  4. P-value does not indicate the size or importance of the observed effect

   

Transposed Conditional Fallacy

 

According to the definition, p-value refers to the probability of observing a result given that some hypothesis is true. In a mathematical expression:

$$ P(Observation \vert Hypothesis = True ) $$

  However, p-value is erroneously used as a “score” or “index” to assess the true or false of a hypothesis:

$$ P( Hypothesis = True \vert Observation ) $$

 

Since the two expressions are different, misunderstanding these two conceptions will cause conditional probability fallacy.

     

Principles to have in mind when using p-values

 

Many research use p-value to test the true or false of their model or classify results. However, p-value cannot be a sole evidence that concludes the viability of research or model. Here are the principles that ASA strongly advise when using p-values:

 

  1. P-values can indicate how incompatible the data are with a specified statistical model.
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p0value passes a specific threshold
  4. Proper inference requires full reporting and transparency
  5. A p-value, or statistical significance does not measure the size of an effect or the importance of a result
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis

   

In a gist, p-value is not a panacea to statistical inference or model development. Scientists or data analyst must not selectively pick the p-value that supports their desirable outcome. In order to verify results of research, they should be reproducible: other scientists or researchers should be able to produce similar or consistent result using the same methodology. P-value is only one cause of the reproducibility crisis. For the development of science, the false conventional use of p-values or statistical significance must be refrained.

           

References

  1. Wasserstein, R. L. & Lazar, N. A. advance online publication The American Statistician (2016); http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
  2. http://www.nature.com/news/reproducibility-1.17552
  3. http://www.nature.com/news/how-scientists-fool-themselves-and-how-they-can-stop-1.18517
  4. http://www.nature.com/news/statistics-p-values-are-just-the-tip-of-the-iceberg-1.17412
  5. http://www.ibric.org/myboard/read.php?Board=news&id=270293
  6. http://www.nature.com/news/psychology-journal-bans-p-values-1.17001