9
$\begingroup$

I came across this fact sheet from the Data Analytics Services Unit at Western Michigan University (https://wmed.edu/dataanalytics):

https://wmed.edu/sites/default/files/P-VALUES%20SIMPLIFIED.pdf

They give the standard definition of a p-value and then write that a p-value can be more simply defined as:

"THE PROBABILITY TO WHICH THE DATA SUPPORT THE NULL HYPOTHESIS"

I like this definition and want to use it in a description of hypothesis testing but want to make sure it is technically correct.

Any thoughts?

Improve this question
$\endgroup$
10
  • 11
    $\begingroup$ It's good you asked, because it's technically wrong and, even as an intuitive explanation and taking in mind the critical assumption "the null hypothesis is true," it still doesn't rise to the category of misleading. Have you looked at our top voted posts on p-values? $\endgroup$ Commented Jul 30 at 21:31
  • 6
    $\begingroup$ As Wolfgang Pauli would say "That's not even wrong." Everything after "A more understandable definition" on that fact sheet is, frankly, garbage, and more confusing than the technical definition. @whuber gave a link to a whole set of questions on p values; I found the first one in that list particularly germane. The formal definition on that page is correct, and you need to understand it It's not even that technical. $\endgroup$ Commented Jul 30 at 21:42
  • 5
    $\begingroup$ I was sympathetic to the general tone ("here is the technical definition, but unfortunately that's confusing") right up to the point where they started with their 'simplified' definition ("Assuming the null hypothesis is true, what do the sample data say about how likely the null hypothesis is to be true?") Maybe "if the NH were true, how surprising would the sample data be?" would be OK, but what they came up with is tragically bad ... $\endgroup$ Commented Jul 30 at 22:10
  • 3
    $\begingroup$ @jginestet I read whubers 'still doesn't rise to the category of misleading' rather differently from the meaning you seem to have taken. To me it appears to be saying it would be overly generous to merely call it misleading - it would have to come closer to talking about the right kind of things to rise to that level; that is, something more in spirit like Peter Flom's 'not even wrong' than 'not misleading'. $\endgroup$ Commented Jul 30 at 23:27
  • 3
    $\begingroup$ @Daoud while I understand the desire among new learners of the concept to have the simplest explanation possible, I've never seen anyone successfully write a simpler one that was actually correct. As simple as possible - but no simpler - is a key requirement of a definition. People don't keep saying it the long way because they like complexity $\endgroup$ Commented Jul 30 at 23:32

1 Answer 1

Reset to default
15
$\begingroup$

Wrong and misleading. For an academic medical center, I think they should aim a bit higher.

  • A point null hypothesis is never actually true, thus there is no reason to "support" or "accept" it, or any variation of that kind of language. You can however say that study evidence was consistent with a null hypothesis.

  • Recall, when performing null hypothesis significance testing - the data are not random, nor is the hypothesis random in regard to potential "true" values of parameters. So there is no associated probability of data or probability of hypothesis to speak of.

  • For Fisher's p-value, the probability we refer to is a multiverse- or a meta-probability. It is the situation in which the study were replicated again and again and again. The frequency of potential values that we infer based on individual replicates within a single study is quantified as the sampling distribution.

  • For instance, I can use variation among students' scores within a classroom to infer how classroom averages might be distributed among other classrooms.

  • The data may be highly inconsistent with the null (even if the null is true or very close to it), what we quantify with a p-value is how rare a result like ours - or one even more extreme - is under that assumption of truth. A low p-value is basically a quantitative way of saying, "that's pretty weird if what you're saying is true". I don't necessarily say that the null hypothesis is false, just that if my data were an outlier, it would be a big one at that. Low grade publishers may expect the authors to outright say that they reject the null.

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ Re "aim a bit higher:" agreed; but a perusal of related documents on that site (which is focused on in-house statistical consulting) suggests there are pervasive problems, both conceptual and with the ability to communicate in English, that militate against achieving anything better. $\endgroup$ Commented Jul 31 at 13:32
  • 5
    $\begingroup$ Concerning your substantive points, they look misleading in many ways. (I am astonished at the number of upvotes). A hypothesis is not intended to be true; under the null hypothesis, the data are random; and the p-value requires neither a multiverse or some "meta" theory of probability to be computed and interpreted. $\endgroup$ Commented Jul 31 at 13:35
  • 1
    $\begingroup$ @whuber +1 In my experience, I agree with you that most of the time we can explain that data are random. There is however, the apparent paradox that the data are already collected and will not change, which is why one sees so many botched interpretations of not just p-values but of confidence intervals as well. The conceptual leap I am trying to make invoking this "meta" stuff is frequentist probability. $\endgroup$ Commented Jul 31 at 14:13
  • 2
    $\begingroup$ I, too, have encountered this misconception of the meaning of "random" many times (including a document about confidence intervals on the web site referenced in the question). The data are what they are, but randomness is not a property of the data: it describes a model used to reason about and draw conclusions from the data. There is no paradox there, but only confusion concerning what one is writing about. $\endgroup$ Commented Jul 31 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.