Timeline for Multiple comparisons with uncertain number of tests
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| 1 hour ago | comment | added | Roger V. | @whuber the problem with adjustment is ttat with increasing sequencing depth we are bound to have so many variables that no adjusted results are significant. "Not over-analysing" needs to be quantified - metzgenomic data are expensive and may take years to collect... moreover, the data are often made public, so they might be exploited by many research groups - in this case we can't know how many tests are made. | |
| 2 hours ago | comment | added | Roger V. | @whuber multiple comparisons considerations the number of decisions matters instead of the number of variables in the context described in the Q., we make a decision per variable - am I wrong? The Q. does admit for a possibility that testing variables as independent might be an incorrect approach - this could be discussed in an answer as well. | |
| 7 hours ago | history | became hot network question | |||
| 13 hours ago | answer | added | Michael Lew | timeline score: 4 | |
| 13 hours ago | comment | added | whuber♦ | BTW, I don't see what's "technical" about pointing out that in multiple comparisons considerations the number of decisions matters instead of the number of variables. That seems pretty straightforward and basic, requiring no jargon or special concepts to communicate or appreciate. | |
| 13 hours ago | comment | added | whuber♦ | That's a fair distinction between demonstration and exploration. I can speak from experience, though, in attesting to the utility of adjusting one's exploratory (hypothesis-developing) conclusions for the depth and extent of the exploration: it's easy to fool yourself that the patterns you see, out of the many you look at, are real and not spurious. That takes us full circle to the basic question of how to keep from chasing too many windmills as a result of over-analyzing the data. Certainly the principles of multiple comparisons play a role. | |
| 13 hours ago | history | edited | kjetil b halvorsen♦ |
edited tags
|
|
| 13 hours ago | comment | added | Roger V. | @whuber Given that the decisions are the fundamental objective of the analysis - actually, I am not sure that this is the case. Certainly, books say so, and I see how it applies in clinical trials or industrial setting. But in scientific research one is often interested in formulating a hypothesis rather then testing it. Sometimes this can be reduced to testing various hypotheses against each other - like the level if filtering or sequencing depth in the example in the Q. This might be part of an answer, but I prefer let experts to speak. | |
| 13 hours ago | comment | added | Roger V. | @whuber in other words, what you wrote in your comment is partially too technical and partially too abstract for me. If you could write a more pedagogical answer and give suggestions for reading, I'll appreciate it. If you judge that this makes the Q. too broad for SE model, and therefore it needs to be closed - I understand and will look for help elsewhere. | |
| 14 hours ago | comment | added | Roger V. | @whuber I am looking for assistance that professional statisticians could give to a non-statctician, not for a technical discussion aming experts (which I am not.) If somebody can clarify this issue, suggest references, or more rigirous procedures - I am a taker. | |
| 14 hours ago | history | edited | Roger V. | CC BY-SA 4.0 |
deleted 1 character in body
|
| 15 hours ago | comment | added | whuber♦ | It is puzzling that this question suggests the reason or basis for multiple comparisons is the number of features--but that has nothing at all to do with it. The objective is to control an overall decision rate, and this depends on how many decisions you will make with the data and (more subtly) on how interdependent those decisions might be. Given that the decisions are the fundamental objective of the analysis, how is it possible that you could not know how many are being made? Are you really probing for opinions about sequential or adaptive procedures? | |
| 15 hours ago | comment | added | Christian Hennig | The first thing to have in mind is that tests are asymmetric and treat the null and alternative hypothesis in different ways. The sharper multiple comparison corrections are, the more power to detect differences you lose. So this is a trade-off, and therefore there is no general recommendation. The researchers need to decide what exactly they want to control, and this comes at a cost. Furthermore it is very legitimate to wonder whether you need hypothesis tests at all. Choosing an optimal subset of features, say, for prediction, is not governed by the standard error probabilities of tests. | |
| 15 hours ago | comment | added | Eli | Can you clarify what your question is? It sounds like you're asking for an overview of Frequentist and Bayesian multiplicity adjustments. That would probably be too broad for a question here. | |
| 16 hours ago | history | asked | Roger V. | CC BY-SA 4.0 |