Dependency and false discovery rate: Asymptotics
Thorsten Dickhaus2007, The Annals of Statistics
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Some effort has been undertaken over the last decade to provide conditions for the control of the false discovery rate by the linear step-up procedure (LSU) for testing n hypotheses when test statistics are dependent. In this paper we investigate the expected error rate (EER) and the false discovery rate (FDR) in some extreme parameter configurations when n tends to infinity for test statistics being exchangeable under null hypotheses. All results are derived in terms of p-values. In a general setup we present a series of results concerning the interrelation of Simes' rejection curve and the (limiting) empirical distribution function of the p-values. Main objects under investigation are largest (limiting) crossing points between these functions, which play a key role in deriving explicit formulas for EER and FDR. As specific examples we investigate equi-correlated normal and t-variables in more detail and compute the limiting EER and FDR theoretically and numerically. A surprising limit behavior occurs if these models tend to independence. Control of the false discovery rate (FDR) in multiple hypotheses testing has become an attractive approach especially if a large number of hypotheses is at hand. The first FDR controlling procedure, a linear step-up procedure (LSU), was originally designed for independent p-values (cf. ) and has its origins in [3] (cf. also ). Meanwhile, it is known that the LSU-procedure controls the FDR even if the test statistics obey some special dependence structure. Key words are MTP 2 (multivariate total positivity of order 2) and PRDS (positive regression dependency on subsets). More formal descriptions of these conditions and proofs can be found in [2] and [13]. In view of testing problems with some ten thousand hypotheses as they appear, for example, in genetics, asymptotic considerations become more and more popular. The first asymptotic investigations concerning expected type I errors of the LSU-procedure, as well as for the corresponding linear step-down (LSD) procedure for the independent case, can be found in [7] and [8]. A first theoretical comparison of classical stepwise procedures controlling a multiple level α [or familywise error rate (FWER) in the strong
Related papers
Some Results on False Discovery Rate in Stepwise multiple testing procedures
The Annals of Statistics, 2002
The concept of false discovery rate (FDR) has been receiving increasing attention by researchers in multiple hypotheses testing. This paper produces some theoretical results on the FDR in the context of stepwise multiple testing procedures with dependent test statistics. It was recently shown by Benjamini and Yekutieli that the Benjamini-Hochberg step-up procedure controls the FDR when the test statistics are positively dependent in a certain sense. This paper strengthens their work by showing that the critical values of that procedure can be used in a much more general stepwise procedure under similar positive dependency. It is also shown that the FDR-controlling Benjamini-Liu step-down procedure originally developed for independent test statistics works even when the test statistics are positively dependent in some sense. An explicit expression for the FDR of a generalized stepwise procedure and an upper bound to the FDR of a step-down procedure are obtained in terms of probability distributions of ordered components of dependent random variables before establishing the main results.
On the False Discovery Rate and Expected Type I Errors
Biometrical Journal, 2001
The paper is concerned with expected type I errors of some stepwise multiple test procedures based on independent p-values controlling the so-called false discovery rate (FDR). We derive an asymptotic result for the supremum of the expected type I error rate (EER) when the number of hypotheses tends to infinity. Among others, it will be shown that when the original Benjamini-Hochberg step-up procedure controls the FDR at level a, its EER may approach a value being slightly larger than a=4 when the number of hypotheses increases. Moreover, we derive some least favourable parameter configuration results, some bounds for the FDR and the EER as well as easily computable formulae for the familywise error rate (FWER) of two FDR-controlling procedures. Finally, we discuss some undesirable properties of the FDR concept, especially the problem of cheating.
Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
Journal of the American Statistical Association, 2005
Consider the problem of testing k hypotheses simultaneously. In this paper, we discuss finite and large sample theory of stepdown methods that provide control of the familywise error rate (FWE). In order to improve upon the Bonferroni method or Holm's (1979) stepdown method, Westfall and Young (1993) make effective use of resampling to construct stepdown methods that implicitly estimate the dependence structure of the test statistics. However, their methods depend on an assumption called subset pivotality. The goal of this paper is to construct general stepdown methods that do not require such an assumption. In order to accomplish this, we take a close look at what makes stepdown procedures work, and a key component is a monotonicity requirement of critical values. By imposing such monotonicity on estimated critical values (which is not an assumption on the model but an assumption on the method), it is demonstrated that the problem of constructing a valid multiple test procedure which controls the FWE can be reduced to the problem of contructing a single test which controls the usual probability of a Type 1 error. This reduction allows us to draw upon an enormous resampling literature as a general means of test contruction.
False discovery rate control and the dependence structure of test statistics
2012
The False Discovery Rate (FDR) was proposed in Benjamini and Hochberg (1995) as a powerful approach to the multiplicity problem that does not require strong control of the familywise error rate (FWER). The original approach was developed for independent test statistics and was later extended to dependent statistics in Benjamini and Yekutieli (2001) and Yekutieli (2008). In this paper we extend the existing results by showing that the assumptions on the dependence structure among classes of univariate statistics, that lead to the FDR control, may be represented by specific copulas.
On stepdown control of the false discovery proportion
Institute of Mathematical Statistics Lecture Notes - Monograph Series, 2006
Consider the problem of testing multiple null hypotheses. A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (F W ER), the probability of even one false rejection. However, if s is large, control of the F W ER is so stringent that the ability of a procedure which controls the F W ER to detect false null hypotheses is limited. Consequently, it is desirable to consider other measures of error control. We will consider methods based on control of the false discovery proportion (F DP) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg (1995) controls E(F DP). Here, we construct methods such that, for any γ and α, P {F DP > γ} ≤ α. Based on p-values of individual tests, we consider stepdown procedures that control the F DP , without imposing dependence assumptions on the joint distribution of the p-values. A greatly improved version of a method given in Lehmann and Romano [10] is derived and generalized to provide a means by which any sequence of nondecreasing constants can be rescaled to ensure control of the F DP. We also provide a stepdown procedure that controls the F DR under a dependence assumption.
Multiple Testing. Part II. Step-Down Procedures for Control of the Family-Wise Error Rate
Statistical Applications in Genetics and Molecular Biology, 2004
The present article proposes two step-down multiple testing procedures for asymptotic control of the family-wise error rate (FWER): the first procedure is based on maxima of test statistics (step-down maxT), while the second relies on minima of unadjusted p-values (step-down minP). A key feature of our approach is the characterization and construction of a test statistics null distribution (rather than data generating null distribution) for deriving cut-offs for these test statistics (i.e., rejection regions) and the resulting adjusted p-values. For general null hypotheses, corresponding to submodels for the data generating distribution, we identify an asymptotic domination condition for a null distribution under which the step-down maxT and minP procedures asymptotically control the Type I error rate, for arbitrary data generating distributions, without the need for conditions such as subset pivotality. Inspired by this general characterization, we then propose as an explicit null ...
Stepdown Procedures Controlling A Generalized False Discovery Rate
2009
Often in practice when a large number of hypotheses are simultaneously tested, one is willing to allow a few false rejections, say at most ki1, for some flxed k > 1. In such a case, the ability of a procedure controlling an error rate measuring at least one false rejection can potentially be improved in terms of its ability to detect false null hypotheses by generalizing this error rate to one that measures at least k false rejections and using procedures that control it. The k-FDR which is the expected proportion of k or more false rejections and a natural generalization of the false discovery rate (FDR) is such a generalized notion of error rate that has recently been introduced and procedures controlling it have been proposed. Many of these procedures are stepup procedures. Some stepdown procedures controlling the k-FDR are presented in this article.
Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing
The Annals of Statistics, 2010
An important estimation problem that is closely related to large-scale multiple testing is that of estimating the null density and the proportion of nonnull effects. A few estimators have been introduced in the literature; however, several important problems, including the evaluation of the minimax rate of convergence and the construction of rate-optimal estimators, remain open. In this paper, we consider optimal estimation of the null density and the proportion of nonnull effects. Both minimax lower and upper bounds are derived. The lower bound is established by a two-point testing argument, where at the core is the novel construction of two least favorable marginal densities f 1 and f 2. The density f 1 is heavy tailed both in the spatial and frequency domains and f 2 is a perturbation of f 1 such that the characteristic functions associated with f 1 and f 2 match each other in low frequencies. The minimax upper bound is obtained by constructing estimators which rely on the empirical characteristic function and Fourier analysis. The estimator is shown to be minimax rate optimal. Compared to existing methods in the literature, the proposed procedure not only provides more precise estimates of the null density and the proportion of the nonnull effects, but also yields more accurate results when used inside some multiple testing procedures which aim at controlling the False Discovery Rate (FDR). The procedure is easy to implement and numerical results are given.
On a generalized false discovery rate
The Annals of Statistics, 2009
The concept of k-FWER has received much attention lately as an appropriate error rate for multiple testing when one seeks to control at least k false rejections, for some fixed k ≥ 1. A less conservative notion, the k-FDR, has been introduced very recently by Sarkar [Ann. Statist. 34 (2006) 394-415], generalizing the false discovery rate of Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289-300]. In this article, we bring newer insight to the k-FDR considering a mixture model involving independent p-values before motivating the developments of some new procedures that control it. We prove the k-FDR control of the proposed methods under a slightly weaker condition than in the mixture model. We provide numerical evidence of the proposed methods' superior power performance over some k-FWER and k-FDR methods. Finally, we apply our methods to a real data set.
On the false discovery rate and an asymptotically optimal rejection curve
The Annals of Statistics, 2009
In this paper we introduce and investigate a new rejection curve for asymptotic control of the false discovery rate (FDR) in multiple hypotheses testing problems. We first give a heuristic motivation for this new curve and propose some procedures related to it. Then we introduce a set of possible assumptions and give a unifying short proof of FDR control for procedures based on Simes' critical values, whereby certain types of dependency are allowed. This methodology of proof is then applied to other fixed rejection curves including the proposed new curve. Among others, we investigate the problem of finding least favorable parameter configurations such that the FDR becomes largest. We then derive a series of results concerning asymptotic FDR control for procedures based on the new curve and discuss several example procedures in more detail. A main result will be an asymptotic optimality statement for various procedures based on the new curve in the class of fixed rejection curves. Finally, we briefly discuss strict FDR control for a finite number of hypotheses.
Related papers
Strong approximations of level exceedences related to multiple hypothesis testing
Bernoulli, 2010
Particularly in genomics, but also in other fields, it has become commonplace to undertake highly multiple Student's t-tests based on relatively small sample sizes. The literature on this topic is continually expanding, but the main approaches used to control the family-wise error rate and false discovery rate are still based on the assumption that the tests are independent. The independence condition is known to be false at the level of the joint distributions of the test statistics, but that does not necessarily mean, for the small significance levels involved in highly multiple hypothesis testing, that the assumption leads to major errors. In this paper, we give conditions under which the assumption of independence is valid. Specifically, we derive a strong approximation that closely links the level exceedences of a dependent "studentized process" to those of a process of independent random variables. Via this connection, it can be seen that in high-dimensional, low sample-size cases, provided the sample size diverges faster than the logarithm of the number of tests, the assumption of independent t-tests is often justified.
A smooth version of the step-up procedure for multiple tests of hypotheses
Journal of Statistical Planning and Inference, 2007
Characterization of Bayes procedures for multiple endpoint problems and inadmissibility of the step-up procedure. Ann. Statist. 33, 145-158;. More on the inadmissibility of step-up. J. Multivariate Anal. 97,[481][482][483][484][485][486][487][488][489][490][491][492] have demonstrated that the popular step-up (SU) multiple testing procedure is inadmissible under a wide variety of conditions. All conditions, however, did assume a permutation invariant (symmetric) model. In this paper we find a necessary condition for admissibility of multiple testing procedures in the asymmetric case. Once again SU does not satisfy the condition and is inadmissible. Since SU has a somewhat less favorable practical property and a less favorable theoretical property, we offer a smooth version of SU which retains the favorable practical properties and avoids some of the less favorable ones. In terms of performance the smooth version and nonsmooth version seem to be comparable at least in low dimensions.
On optimality of the Benjamini–Hochberg procedure for the false discovery rate
Statistics & Probability Letters, 2008
The Benjamini-Hochberg step-up procedure controls the false discovery rate (FDR) provided the test statistics have a certain positive regression dependency. We show that this procedure controls the FDR under a weaker property and is optimal in the sense that its critical constants are uniformly greater than those of any step-up procedure with the FDR controlling property.
Further results on controlling the false discovery proportion
The Annals of Statistics, 2014
The probability of false discovery proportion (FDP) exceeding γ ∈ [0, 1), defined as γ-FDP, has received much attention as a measure of false discoveries in multiple testing. Although this measure has received acceptance due to its relevance under dependency, not much progress has been made yet advancing its theory under such dependency in a nonasymptotic setting, which motivates our research in this article. We provide a larger class of procedures containing the stepup analog of, and hence more powerful than, the stepdown procedure in Lehmann and Romano [Ann. Statist. 33 (2005) 1138-1154] controlling the γ-FDP under similar positive dependence condition assumed in that paper. We offer better alternatives of the stepdown and stepup procedures in Romano and Shaikh [IMS Lecture Notes Monogr. Ser. 49 (2006a) 33-50, Ann. Statist. 34 (2006b) 1850-1873] using pairwise joint distributions of the null p-values. We generalize the notion of γ-FDP making it appropriate in situations where one is willing to tolerate a few false rejections or, due to high dependency, some false rejections are inevitable, and provide methods that control this generalized γ-FDP in two different scenarios: (i) only the marginal p-values are available and (ii) the marginal p-values as well as the common pairwise joint distributions of the null p-values are available, and assuming both positive dependence and arbitrary dependence conditions on the p-values in each scenario. Our theoretical findings are being supported through numerical studies.
A note on invariance of multiple tests
Statistica Neerlandica, 2008
A note on control of the false discovery proportion
Applicationes Mathematicae, 2009
Robustness of multiple testing procedures against dependence
The Annals of Statistics, 2009
An important aspect of multiple hypothesis testing is controlling the significance level, or the level of Type I error. When the test statistics are not independent it can be particularly challenging to deal with this problem, without resorting to very conservative procedures. In this paper we show that, in the context of contemporary multiple testing problems, where the number of tests is often very large, the difficulties caused by dependence are less serious than in classical cases. This is particularly true when the null distributions of test statistics are relatively light-tailed, for example, when they can be based on Normal or Student's t approximations. There, if the test statistics can fairly be viewed as being generated by a linear process, an analysis founded on the incorrect assumption of independence is asymptotically correct as the number of hypotheses diverges. In particular, the point process representing the null distribution of the indices at which statistically significant test results occur is approximately Poisson, just as in the case of independence. The Poisson process also has the same mean as in the independence case, and of course exhibits no clustering of false discoveries. However, this result can fail if the null distributions are particularly heavy-tailed. There clusters of statistically significant results can occur, even when the null hypothesis is correct. We give an intuitive explanation for these disparate properties in light-and heavy-tailed cases, and provide rigorous theory underpinning the intuition.
Comments on: Control of the false discovery rate under dependence using the bootstrap and subsampling
TEST, 2008
We congratulate Romano, Shaikh, and Wolf for their interesting work. Our only criticism to the presentation of the article, which is otherwise very readable, concerns Remark 1 on p. 8. This is crucial to understanding the method, because it explains that the estimates of the probabilities under the null are determined by the smaller test statistics, so it should have been made explicit at an earlier stage in Sect. 5. Incidentally, the use of 'rth largest' and 'rth smallest' to denote the rth order statistic on pp. 6 and 8 is confusing. The assumption that n is large and that the θ j 's are uniformly away from zero ensures that few non-null statistics will be mixed with the null ones and hence that the estimates of the probabilities in (10) are approximately correct. Since the models used in the simulation study conform to this assumption, we guess that the bootstrap method is shown here at its best. We wonder how it will perform under a sequence of alternatives which approach the null in a more continuous fashion, a more plausible scenario in real-life applications. One interesting aspect of the simulation results presented in Tables 1 and 2 is how well the 'standard' Benjamini-Hochberg method (BH) works in all scenarios of dependence: the FDR is kept below the required 10%, while the power is on average 80% of that of the bootstrap method proposed by the authors. This suggests
Generalizations of the familywise error rate
The Annals of Statistics, 2005
Consider the problem of simultaneously testing null hypotheses H1,. .. , Hs. The usual approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, particularly if s is large, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which we call the k-FWER. We derive both single-step and stepdown procedures that control the k-FWER, without making any assumptions concerning the dependence structure of the p-values of the individual tests. In particular, we derive a stepdown procedure that is quite simple to apply, and prove that it cannot be improved without violation of control of the k-FWER. We also consider the false discovery proportion (FDP) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289-300] controls E(FDP). Here, we construct methods such that, for any γ and α, P {FDP > γ} ≤ α. Two stepdown methods are proposed. The first holds under mild conditions on the dependence structure of pvalues, while the second is more conservative but holds without any dependence assumptions.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.