Before discussing Mixture Discriminant Analysis (MDA), it is convenient to briefly review Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) 21. We assume that we have a training data set X of dimension p × N, where p is the number of dimensions (ie, genes) and N is the number of training samples (ie, tumour samples). We also assume that we have a test set Y of dimension p × n and that we have C phenotype classes among the training set samples.

In the training process of discriminant analysis one attempts to learn parameters that specify the clusters associated with each of the phenotype classes. In the maximum likelihood framework, one learns parameters (π, θ) = (π_k, θ_k = 1,..., C) such that the likelihood function

L ( π , θ ) = p ( X | π , θ ) = ∏ i = 1 N ∑ k = 1 C π k f k ( x i | θ k ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemitaWKaeGikaGIaeqiWdaNaeGilaWIaeqiUdeNaeGykaKIaeGypa0JaemiCaaNaeGikaGIaemiwaGLaeGiFaWNaeqiWdaNaeGilaWIaeqiUdeNaeGykaKIaeGypa0ZaaebCaeqaleaacqWGPbqAcqaI9aqpcqaIXaqmaeaacqWGobGta0Gaey4dIunakmaaqahabeWcbaGaem4AaSMaeGypa0JaeGymaedabaGaem4qameaniabggHiLdGccqaHapaCdaWgaaWcbaGaem4AaSgabeaakiabdAgaMnaaBaaaleaacqWGRbWAaeqaaOGaeGikaGIaemiEaG3aaSbaaSqaaiabdMgaPbqabaGccqaI8baFcqaH4oqCdaWgaaWcbaGaem4AaSgabeaakiabiMcaPaaa@5EDC@

is maximised. In the above, f_k denotes the probability function that specifies the probability that the observation x_i is generated from cluster k, π_k denotes the weight of this cluster and θ_k parameterises the cluster. The optimisation of the likelihood is performed using the EM-algorithm, subject to the constraint that ∑k=1Cπk=1 MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaWaaabmaeqaleaacqWGRbWAcqaI9aqpcqaIXaqmaeaacqWGdbWqa0GaeyyeIuoakiabec8aWnaaBaaaleaacqWGRbWAaeqaaOGaeGypa0JaeGymaedaaa@398D@, yielding estimates (π^,θ^) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeGikaGIafqiWdaNbaKaacqaISaalcuaH4oqCgaqcaiabiMcaPaaa@3407@.

Having estimated the parameters, we can now classify a test sample y using Bayes' Theorem as follows. The probability that y belongs to class k is just the posterior probability p(k|y), which by Bayes' Theorem can be written as

p ( k | y ) = π ^ k f k ( y | θ ^ k ) ∑ c = 1 C π ^ c f c ( y | θ ^ c ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemiCaaNaeGikaGIaem4AaSMaeGiFaWNaemyEaKNaeGykaKIaeGypa0tcfa4aaSaaaeaacuaHapaCgaqcamaaBaaabaGaem4AaSgabeaacqWGMbGzdaWgaaqaaiabdUgaRbqabaGaeGikaGIaemyEaKNaeGiFaWNafqiUdeNbaKaadaWgaaqaaiabdUgaRbqabaGaeGykaKcabaWaaabCaeqabaGaem4yamMaeGypa0JaeGymaedabaGaem4qameacqGHris5aiqbec8aWzaajaWaaSbaaeaacqWGJbWyaeqaaiabdAgaMnaaBaaabaGaem4yamgabeaacqaIOaakcqWG5bqEcqaI8baFcuaH4oqCgaqcamaaBaaabaGaem4yamgabeaacqaIPaqkaaaaaa@59E5@

Assigning y to the class which maximises this posterior probability (the maximum probability criterion) minimises the expected misclassification error. Thus,

k = class(y) = max{p(c|y)|c = 1,..., C}

To compute the posterior probabilities one needs to estimate the functions f_k or, if the functional form is prespecified, the parameters θ_k. The simplest functional approximation one can make is to assume that the clusters are multivariate Gaussians, so that

f k ( y | θ k ) = G ( y | μ k , Σ k ) = 1 2 π d e t Σ k e − 1 2 ( y − μ k ) T Σ k − 1 ( y − μ k ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeaabiqaaaqaaiabdAgaMnaaBaaaleaacqWGRbWAaeqaaOGaeGikaGIaemyEaKNaeGiFaWNaeqiUde3aaSbaaSqaaiabdUgaRbqabaGccqaIPaqkcqaI9aqpcqWGhbWrcqaIOaakcqWG5bqEcqaI8baFcqaH8oqBdaWgaaWcbaGaem4AaSgabeaakiabiYcaSiabfo6atnaaBaaaleaacqWGRbWAaeqaaOGaeGykaKcabiqaaaubcaWLjaGaeGypa0tcfa4aaSaaaeaacqaIXaqmaeaadaGcaaqaaiabikdaYiabec8aWjabcsgaKjabcwgaLjabcsha0jabfo6atnaaBaaabaGaem4AaSgabeaaaeqaaaaakiabdwgaLnaaCaaaleqabaGaeyOeI0scfa4aaSaaaeaacqaIXaqmaeaacqaIYaGmaaWccqaIOaakcqWG5bqEcqGHsislcqaH8oqBdaWgaaqaaiabdUgaRbqabaGaeGykaKYaaWbaaeqabaGaemivaqfaaiabfo6atnaaDaaabaGaem4AaSgabaGaeyOeI0IaeGymaedaaiabiIcaOiabdMha5jabgkHiTiabeY7aTnaaBaaabaGaem4AaSgabeaacqaIPaqkaaaaaaaa@6E85@

where μ_k is the mean and Σ_k the covariance matrix of the Gaussian. If, furthermore, we assume that the covariance matrices are identical for each cluster (ie, Σ_k = Σ ∀ k), then the classification function becomes a linear function of y, known as LDA. In the more general case where the covariance matrices of each class are allowed to differ, the classification function is a quadratic form of the y and the analysis is known as QDA.

The assumption that a phenotype class is best modelled by a multivariate Gaussian is often violated. In the context of gene-expression analysis, gene expression profiles often exhibit bi-or multimodality, even when restricted to one phenotype class 5. Similarly, gene expression profiles typically also have longer tails than Gaussians. In such circumstances, it seems more appropriate to model each f_k as a mixture of multivariate Gaussians, since any general density can be approximated by such a mixture. Therefore, one assumes that

f k ( y | θ k ) = ∑ j = 1 G k τ k j G ( y | μ k j , Σ k j ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaemOzay2aaSbaaSqaaiabdUgaRbqabaGccqaIOaakcqWG5bqEcqaI8baFcqaH4oqCdaWgaaWcbaGaem4AaSgabeaakiabiMcaPiabi2da9maaqahabeWcbaGaemOAaOMaeGypa0JaeGymaedabaGaem4raC0aaSbaaeaacqWGRbWAaeqaaaqdcqGHris5aOGaeqiXdq3aaSbaaSqaaiabdUgaRjabdQgaQbqabaGccqWGhbWrcqaIOaakcqWG5bqEcqaI8baFcqaH8oqBdaWgaaWcbaGaem4AaSMaemOAaOgabeaakiabiYcaSiabfo6atnaaBaaaleaacqWGRbWAcqWGQbGAaeqaaOGaeGykaKcaaa@56DF@

where the number of Gaussians to use for phenotype label k is given by G_k. This number may or may not be specified in advance resulting in a variety of different implementations. In ordinary MDA 22, one assumes that G_k is known in advance for each class k and that the covariance matrices are all identical (ie, Σ_kj = Σ). However, these assumptions are not necessary and instead one can use the training data to learn the best mixture model fit for each phenotype class using for example the Bayesian Information Criterion (BIC) 21 or a variational Bayesian framework for model selection 23. This model selection step is a cluster-inference procedure that yields estimates for(τ_kj,μ_kj,Σ_kj, G_k), from which classification of test samples proceeds as before using the maximum probability criterion. Therefore, MDA is a direct generalisation of LDA and QDA and may reduce to these if the data does not support multiple components per phenotype class 21.

Using mixtures of Gaussians, the densities of each phenotype class can be estimated more accurately. Thus, provided that the inferred Gaussian components are biologically meaningful, this approach should in general lead to an improved classification performance. However, the implicit assumption in MDA is that we are still interested in classifying samples into the C phenotype classes, whereas in certain circumstances we may be only interested in classifying into certain subtypes within the phenotype classes. Therefore, while in MDA one allows for heterogeneity of each phenotype label by estimating the density of each class as a mixture of Gaussians, classification is subsequently performed into each phenotype class. On the other hand, it is possible to classify samples into the Gaussian subcomponents inferred for each phenotype class, a variation of MDA called Heterogeneous Mixture Discriminant Analysis (MDAhet), because this explicitly takes the heterogeneity of each phenotype class into account by attempting to classify the samples into these subcomponents.

As an example, consider the case of two phenotype classes with MDA predicting two Gaussian components for each class. Thus, training data is used to learn the parameters and weights for four Gaussian clusters and classification of test samples is subsequently performed via the Bayes' classifier (equation 3) on these four subclasses. Note therefore that in MDAhet, the cluster-inference step of MDA is used to define the classes for which classification is then performed. Since these inferred classes make up subtypes of the original phenotype labels, this classification framework explicitly takes the heterogeneity of the phenotypes into account.

In the context of cancer gene-expression studies it has been a problem in certain cancers to derive reliable prognostic classifiers as is the case for ER- breast cancer. Typically, in the context of prognosis one would expect discriminative gene-expression profiles to exhibit bimodal distributions with the two modes mapping roughly to the two prognostic groups (good and poor) 11. However, as previously shown 5, the best candidate gene-expression prognostic markers can also exhibit bimodal (or multimodal) profiles (ie, mixtures of Gaussians) within a given prognostic class, indicating that these phenotypes are themselves heterogeneous and that classification analysis should attempt to take this heterogeneity explicitly into account. Thus, in such circumstances the proposed classifier MDAhet seems the more appropriate classification scheme to use.

Following the work by Heagerty and colleagues 24, we estimate time-dependent sensitivity SE(t) and specificity SP(t) values using Kaplan-Meier estimators for the predicted subclasses. In our context, we assume that samples have been classified into two groups, so that the predictor X = 1 predicts poor prognosis, while X = 0 predicts good prognosis (ie, the 'good-up' group) Thus,

S E ( t ) = ( 1 − S ^ K M ( t | X = 1 ) ) p ( X = 1 ) 1 − S ^ K M ( t ) S P ( t ) = S ^ K M ( t | X = 0 ) ) p ( X = 0 ) S ^ K M ( t ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeaabiqaaaqaaiabdofatjabdweafjabiIcaOiabdsha0jabiMcaPiabi2da9KqbaoaalaaabaGaeGikaGIaeGymaeJaeyOeI0Iafm4uamLbaKaadaWgaaqaaiabdUealjabd2eanbqabaGaeGikaGIaemiDaqNaeGiFaWNaemiwaGLaeGypa0JaeGymaeJaeGykaKIaeGykaKIaemiCaaNaeGikaGIaemiwaGLaeGypa0JaeGymaeJaeGykaKcabaGaeGymaeJaeyOeI0Iafm4uamLbaKaadaWgaaqaaiabdUealjabd2eanbqabaGaeGikaGIaemiDaqNaeGykaKcaaaGcbaGaem4uamLaemiuaaLaeGikaGIaemiDaqNaeGykaKIaeGypa0tcfa4aaSaaaeaacuWGtbWugaqcamaaBaaabaGaem4saSKaemyta0eabeaacqaIOaakcqWG0baDcqaI8baFcqWGybawcqaI9aqpcqaIWaamcqaIPaqkcqaIPaqkcqWGWbaCcqaIOaakcqWGybawcqaI9aqpcqaIWaamcqaIPaqkaeaacuWGtbWugaqcamaaBaaabaGaem4saSKaemyta0eabeaacqaIOaakcqWG0baDcqaIPaqkaaaaaaaa@7429@

where S^KM MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafm4uamLbaKaadaWgaaWcbaGaem4saSKaemyta0eabeaaaaa@317D@(t) denotes the Kaplan-Meier estimator for the overall survival function, while S^KM MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGafm4uamLbaKaadaWgaaWcbaGaem4saSKaemyta0eabeaaaaa@317D@(t|X = c) denotes the Kaplan-Meier survival estimate for the particular subgroup X = c (c = 1, 2) 24. In our context, however, the most important performance measure is the negative prective value (NPV), since this is the probability of correctly identifying a patient with a good prognosis. Adapting the same methods as used by Heagerty and colleagues 24 we can obtain time-dependent estimates for the NPV and positive predictive value (PPV) simply as:

N P V ( t ) = S ^ K M ( t | X = 0 ) P P V ( t ) = 1 − S ^ K M ( t | X = 1 ) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeaabiqaaaqaaiabd6eaojabdcfaqjabdAfawjabiIcaOiabdsha0jabiMcaPiabi2da9iqbdofatzaajaWaaSbaaSqaaiabdUealjabd2eanbqabaGccqaIOaakcqWG0baDcqaI8baFcqWGybawcqaI9aqpcqaIWaamcqaIPaqkaeaacqWGqbaucqWGqbaucqWGwbGvcqaIOaakcqWG0baDcqaIPaqkcqaI9aqpcqaIXaqmcqGHsislcuWGtbWugaqcamaaBaaaleaacqWGlbWscqWGnbqtaeqaaOGaeGikaGIaemiDaqNaeGiFaWNaemiwaGLaeGypa0JaeGymaeJaeGykaKcaaaaa@56B2@

Applying a feature selection method designed to remove false positives 11 to an integrated expression data set of 186 untreated ER- samples across 5007 genes 3810, we previously identified a total of 22 prognostic genes, seven of which were associated with immune response functions (XCL2, HLA-F, C1QA, TNFRSF17, SPP1, IGLC2, LY9) 5. Furthermore, mapping the seven-genes into those available on two external platforms we were able to separate two independent untreated populations of ER- breast cancer patients 912 into two subgroups with statistically significant differences in survival outcome 5. Specifically, samples overexpressing this module had significantly better clinical outcomes, as measured by absence of a poor outcome event (disease-specific death or the surrogate distant metastasis if the former was unavailable) (Figures 1a, b).

These results motivated us to investigate the prognostic role of the immune response-module further in four additional ER- data sets for which gene expression and clinical data were available 6181920. Using the same partitioning around medoids algorithm to separate each of these additional independent cohorts into two subgroups we were able to confirm the prognostic role of the immune response-module across a total of 469 ER- tumours (Figures 1c to 1f). Given that overexpression of the immune response-module consistently identified a good prognosis subgroup of ER- breast cancer, we asked if we could derive a robust single-sample prognostic predictor.

To derive a single-sample prognostic classifier we first applied a mixture discriminant classifier to the same training set of 186 ER- patients and across the seven identified genes. The heterogeneity of the good-prognosis phenotype, as shown by the gene expression patterns of the immune response-module (Figure 1), suggested to us that MDA 21 would be an appropriate classification method to use, since it is designed to work for such heterogeneous phenotypes. Specifically, the MDA classifier estimates, from the training data, densities for each of the good and poor prognosis phenotypes as mixtures of two Gaussians (Figure 2). The choice of two Gaussians to model each phenotype was not arbitrary but followed from the application of a variational Bayesian algorithm that infers the optimal number of Gaussians to use 23 (data not shown). Thus, using the training data, patients with a good prognosis were divided up into two groups, one with high relative expression of the immune response-genes (the 'good-up' group) and another with relative low expression (the 'good-down' group). A similar subdivision was performed for the poor prognosis patients to yield 'poor-up' and 'poor-down' subgroups. The training process involves learning the mean expression vectors, covariance matrices and weights for each of the four subgroups (Table 1).

Having estimated the parameters for each of the phenotypes, external samples can then be classified by applying the MDA to the test sample's gene expression profile, yielding probabilities of the sample belonging to each phenotype class, and subsequently using the maximum probability criterion for class assignment. Since each phenotype class is modelled as a mixture of two Gaussians (Figure 2), class assignment can also be made on the four subclasses, a novel variation of MDA called MDAhet because this explicitly takes the heterogeneity of each phenotype in the classification process into account. This novel variation of MDA is crucial as it allows for a more reliable identification of good prognosis samples (ie, the NPV).

In detail, MDAhet assigns a test sample with a seven-gene expression profile y to one of the four subclasses c (c = 1, 2, 3, 4) using the maximum probability criterion

c = { j : m a x j = 1,2,3,4 π ^ j G ( y | μ ^ j , Σ ^ j ) ∑ k = 1 4 π ^ k G ( y | μ ^ k , Σ ^ k ) } MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xI8qiVKIOFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaem4yamMaeGypa0JaeG4EaSNaemOAaOMaeiOoaOZaaybuaeqaleaacqWGQbGAcqaI9aqpcqaIXaqmcqaISaalcqaIYaGmcqaISaalcqaIZaWmcqaISaalcqaI0aanaeqakeaacqGGTbqBcqGGHbqycqGG4baEaaqcfa4aaSaaaeaacuaHapaCgaqcamaaBaaabaGaemOAaOgabeaacqWGhbWrcqaIOaakcqWG5bqEcqaI8baFcuaH8oqBgaqcamaaBaaabaGaemOAaOgabeaacqaISaalcuqHJoWugaqcamaaBaaabaGaemOAaOgabeaacqaIPaqkaeaadaaeWbqabeaacqWGRbWAcqaI9aqpcqaIXaqmaeaacqaI0aanaiabggHiLdGafqiWdaNbaKaadaWgaaqaaiabdUgaRbqabaGaem4raCKaeGikaGIaemyEaKNaeGiFaWNafqiVd0MbaKaadaWgaaqaaiabdUgaRbqabaGaeGilaWIafu4OdmLbaKaadaWgaaqaaiabdUgaRbqabaGaeGykaKcaaOGaeGyFa0haaa@6B40@

where G denotes the seven-dimensional multivariate Gaussian and the parameters (μ^j,Σ^j,π^j) MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuqqRPxAKvMB6bYrY9gDLn3AGiuraeXatLxBI9gBaebbnrfifHhDYfgasaacPi6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaeGikaGIafqiVd0MbaKaadaWgaaWcbaGaemOAaOgabeaakiabiYcaSiqbfo6atzaajaWaaSbaaSqaaiabdQgaQbqabaGccqaISaalcuaHapaCgaqcamaaBaaaleaacqWGQbGAaeqaaOGaeGykaKcaaa@3B3A@ are estimated from the training set (Table 1).

The classification distribution of samples from the six external cohorts into the four subclasses as determined by MDAhet showed that test samples classified most often into the 'poor-down' and 'good-up' classes (Table 2). Since samples falling into the 'good-down' and 'poor-down' categories could not be discriminated in terms of prognosis (a sign that these subclasses are not distinguishable on the basis of the expression of these seven genes) we can pool these together in order to compare more objectively the predicted proportions with those estimated from the training set. This revealed that for four cohorts, JRH-2 (8 vs. 16), CAL (13 vs. 33), UNC248 (28 vs. 56) and Loi (13 vs. 27), the 'good-up' group is about half the size of the pooled 'down' group (Table 2), which is consistent with the relative proportions estimated from the training set (0.28 vs. 0.63). For the other two cohorts, relative proportions still did not deviate markedly from the training set proportions, although some deviations might be expected due to inherent cohort differences.

To evaluate the performance of the MDAhet classifier in the training and test cohorts we used several different measures and models of prognostic separation, depending on the variable of clinical outcome used. As binary outcome we used absence or presence of a disease-specific death event, or the surrogate-distant metastasis if the former was not available. Since this does not take time dependence of events into account, binary outcome was also used at four years after surgery adapting methods for time-dependent receiver operator curve (ROC) analysis 24. In addition, we considered continuous outcome in full stratified Cox-proportional hazard regression models, where stratification was performed on a per cohort basis to take inter-cohort differences in the types of survival data (ie, whether disease-specific survival or distant metastasis) into account.

Performance indicators based on the binary outcome measures are shown in Table 3. The most important performance indicator in our context is the NPV, since this represents the probability of correctly identifying a good prognosis patient. As shown, the NPV was very high with average values of 0.8 in the training sets and 0.96 in the test sets (range 0.85 to 1). Indeed, a significant improvement over simple predictions based on a priori known proportions was observed in all test sets (Table 3). In line with these results, sensitivity values were also very high with average values of 0.84 in training sets and 0.94 in test sets (range 0.76 to 1). Results evaluated at four years after surgery were, as expected, not markedly different, indicating that the prognostic classifier performs equally well in terms of short-term survival outcomes (Table 3).

Stratified Cox-regression models further confirmed the much better prognosis of the predicted subclass overexpressing the immune response-module relative to samples classified as poor prognosis (Table 4). Specifically, samples classified as good prognosis with overexpression of the immune response-module ('good-up' group) have less than half the risk of a poor outcome event (death or distant metastasis) relative to samples classified as poor prognosis, a result that we found to be independent of LN status and chemotherapy (Table 4). Note that four of the test cohorts were untreated (no chemotherapy) populations (Table 3), such as the training set itself, confirming the prognostic relevance of the classifier, and that chemotherapy itself was not prognostic in the two partially treated populations (Table 4).

Kaplan-Meier survival curves stratified according to the type of survival data (disease-specific death or distant metastasis) further confirmed the better prognosis of the predicted 'good-up' group (Figure 3). These survival curves further show that the classifier in the test sets is unable to discriminate the good prognosis samples that do not overexpress the immune response-module ('good-down') from the poor outcome samples. This result is expected since the seven-gene module is hypothesised to only identify a particular subgroup of good prognosis 5.

Since the maximum probability criterion assigns test samples to classes without regard to how large the maximal posterior class probabilites are, we tested the robustness of our results by only classifying samples passing a minimum probability threshold. For a probability threshold of 0.3 (already significant compared with the minimum possible maximal probability of 1/4 = 0.25), 94% of all test samples passed this threshold, indicating that our results are indeed robust. For a threshold of 0.4, we found 68%of samples were classifiable and results were still in line with those reported for the minimum threshold of 0.25 (data not shown).