WHAT is ANOVA? No, it is not an exploding star. It is a widely used method for testing the statistical significance of differences in the results of several trials, such as market tests of various designs of wallpaper or flavours of pudding. Like all statistical methods, it has its limitations. But provided those limitations are respected, it is a powerful and useful weapon in the never-ending battle against ignorance.
You may recall the story of Westhampton City Council (see Inside Science No. 82, 鈥淪tatistical sampling鈥�), which wanted to be sure that the quantity of bacteria along its beaches was below the European Approved level of 200 bacteria per unit volume. The council hired an analyst who tested 10 samples of water, getting results with a mean value of 194.8. At first sight this looked safe because it was below the limit of 200. But the council wanted to be at least 99 per cent certain that errors in sampling did not invalidate that conclusion. By applying a statistical technique called the t-test the council discovered that it could not be so certain of the water quality.
We pick up the story a few years on. Having improved the quality of sewerage treatment for outfalls in the area, the council again took samples. This time the mean was 165.3 which looked like an obvious improvement. But again the council was cautious and wanted to be at least 99 per cent certain that the situation really had got better.
Advertisement
Before it could carry out the necessary statistical test, the nearby resort of Easton-super-Mare put out a television advertisement announcing that tests on its beaches had produced a mean value of 148.4 for the bacterial count; clearly much better than Westhampton鈥檚.
Westhampton Council strongly suspected that this 10 per cent difference was due to chance variations in the samples taken and that it was not statistically significant. However, before running its own advertising campaign to counter Easton-super-Mare鈥檚 claims, Westhampton Council wanted to be very sure that its suspicions were correct.
Making comparisons
Over ANOVA again
WESTHAMPTON was aware that it could not repeat the t-test calculations because the nature of the problem was now different. The original problem involved comparing a single sample mean with some specified value. Both of the new problems that arose required the comparison of two different sample means. To demonstrate the improved quality of its beaches the council wished to compare the original sample mean of 194.8 with the new value of 165.3. To provide useful ammunition for the battle with Easton-super-Mare, Westhampton had to compare its own sample mean of 165.3 with its competitor鈥檚 value of 148.4. The question in both cases was whether the apparent differences were statistically significant.
Problems of this kind, in which the results of many different trials are compared with each other, are tackled using a technique known as analysis of variance, or ANOVA for short.
ANOVA has many important applications. Industrial trials often test several different designs of a product simultaneously. For example, Plodbury鈥檚 Sweets could decide to carry out a market survey of customers鈥� responses to 17 new flavours of Ploddo Bar with the intention of marketing only the three flavours that are most widely admired. And medical trials can involve testing a range of treatments to find out which perform better. In these cases the problem is determining the significance of the differences. Solving this problem could prevent Plodbury鈥檚 producing the wrong flavours of Ploddo Bar merely because of random effects in the market survey. Likewise, an effective medical treatment could be ignored because of chance effects in the sample of patients tested.
Like any statistical technique, ANOVA has limitations. In particular, it makes some assumptions about the statistical model that should be employed for the test data. For example, in each trial the data should be normally distributed (see Inside Science No. 74, 鈥淪tatistical modelling鈥�). If those assumptions are unfulfilled, the ANOVA results should not be taken seriously. A test for normality is given in Box 1.
Taking samples
t-test for two
BEFORE embarking on ANOVA we鈥檒l warm up with the Westhampton/Easton data. For these we can use a two-sample test. The same technique will also resolve Westhampton鈥檚 worries about the significance of its own improvement. But the question cannot be answered if all we know are the means of the two sets of data. The reason is illustrated in Figure 1 which shows that we also need to know the extent to which the sample data are spread out around those means.
In Figure 1 the sample data on the left are tightly clumped and their standard deviation is relatively small. Intuitively this suggests that the observed mean values are fairly accurate and that large deviations from those values are extremely unlikely to occur purely by chance. Therefore we conclude that the differences in the means are likely to be significant. By contrast, the data on the right are widely spread and the standard deviations are large. In this case the difference in means is quite likely to be due to chance.FIG-20337401.jpg
These arguments are qualitative: they do not tell us how likely or unlikely the observed differences in means might be. What is needed is a quantitative test which is provided by the t-statistic which we have already met in Inside Science No. 82.
Suppose that we are comparing two trials, one taken from a fixed population P1 and the other from a fixed population P2. (In our current example, P1 is the population of bacteria off the beaches of Westhamption and P2 is the population of bacteria off the beaches of Easton-super-Mare.) Assume first that we know the values of the two population means-that is, the mean
First we analyse the sample data to work out four quantities: 
n2, the standard deviation of the n2 sample values from P2. (The symbols are traditional; s12 is the sum of the square differences between the measured values for x1 and the sample mean. The factor n1 arises because the sample variance is s12/n1 and the sample standard deviation is the square root of this. The same applies for population P2.)
Having calculated these we form the two-sample t-statistic:
This may look a complicated formula but it is simpler than it seems. What it does is compare the difference between the sample means, 
1鈥�2. The expression in the denominator of the fraction merely adjusts the overall variance to a standard value.
We use the two-sample t-statistic in exactly the same way that we used the usual t-statistic in Inside Science No. 82. First we choose a confidence level for the degree of significance that we require. If we choose a 99 per cent confidence level then we want to be sure that the difference we are testing can arise by chance only with probability 0.01 or less; if we choose a 95 per cent confidence level then we want to be sure that the difference we are testing can arise by chance only with probability 0.05 or less, and so on. Then we use standard tables to look up what the value of the t-statistic should be at the confidence level we have chosen and compare this with the calculated value.
Two points should be borne in mind:
- Associated with the t-statistic is a number called its degrees of freedom. For a single sample this is one less than the sample size. For the two sample case it is the smaller of the numbers n1-1 and n2 -1.
- The theoretical probability distribution of the quantity t we have calculated is not precisely a t-distribution. However, the usual t-distribution is an excellent approximation, and if we use it we always err on the safe side. In other words, we can safely use the tables for the t-distribution even though they are not quite correct.
This is all very well when the true populations means 1 and 2 are known, but in many cases we do not know the population means. The dispute between Westhampton and Easton is one such case. When faced with this difficulty we calculate a different statistic which is also called t. But now we use the formula:
Again, this is not as nasty as it looks. The expression in the denominator is a way to standardise the overall variance and the numerator is the difference in the sample means. We use this expression to test the hypothesis that the (unknown) population means 1 and 2 are equal, just as for a standard t-test.
For example, suppose that Westhampton makes 10 measurements: 131, 172, 169, 144, 153, 167, 168, 190, 176, 183; whereas Easton-super-Mare makes only eight: 142, 160, 159, 152, 143, 149, 137, 145.
Then:
The number of degrees of freedom is the smaller of 10 鈥� 1 = 9 and 8 鈥� 1 = 7; that is, 7. The value required for the t-statistic with 7 degrees of freedom at the 99 per cent confidence level is 2.998 or higher. So we conclude that the difference in means is not statistically significant at the 99 per cent level. Therefore Westhampton can confidently assert that Easton-super-Mare鈥檚 claims are probably due to random sampling errors. In fact, the value required for the t-statistic with 7 degrees of freedom at the 80 per cent confidence level is 0.896, close to the calculated value of 0.91. So there is about an 80 per cent probability that the results of the two tests could have come about from two populations with identical means.
Statistical models
Tables of data
WHEN we wish to compare the results of more than two trials we use the ANOVA technique. We shall introduce this by way of an example. Pfertilizer Inc. has conducted a series of tests on four different types of potting compost for growing miniature daisies. Each consists of a standard potting compost that has been treated with various chemicals to make it suitable for daisies. To test the effectiveness of these treatments Pfertilizer has grown samples of plants using each type of compost. Figure 3 shows the height in centimetres of the resulting plants.FIG-20337403.jpg
The mean heights of the four trials of daisy-growing compost are 12 for Gro-Mo, 17 for Dasigro, 16 for Lazi-Dazi and 15 for Compodaze. Pfertilizer Inc. wants to know whether the apparent inferiority of Gro-Mo and the apparent superiority of Dasigro are statistically significant.
The first step is to formulate a reasonable statistical model for the variability that is apparent in the data. To see how this is done, we first tabulate the data systematically. The symbols yij represent the jth data value in trial i. There are four trials and in general the number of trials is denoted by the symbol k, so here k = 4.
The number of data values in trial i is written ni, so: n1 = 5; n2 = 4; n3 = 7; n4 = 6.
Next we calculate the trial means or sample means:
These are the means of the data for trial i, in each of the cases i = 1, 2, 3, 4, and here they are:
The grand mean is just the mean of all the data values,
or in 
-notation:
We also calculate the trial sums of squares representing the total square deviation of each data value from the corresponding mean: s12 = 38; s22 = 30; s32 = 12; s42 = 14. The standard deviation of the data values for trial i is then si/ni.FIG-mg203374M9.gif
Our statistical model assumes that the deviation between the heights of plants grown with differently treated composts can be split into two different components: the overall deviation due to the treatment of the compost and random deviation within a given trial. The deviation due to treatment i is the difference between the mean in trial i and the grand mean:
The deviation due to random errors is the difference between the observed data and the appropriate trial mean: which is usually called the residual. Note that:
Constructing arrays
Observed deviations
WE CAN tabulate all of these values in four numerical arrays, like this:
where figures in a corresponding position in each array refer to a given item of data. The first array shows the data listed in rows that correspond to the four trials. The second array consists solely of the grand mean. The third array shows the deviations due to treatment and the final array shows the residuals.FIG-mg203374M11.gif
If the treatments have not had any genuine effect, we would expect the deviations due to treatment (the entries of the third array) to be close to zero. The overall variation due to differences in the treatment means can be measured by forming the sum of squares of all these values, , known as the treatment sum of squares (TSS). Here its value is: 5(-3)2 + 4(2)2 + 7(1)2 + 6(0)2 = 68.FIG-mg203374M14.gif
Similarly, the overall variation due to random errors is measured by the sum of squares of the residuals, or error sum of squares (ESS):
This is the sum of the squares of the numbers in the final array, which is equal to 94.
We can also compute the total square deviation of the original data values from the grand mean, the total sum of squares:
This is the sum of the squares of the numbers in the first array which is equal to 162.
Note that 162 = 68+94, so the total sum of squares is the sum of the treatment sum of squares and the error sum of squares. This is not a coincidence and an algebraic calculation proves that it is generally valid.
Having calculated these statistical quantities, Pfertilizer is finally in a position to decide whether the observed deviations in the trial means are significant. It does this using the technique of hypothesis testing (see Inside Science No. 67). It formulates the null hypothesis that the differences are due to random sampling errors and that all four trials arise from the same statistical population. The alternative hypothesis is that they do not.
Finally, Pfertiliser must calculate the F-statistic:
where n = n1 +鈥� + n4 (so n = 22 in this example). Notice that although there are four trials the treatment 鈥渕ean鈥� square is the TSS divided by 3, not by 4. The reason for this is that the number of degrees of freedom in the trials is 3 (and in general for k trials it is k-1). This is because for a given grand mean, only three of the trial means are independent of each other. Similarly, the factor n-4 that occurs in the denominator for the ESS (which for k trials becomes n-k) is the number of degrees of freedom of the ESS.
Naturally, the F-statistic has an F-distribution which can be found in standard statistical tables. The F-distribution has two different kinds of degrees of freedom (k-1, n-k), so be careful to use the right section of the tables.
For the Pfertilizer problem the appropriate degees of freedom are (3, 18), and we have TSS = 68; ESS = 94; TSS/3 = 22.67; ESS/18 = 5.22; and F = 22.67/5.22 = 4.34.
Statistical tables for F show that at a 95 per cent confidence level we require an F-value of at least 3.16 to conclude that the apparent difference in means is significant. Since the calculated F-value is larger than 3.16 we conclude that there is a significant difference in the means since the probability of the difference arising by chance is less than 0.05.
For a general number k of trials, we proceed in exactly the same manner. But this time there are k trial sums of squares s12,鈥�,蝉k2 and n = n1 + 鈥� + nk. As already noted, the appropriate degrees of freedom for the F-statistic are (k-1, n-k).
This is just the beginning of an extensive battery of closely related statistical tests that can be used to decide whether differences in the results obtained in multiple trials are statistically significant. These methods are widely used in industrial quality control and process control, in medical trials and in the assessment of government statistics. ANOVA may not be an exploding star, but it still puts many things in a new light.
* * *
1: Normal quantile plots
ANOVA is based on the assumption that the data are selected from approximately normal distributions. You should not use ANOVA on data without verifying this assumption because the results may be meaningless if it is violated.
A common way to verify normality of a distribution is to form a normal quantile plot. In essence this arranges the data in increasing order and then plots each data value at a position that corresponds to its ideal position in a normal distribution. A quantile of a probability or frequency distribution is a number that lies between 0 and 1 and is denoted by x. The xth quantile is the value of the random variable z for which exactly the fraction x of the distribution takes the value y or less. For a continuous probability distribution, which has total area 1, this means that the area under the probability density function that lies to the left of the vertical line at value z should be exactly x (see Figure 2). The median is the 0.5 quantile and the first and third quantiles are the 0.25 and 0.75 quantiles (see Inside Science No. 61).FIG-20337402.jpg
Quantiles for the standard normal distribution can be calculated by working backwards from a table of the values of this distribution. Consider the Westhampton data: 131, 172, 169, 144, 153, 167, 168, 190, 176, 183. Are these normally distributed?
The first step is to arrange the data in increasing numerical order: 131, 144, 153, 167, 168, 169, 172, 176, 183, 190. We have 10 data points which we can consider as lying at the quantiles 0.1, 0.2, 0.3, and so on up to 1.0. The first data value 131 lies at the 0.1 quantile. This is the value z for the normal distribution such that 0.1 of the area under the curve lies to the left. We search through a table of standard normal probabilities to find which z-value gives an area of 0.1 and we obtain z = -1.28. The second data value 144 lies at the 0.2 quantile, for which the z-value is -0.84, and so on. In this way we build up a table:
We then plot the data (vertically) against the z-value (horizontally), as in Figure 4 on page 4. If the data are normally distributed, all points in the plot should lie on a straight line. If the plot deviates noticeably from a straight line, the distribution is not normal and we should not use ANOVA. In this case, the line is reasonably straight so the data are suitable for ANOVA.FIG-20337404.jpg
* * *
2: ANOVA tables
IT IS usual to record the relevant data in the form of an ANOVA table,
which in the case of Pfertilizer鈥檚 test takes the following form:
In general, an ANOVA table will look like this:
