鈥淓IGHT out of ten cats prefer new improved Katphood with nourishing vitafibres.鈥�
How often have you seen a claim of this kind and wondered what it really meant? How many cats took part in the trials, for example? Was the sample big enough to give accurate results? Taken literally, the claim could mean that the trial of new Katphood involved ten cats and was eaten by eight of them. The manufacturer could have tested one hundred cats of which eighty showed a preference. What the manufacturer wants you to believe is that 80 per cent of all cats would prefer new Katphood, if tested 鈥� and that your cat could be one of them.
The cat food company cannot have tested its product on every cat in the world. Instead, the company scientists almost certainly used a statistical technique known as sampling. They selected a small sample 鈥� one hundred cats chosen at random, perhaps 鈥� and tested those. Then they assumed that the sample was representative of the entire cat population so that the same figures would apply in general.
Advertisement
Taking a sample
Opinion polls
THIS technique raises all sorts of questions. What do we mean by a random sample? How big must the sample be to give accurate results? How safe is it to assume that the sample is truly representative? Statisticians have developed an extensive theory of sampling to answer such questions.
One of the most high-profile uses for sampling is in carrying out opinion polls. Newspapers or other organisations commission various specialist companies to sample public opinion on some question of national importance such as which party they would vote for in a general election, or whether they think that Britain should agree to a common European currency, or whether fox-hunting should be banned. Because it is impractical to ask such questions to the entire population, the pollsters select a sample of people and ask them instead.
The typical size of such a sample is surprisingly small. For many purposes, a sample of about 500 people is entirely satisfactory. For a really credible sample, the figure might increase to 1000 or so. Is it really possible to work out what 50 million people think by asking only 1000? If we are careful to avoid some simple pitfalls, the answer is yes.
These pitfalls are various kinds of bias in our choice of people in the sample. For example, people who live in cities are more likely to want fox-hunting banned than people who live in the country who tend to be familiar with the bad effects that foxes can have. Taking this point to extremes, we would get very different answers from a sample containing only members of an animal rights society compared with a sample of members of an association of chicken breeders. (Here we are not discussing whose opinion is right, just what the opinions are.)
There are more subtle ways in which we can choose a biased sample inadvertently. A good example occurred in the early days of telephone polling when pollsters phoned a sample of people and asked them their opinion on various issues. The sample included only people who owned telephones which were expensive at that time. Consequently, the sample was biased towards those who were well off. This kind of bias can make a very big difference to the response resulting in a poor estimate of opinion in the country as a whole. A similar problem occurs with postal questionnaires. Some people cannot be bothered to fill them in or return them. So the answers come only from those people who care enough about the issues to devote time to the questions.
We can deal with this kind of problem by setting up a procedure for choosing people that does not introduce such biases. For example, suppose that we want to obtain a truly representative sample of 100 citizens from Westhampton, a seaside town with 100 000 inhabitants. We start with the idea of picking 100 people at random from the phone book but then realise that we will miss out people who do not have phones. Instead, we could try dividing up a map of the town into ten regions and then sending 10 people around on bicycles to knock on 10 doors chosen at random from their allotted region. That does not work either. Some regions in the town could have many more people in them than others and it would be unrepresentative to choose 10 people from a housing estate of 5000 and another 10 from an expensive part of town that contains only 200 people.
One way to tackle the problem is to work out a profile of the number and types of people in the town. We might divide the citizens of Westhampton into, say, ten age groups of 0 to 9, 10 to 19 and so on up to 90 to 99. Any one aged 100 or over could be included in this final group. We could also classify them by sex and by their level of income 鈥� say five groups ranging from 拢0 to 拢4999, 拢5000 to 拢9999, 拢10000 to 拢19999, 拢20000 to 拢49999 and 拢50 000 or more. If anything else seems likely to influence their decision we must take this into account as well.
Then we randomly choose our sample of people to have the same composition as the whole population. For example, if there are 7000 women in Westhampton who are aged betwen 20 and 29 and have annual incomes ranging between 拢20 000 and 拢49 999, then the sample should include exactly seven of them chosen at random. If there are 18 000 men aged between 50 and 59 with incomes in the range 拢10 000 to 拢19 999 then the sample should include 18 of them. The sample size is one thousandth of the total population so each category of people should be represented proportionately.
Practical problems
Computer databases
MANY practical problems remain. Ideally, we will need a complete list of all 100 000 inhabitants with the appropriate data on age, sex and income. We must be able to pick out each group, count the number of people in that group and choose the appropriate number from it at random. Somebody has to visit their homes repeatedly until they are in and then persuade them to answer the questionnaire. A computerised database of the town鈥檚 population would help but, instead, we will probably have to compromise and make educated guesses about them based on data on the entire nation obtained from the most recent census. What if a group contains 14 567 people requiring us to have 14.567 of them in our sample? We would have to have rules for rounding up or down to whole numbers. Some people may refuse to answer so we will need reserves in each group. It can become very messy. Nevertheless, this description should give you a general idea of the principles involved.
The way questions are asked can also influence people鈥檚 answers. For example, the question 鈥淎re you in favour of more public services?鈥� might get a very different answer from 鈥淒o you think that income tax ought to go up?鈥� Many people would say yes to the first question but no to the second. A lobby group favouring more public services would ask the first question. People opposed to increasing public services would observe that this requires higher taxes and could ask the second one. To a certain extent, bias of this kind can be avoided by not asking leading questions.
Some types of bias are very hard to control. For example, in the most recent general election in Britain, most polls overestimated the Labour vote by about 5 per cent. This led to the prediction of a narrow Labour victory but, in the event, the Conservatives won instead. Afterwards, the error was traced to two possible sources: either that many voters changed their mind at the last minute or that they did not tell the pollsters the truth. Even if these sources of bias could have been foreseen, it would have been difficult to construct a sampling technique that avoided them. 鈥淎re you telling the truth?鈥� is not a helpful question when determining someone鈥檚 veracity.
Even if we choose the sample that avoids all the sources of bias that we can think of, we will occasionally get wrong answers. This happens when the sample fails to be representative by pure bad luck, just as you might toss a coin 10 times in a row and get heads every time. This is not representative of a typical sequence of tosses 鈥� which should contain about five heads and five tails 鈥� but it will happen once in every 1024 trials, on average.
Fact or fiction
Statistical significance
FOR this reason, statisticians have worked out ways of quantifying the significance of results obtained by sampling. The best way to do this is to state the probability that the results from the full population would be significantly different from those from the sample. For example, we might expect that 95 per cent of all random samples give results within 2 per cent of the figures found in our survey. In practice, there are many ways to provide this kind of information. However, all of them are relatively technical so newspapers tend to print the result of a poll without any statistical assessment of the likelihood that it could be wrong.
Specific data
Sizes of gerbils
THIS is all rather general, so let us consider a specific case: estimating the average weight of a gerbil. Imagine a population of gerbils that is too large to measure directly. Instead we select a sample from this population that is small enough to study in detail. Then we measure the weight of each gerbil in this sample 鈥� this is known as the random variable. Finally, we work out a statistic, in this case the mean or average weight.
Suppose the sample contains 10 gerbils with weights (in grammes) of: 560, 825, 934, 864, 899, 538, 745, 727, 553 and 975. The mean is 762. How certain can we be that this represents the average weight of the entire gerbil population? Suppose we have accidentally sampled too many fat gerbils or too many thin ones?
To answer this question, we have to rely on the statistical theory of sampling. In a real situation we can usually take only one sample, perhaps because it is too expensive to take more. But to understand how the theory works, let us assume that we can repeat the experiment many times. Suppose we take 100 random samples of 10 gerbils and work out the mean for each. On the first occasion above, it was 762, the next time 743 and so on. Rather than list all 100 samples, we can draw a picture known as a histogram showing how frequently any given value occurs. For example, if a value between 740 and 749 occurs on 8 occasions, we draw a bar of height 8 above the range of values 740-749. The result is shown in Figure 1.
The histogram illustrates a number of points. For example, different samples can give different means. One sample gave a mean in the range 640-649 and two were in the range 860-869. Despite this variability, a good sign is that most of the samples produce means near the middle of the histogram. In fact, half of them were between 710 and 780.
Now let us compare the figures from random samples with the mean of the entire gerbil population. Rounded to the nearest integer, this is 750. I know that because I designed the numbers on that basis. My imaginary gerbil population had weights that were uniformly distributed between 500 and 999. In other words, the probability that a gerbil would have any particular weight in that range was the same, namely 1/500.
This numerical experiment tells us that half the samples give a mean that is within 40 of the true value. This represents a maximum error of 5.3 per cent. So in this example there is a good chance that the sample mean is not far wrong.
Random samples
Central limit theorem
THAT鈥橲 not a very mathematical statement. How good is a chance? How wrong is 鈥渘ot far鈥�? The answer to these questions can be derived from a very beautiful theorem, called the central limit theorem. This is linked to a mathematical formula or function called a probability distribution which gives the probability that a particular observation takes a given value (see Inside Science No. 74, 鈥淪tatistical modelling鈥�). It says that the probability that the observation takes a value x is given by a function F(x). In the case of a continuous probability distribution, where x can be any number in some range, this formula is called a probability density function f(x). The probability that the measurement x lies in the range a 
For the gerbil population, the probability density function f(x) satisfies the equation:
f(x) = 1 if 500
f(x) = 0 otherwise.
A graph showing these equations appears in Figure 3. This is a mathematical way of saying that the probability of any given value between 500 and 999 was the same. (I cheated slightly by using 999 instead of 999.999 鈥�).
The histogram in Figure 1 can also be thought of as an approximation to a probability density function, for the random variable 鈥渧alue of the sample mean鈥�, usually denoted by the img . Statisticians call this an approximation to the sampling distribution of the mean. If instead of the rectangular bars you were to draw a smooth curve then the area under the curve would determine the probability of getting a sample mean in a chosen range. What the central limit theorem tells us is what shape this ideal curve should be if we take the mean of a large number of samples.
In Inside Science No. 74 we discovered that a normal distribution is a probability density function with the symmetric bell-shaped curve shown in Figure 4. If the mean of this probability density function is 
Large samples
Z-statistic
THE CENTRAL limit theorem tells us that when the size of the sample is large, the distribution of the sample mean is approximately normal. It works in all cases not just those like the gerbil example where the distribution is uniform. Moreover, it is a theorem with a proof and not just a numerical experiment like a histogram. In fact, if the sample size is n, then the distribution of the sample mean is very close to N(
Another way to say this is to calculate the standardised statistic:
which is called the Z-statistic. The distribution of Z is N(0,1) which is the standard normal distribution with mean 0 and a standard deviation of 1.
The great beauty of the central limit theorem is that, to a very good approximation, the distribution of the sample mean does not depend on the distribution of the corresponding random variable for the original population. In our example, the weight X of the gerbils was uniformly distributed over the range 500
To test hypotheses about a population mean we can use the Z-statistic. For example, the manufacturer of Katphood has carried out trials on cats that eat its product to determine how shiny their fur is. The shininess of cat fur is measured in units known as arthurs. When large numbers of cats are fed old Katphood without nourishing vitafibres, the mean value of their fur shininess is 240 arthurs and the standard deviation is 20 arthurs. The manufacturer plans to test a random sample of 25 cats using new improved Katphood with nourishing vitafibres to find out if their fur becomes shinier. What value of the mean shininess of this sample will let it be 95 per cent confident that new improved Katphood with nourishing vitafibres makes cats鈥� fur shinier?
This is an example of hypothesis testing (Inside Science No 67, 鈥淭esting Hypotheses鈥�). The upshot of that technique is that the observed mean must have an 0.05 probability of being greater than some threshold, denoted by the letter c. (0.05 is 5 per cent or 100 per cent minus 95 per cent.) How they can do this is explained in 鈥淒oes new Katphood make cats鈥� fur shine?鈥� The box shows that if the mean shininess of the sample of 25 cats is at least 246.6 arthurs, then the manufacturer can be 95 per cent certain that if all cats were fed new Katphood, the mean shininess would be higher than when they were fed on old Katphood.FIG-mg198175M1.gif
If you used a smaller sample, say 9 cats, then
Even if we don鈥檛 know the standard deviation for the entire population, we can replace it in the Z-statistic by the standard deviation of our sample. This makes no serious difference to the answer, provided the sample is large.
What if we can only take small samples? It turns out that we can follow a similar approach but this time it works only if the population distribution is normal. The sample statistic that we should use in this case is called the t-statistic:
Although this looks very similar to the Z-statistic, there is one significant difference. Instead of the true standard deviation of the entire population, we use the standard deviations of the sample. Moreover, the distribution of t is no longer normal and depends on the sample size n. It is called Student鈥檚 t-distribution with n-1 degrees of freedom. (鈥淪tudent鈥� was the pseudonym of W S. Gosset, a British chemist who dabbled in statistics and published his ideas in 1908.) The number of degrees of freedom for the t-statistic is the sample size minus one. We can find the t-distribution in standard statistical tables and we use it just like the N (0,1) distribution.
Level of confidence
Hypothesis testing
鈥淲ater quality near Westhampton鈥檚 beaches鈥� shows how to use the t-test to assess how safe it is to swim near a beach. Even though the level of harmful bacteria in the sample is below the permitted safety level, the t-test shows that Westhampton Council may have something to worry about. The reason is that further samples might produce a larger mean and the t-test shows that this possibility is sufficiently likely that it cannot be rejected. Here we see how sampling theory lets us avoid making the obvious but wrong deduction from a random sample which is what sampling theory is all about.
1: Does new Katphood make cats鈥� fur shine?
SUPPOSE that is the mean shininess (in arthurs) of a random sample of 25 cats fed on new improved Katphood with nourishing vitafibres. The central limit theorem implies that, to a good approximation, has a normal distribution denoted by N(
N(240,20/25) =
N(240,20/5) = N(240,4)
We know from tables that if some statistic Y has normal distribution N(0,1) then the corresponding value of the threshold c is 1.65. This is written as:
P(Y > 1.65) = 0.05
This is why we use the standardised statistic Z instead of the mean. We have:
so: P(Z > 1.65) = 0.05.FIG-mg198175M1.gif
Therefore we want:
so that:
,
or finally:
.
So the value of c that we need is 246.6.
2: Water quality near Westhampton鈥檚 beaches
WESTHAMPTON CITY COUNCIL wants to test whether the bacteria level along the city鈥檚 beaches is below the European Approved Level of 200 bacteria per unit volume. It hires an analyst, who takes 10 samples, getting the results:
175, 180, 184, 190, 193, 196, 198, 207, 210, 215.
The mean is 194.8 which satisfies the criteria of being less than 200. But the council wants to be 99 per cent sure that the actual value is significantly lower than the permitted level.
Letting denote the actual mean level of bacteria, then the City Council must compare (at the 99 per cent confidence level) the following two hypotheses:
The null hypothesis = 200
The alternative hypothesis
The Council assumes that the bacteria levels in the sea are normally distributed which is perfectly reasonable in this case and calculates the t-statistic:
Here the sample mean is 194.8, the sample standard deviation is s = 13.14, and the number of degrees of freedom is n-1 = 10-1 = 9. The reason for the minus one is that the mean value is effectively fixed (to zero) by the definition of t. So only nine values can be independently changed 鈥� the tenth has to make the mean come out to zero.
The value of t is:
According to the statistical tables, at the 99 per cent confidence level the null hypothesis should in fact be rejected if t -2.821. Since t = -1.25 is larger than -2.821, the null hypothesis should not be rejected. In other words, the Council cannot be 99 per cent confident that the bacteria levels are safe.