11.4 Sample size

Let’s assume you have found a representative sampling frame, and that you are using one of the probability sampling approaches we reviewed in section 11.2. That should help you recruit a representative sample, but how many people do you need to recruit into your sample? While your quantitative research question would likely benefit from thousands of respondents, that may not always be feasible. Make sure you note any limitations in your final article.

As a rule of thumb, you would perfer a larger sample over a smaller sample. But why? Let’s think about an example you probably know well. Have you ever watched the TV show Family Feud? Each question the host reads off starts with, “we asked 100 people…” Believe it or not, Family Feud uses simple random sampling to conduct their surveys on the American public. Part of the challenge on Family Feud is that people can usually guess the most popular answers, but those answers that only a few people chose are much harder. They seem bizarre, and are more difficult to guess. That’s because 100 people is not a lot of people to sample. Essentially, Family Feud is trying to measure what the answer is for all 327 million people in the United States by asking 100 of them. As a result, the weird and idiosyncratic responses of a few people are likely to remain on the board as answers, and contestants have to guess answers fewer and fewer people in the sample provided. In a larger sample, the oddball answers would likely fade away and only the most popular answers would be represented on the game show’s board.

So what is the right number? Theoretically, we could gradually increase the sample size so that the sample approaches closer and closer to the total size of the population (Bhattacherjeee, 2012).^[1] But as we’ve talked about, it is not feasible to sample everyone.

How do we find that middle ground? To answer this, we need to understand the sampling distribution. You can see this online textbook for more information on sampling distributions and you will learn more about sampling distributions in your statistics courses. Imagine in your agency’s survey of the community, you took three different probability samples from your community, and for each sample, you measured whether people experienced domestic violence. If each random sample was truly representative of the population, then your rate of domestic violence from the three random samples would be the same and equal to the true value in the population. But this is extremely unlikely, given that each random sample will likely constitute a different subset of the population, and hence, the rate of domestic violence you measure may be different from sample to sample.

Think about the sample you collect as existing on a distribution of infinite possible samples. Most samples you collect will have characteristics close to the population’s average, but many will not. The degree to which they differ is associated with how much the subject you are sampling about varies in the population. If you are sampling from a fairly homogenous population, then there won’t be much difference between the samples’ characteristics and the population’s. But if the population has a very wide range of diversity in its characteristics, then each sample may differ more from the population.

The difference between the characteristics we find in our sample and the characteristics in our overall population is called the sampling error. We’ll talk more about sampling error in the next section. The nature of sampling is that there will always be some degree of sampling error unless the population has no variation on the variables of interest, which is almost never the case in human populations.  One way to minimize sampling error is to increase the number of participants in your sample.

Increasing the number of people in your sample also increases your study’s predictive power, or the odds you will detect a significant association between variables when one is truly present in your sample. It is important to conduct a power analysis to determine the appropriate sample size for your project. You can follow this excellent video series from the Center for Open Science on how to conduct power analyses using free statistics software. A faculty members who teaches research or statistics or your university/school statistical consultant could check your work. You may be surprised to find out that there is a point at which you adding more people to your sample will not make your study any better.

Another note regarding sample size has to do with the statistical tests you plan to use to analyze your data. Some statistical tests have a minimum sample size requirements in order to conduct the analysis. You will complete a data analysis plan before you begin your project and start sampling, so you can always increase the number of participants you plan to recruit based on that plan.