Bias in Data Collection
Section: Statistics | Syllabus: Cambridge Lower Secondary Checkpoint Mathematics (0862)
What is Bias?
Bias is a systematic error in data collection or sampling that consistently skews results in one direction, making them unrepresentative of the true population - unlike random variation, which isn't repeatable or one-directional.
A biased sample skews toward one part of the population; a representative sample reflects it fairly
Sources of Bias in Sampling
- Sampling frame bias: the list used to select a sample doesn't include everyone in the population.
- Self-selection (voluntary response) bias: people choose for themselves whether to take part, e.g. online polls.
- Undercoverage: certain groups within the population are left out of the sampling process.
- Non-random (convenience) selection: picking whichever individuals are easiest, nearest, or most extreme to measure, instead of choosing genuinely randomly.
-
Question: A researcher wants the average screen time of all teenagers in a city, but only surveys students at one school. Identify the source of bias.
- Answer: Sampling frame bias / undercoverage - students at one school may not represent the whole city's teenagers (different backgrounds, school policies, etc.)
-
Question: To estimate the average height of sunflowers in a large field, a gardener picks a random 1 m² patch, then measures only the 5 tallest sunflowers within that patch. Identify the source of bias.
- Answer: Non-random (convenience) selection - even though the patch was chosen randomly, deliberately picking the TALLEST sunflowers within it (rather than a genuinely random selection from the patch) will bias the average height upward
Sample Size and Reliability
A sample that's too SMALL can give unreliable results even if it was chosen fairly - it simply doesn't contain enough data to represent a large population confidently. This is a separate issue from bias, which is about HOW a sample is chosen rather than how big it is.
-
Question: To estimate the average height of 1000 students at a school (aged 4 to 18), a researcher measures just 5 students, all from one 18-year-olds' class. Give two separate reasons why this sample is unreliable.
- Reason 1 (sample size): 5 students out of 1000 is far too small a sample to represent the whole school reliably
- Reason 2 (bias): sampling only 18-year-olds ignores the full 4-18 age range, and older students are likely taller than average - this is sampling frame bias
- Answer: Both the small sample size AND the narrow age range make this sample unreliable, for two separate reasons
Sources of Bias in Data Collection
- Leading questions: phrased to encourage a particular answer.
- Question wording/ambiguity: unclear questions leading to inconsistent answers.
- Non-response bias: people who don't respond may differ systematically from those who do.
- Observer bias: the person collecting data influences the results.
- Timing/location bias: when or where data is collected affects who is included.
-
Question: A survey asks, "Do you agree that our excellent new product is worth the price?" Identify the issue and suggest an improvement.
- Issue: This is a leading question - "excellent" nudges the respondent toward agreeing
- Answer: Rephrase neutrally: "Do you think the new product is worth the price?"
Identifying Further Questions to Ask
After spotting a potential issue, ask: Who was included or excluded? How were participants selected? Was the wording neutral? When and where was data collected? Could non-respondents differ from respondents?
-
Question: A company claims "90% of customers love our product" based on reviews left on their website. What further questions should be asked before trusting this claim?
- How many total customers were there, and what proportion left a review (the response rate)?
- Are customers who leave reviews representative of all customers, or mostly very satisfied or unsatisfied ones (self-selection bias)?
- Were any negative reviews removed or filtered out?
Real-World Applications
Recognising bias matters across many fields:
- Political polling: avoiding biased samples that skew election predictions.
- Product reviews: recognising self-selection bias in online reviews.
- Scientific research: designing unbiased experiments and surveys.
- Journalism: evaluating whether survey-based news stories are trustworthy.
- Market research: ensuring feedback represents the whole customer base.
Exam Tips
Common Mistakes
MistakeAssuming a large sample size automatically avoids bias
Fixbias is about HOW a sample is chosen, not how big it is - a huge biased sample stays unrepresentative
MistakeNot recognising leading or loaded question wording
Fixcheck whether a question's wording nudges the respondent toward a particular answer
MistakeIgnoring non-response bias when a survey has a low response rate
Fixconsider whether people who didn't respond might differ from those who did
MistakeConfusing "bias" with a simple mistake or random error
Fixbias is a systematic, repeated, one-directional issue, not a one-off random error
MistakeTrusting a claim based on self-selected data without questioning who chose to respond
Fixalways ask who was included, and who might have been left out or self-selected
MistakeAssuming a sample is unbiased just because the general area or group it came from was chosen randomly
Fixcheck how individuals were picked WITHIN that area or group too - picking only the most extreme or convenient ones still biases the result
MistakeOnly giving one reason when a question asks for two separate issues with a sample
Fixsample size and bias are different, separate issues - a small sample can be unreliable even without being biased, and vice versa
For Exams
- Learn key bias types: sampling frame bias, self-selection bias, undercoverage, non-random selection, leading questions, non-response bias, observer bias, timing/location bias.
- Describe HOW a method could skew results, not just that it's "bad."
- Suggest a specific improvement, not just "make it better."
- Practise writing follow-up questions: who, how, when, where was the data collected?
- If asked for TWO reasons, check whether sample size and bias each give you a separate, distinct point.
Interactive revision notes, videos and practice questions load below.