One for All SAT › SAT notes › Problem-Solving and Data Analysis › Evaluating statistical claims: Observational studies and experiments
Evaluating statistical claims: Observational studies and experiments: SAT Math notes
These questions test whether you can judge what a survey or study really shows: which population its results apply to, whether its sample is biased, and what can and can't be concluded from it.
Study it with worked examples and 6 practice questionsPractise Evaluating statistical claims: Observational studies and experiments questions
What this topic tests
Every study starts with a sample. If the sample is chosen at random from a population, its results can be generalized to that population, and only that population. A random sample of fourth graders at one school tells you about the fourth graders at that school, not about the whole school or the whole county.
If the sample isn't random, or comes from a group that's likely to answer one way (dog owners asked about a dog park, families at a playground asked about children), the results are biased and can't be trusted to represent everyone.
Even a good random sample only gives an estimate. A different random sample would give a slightly different result, so a statement that a result must be exactly true for everyone is almost always wrong.
Key ideas
- Random selection lets you generalize from a sample to the population it was drawn from.
- The largest population you can generalize to is exactly the group the sample was randomly drawn from: no wider, and not only the people in the sample.
- Random assignment to groups in an experiment lets you conclude cause and effect. Without it, you can only claim an association.
- Samples where people choose themselves (an online poll) or where the researcher asks whoever is handy (families at a playground) are biased.
- A sample result is an estimate. Another random sample from the same population will probably give a slightly different number.
- Watch for the words must, exactly and all in the choices. They're usually too strong.
The 3 kinds of Evaluating statistical claims: Observational studies and experiments question
Sorted from every Evaluating statistical claims: Observational studies and experiments question in the site's bank: each kind keeps coming up, and has a method that works every time.
1. Largest population the results apply to
How to spot it. The question asks for the largest population (or group) the results can be generalized or applied to, or which inference about a group is appropriate. The choices are usually nested groups, from the people in the sample up to everyone in the country.
How to answer it:
- Find the sentence that says where the sample was randomly selected from.
- Copy that group exactly, with every limit it has: grade, school, region, age, species.
- Pick the choice that matches it. Reject anything wider (more schools, all ages, the whole state) and anything narrower (only the people sampled, or only part of the sample).
The trap. Two traps pull in opposite directions. Choices that are too narrow, like the 40 students surveyed or only the teams shown in a table, ignore the power of random sampling. Choices that are too broad, like all students at the school when only fourth graders were sampled, go beyond who could have been picked.
2. Spot the biased sample
How to spot it. A survey or poll is described, and the question asks what's wrong with it, or why its results may not represent the population. The sample was usually chosen in a way that favors one kind of answer.
How to answer it:
- Name the population the researcher wants to learn about.
- Name who was actually asked, and how they were chosen.
- Ask whether that group is random and representative, or whether it leans one way (dog owners, families at a playground, viewers who chose to answer).
- Pick the choice that says the sample isn't random or representative.
The trap. Blaming the wrong thing, like the sample size, the time allowed, or percentages that don't add to 100%. The real problem is who was asked. Asking more of the same people (more dog owners) wouldn't fix the bias.
3. What must be true from one sample
How to spot it. A random survey reports a percent, and you're asked which statements must be true: that the whole population has exactly that percent, or that another sample would give the same percent.
How to answer it:
- Remember that a random sample gives an estimate, not the exact population value.
- Another random sample from the same population will usually come out a little different, so a repeat can't be guaranteed to match.
- A sample from one population says nothing certain about a different population.
- Test each statement against the word must. If it could fail, it isn't a must.
The trap. Treating the sample percent as a fact about everyone. 78% satisfied in a good random sample is strong evidence that roughly 78% of the city is satisfied, but it doesn't make that exact figure certain.
Practise Evaluating statistical claims: Observational studies and experiments
On One for All SAT every question has a step-by-step solution, worked both by hand and in Desmos, the calculator built into the digital SAT, and the notes have worked examples of each kind of question, then practice questions you can check as you go. It's free, and there's nothing to sign up for.
Open the Evaluating statistical claims: Observational studies and experiments notes