A company finds that its hiring test barely predicts job performance among its employees. A university finds that admission scores barely predict grades among its students. Both conclude the test is nearly useless. Often the test is fine. The sample is the problem.
What range restriction is
A correlation describes how two variables move together across the range of values in the sample. If the sample only contains people from a narrow slice of one variable, there is less variation to work with, and the correlation shrinks, even when the underlying relationship is just as strong.
Employees were hired because they scored well on the test. Students were admitted because their scores were high. Nobody with a low score made it into the data set. Within the narrow band that remains, other factors (motivation, luck, the particular team or course) account for most of the differences, so the test looks weak.
A picture to keep in mind
Imagine a scatter plot of test scores against later performance for everyone who applied. It forms a clear upward-sloping cloud. Now cover the left half of the plot with your hand, keeping only high scorers. The remaining strip looks much flatter and rounder. The relationship didn't change; you just stopped looking at most of it.
Where it shows up
- Selection and admissions research: validating a test only on people who were selected by that test.
- Clinical studies: recruiting only patients within a narrow range of severity, then finding that severity "doesn't matter" for outcomes.
- Sports and expertise: among professional athletes, height may seem unrelated to success, because everyone in the sample is already tall for the general population.
- Measurement validation: testing a questionnaire in a homogeneous group and concluding it has poor validity.
What researchers do about it
- Report the sample's range and compare it with the population the conclusions are meant to cover.
- Apply corrections for range restriction when the population variance is known (standard formulas exist in psychometrics), and say clearly that a correction was applied.
- Use regression slopes alongside correlations. A slope is less affected by restriction on the predictor than a correlation is, although it is still affected by selection on the outcome.
- Design the study to include the full range when possible, for example by collecting data from applicants who were not selected.
The takeaway
A weak correlation inside a selected group does not mean the variables are weakly related in general. Before reading a correlation, ask who was allowed into the sample, and on what basis.
More research explainers: https://www.crushsci.com