With Census 2020 coming around, the topic of survey bias will certainly arise. Drafting neutral surveys free of bias requires understanding in several disciplines from math, to language, psychology, demographics, and a quite a bit of experience & judgement. Here are some of the more obvious forms.

Sample Bias

People living in the same neighborhoods have some common demographics and common opinions on certain topics. This also applies in the virtual world: people who visit certain web sites (say, New York Times, Wired, and Wall Street Journal).

Sometimes sample bias can be unintentional and subtle. The people you surveyed had something in common that you didn’t know about.

Framing Effect Bias

People respond differently to questions depending on how you ask them or “frame” the question. This is one of the most important biases.

For example, 93% of students registered early when a late penalty fee was assessed. But only 67% registered early when the fee was called a discount for early registration.

Another example: suppose 600 people have a deadly disease. Treatment A is predicted to result in 400 deaths. Treatment B is 33% likely to have no deaths, but 67% likely for all 600 to die.

When framed positively: A saves 200 lives, and B has a 33% chance of saving all 600, and 67% chance to save nobody.
Here, A was preferred by 72% of people.

When framed negatively: With A, 400 people will die. B has a 33% chance that nobody will die, and a 67% chance that all 600 die.
Here, A was preferred by 22% of people.

In the long term, outcomes from A and B are the same. Yet how the question is framed made a huge difference in which people preferred.

Response (and non-Response) Bias

This is similar to sample bias. Different people have different rates of response to your survey. Here, you can get burned either way. If you sample every group at the same rate, the uneven response rates can bias your data. If you sample groups at different rates, you can introduce a new bias. Eliminating this kind of bias requires measuring the different response rates and carefully targeting your sampling.

Question Order Bias

The answers people give to early questions influence how they answer later questions. Thus, questions can be ordered to lead people to answer later questions in certain ways. In multiple choice surveys, this also applies to the order in which each question’s potential answers are provided.