Back to: Assess for Risk of Bias

If we were to simplify the process for assessing a study’s risk of bias, we’d say it boils down to answering a series of increasingly complex questions.

All risk of bias assessments start by asking:

1. Did the study’s methods, execution, and analysis follow basic steps to avoid bias?

Some researchers stop here and evaluate the studies in their systematic review based solely on how the studies were designed and conducted.

Other researchers go a step further and ask:

2. Are potential problems with the way the studies were designed and conducted likely to have affected the results?

In this case, systematic reviewers take a deeper look at the problems with the methods and decide whether they might have led to misleading results in the study.

Answering this second question adds more subjectivity to the assessment, but it also better characterizes the possibility of bias.  

In rare cases, researchers take one more step and ask:
3. If the methods affected the results, by how much? And can the results be fixed?

In this step, researchers make their best “guesstimate” about how much the flawed methods affected the results, based on statistical models. To build these models, systematic reviewers must make a series of assumptions and must  have the right kind of data.

In this course, we will focus on the first two questions. Techniques for answering Question 3 are beyond the scope of this course.

For an example of how tricky and subjective this work is, let’s take a look at risk of bias assessments in action.