UW Interactive Data Lab
IDL logo

Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths

Alex Kale, Matthew Kay, Jessica Hullman. ACM Human Factors in Computing Systems (CHI), 2019
Figure for Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
A pictorial representation of strategies used to navigate the garden of forking paths. The diagram at the top depicts observed temporal orderings of these strategies, e.g., multiplexing across choices in a loop of acknowledgement and reduction.
Materials
Abstract
To make evidence-based recommendations to decision-makers, researchers conducting systematic reviews and meta-analyses must navigate a garden of forking paths: a series of analytical decision-points, each of which has the potential to influence findings. To identify challenges and opportunities related to designing systems to help researchers manage uncertainty around which of multiple analyses is best, we interviewed 11 professional researchers who conduct research synthesis to inform decision-making within three organizations. We conducted a qualitative analysis identifying 480 analytical decisions made by researchers throughout the scientific process. We present descriptions of current practices in applied research synthesis and corresponding design challenges: making it more feasible for researchers to try and compare analyses, shifting researchers' attention from rationales for decisions to impacts on results, and supporting communication techniques that acknowledge decision-makers' aversions to uncertainty. We identify opportunities to design systems which help researchers explore, reason about, and communicate uncertainty in decision-making about possible analyses in research synthesis.
BibTeX
@inproceedings{2019-uncertainty-research-synthesis,
  title = {Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths},
  author = {Kale, Alex AND Kay, Matthew AND Hullman, Jessica},
  booktitle = {ACM Human Factors in Computing Systems (CHI)},
  year = {2019},
  url = {https://idl.uw.edu/papers/uncertainty-research-synthesis},
  doi = {10.1145/3290605.3300432}
}