In this paper, we make three contributions related to the selection and use of distractors (lines of code reflecting common errors or misconceptions) in Parson’s problems. First, we demonstrate a process of by which templates for creating distractors can be selected through the analysis of student submissions to short answer questions. Second, we describe the creation of a tool that uses these templates to automatically generate distractors for novel problems. Third, we perform a preliminary analysis of how the presence of distractors impacts performance, problem solving efficiency, and item discrimination when used in summative assessment. Our results suggest that distractors should not be used in summative assessment because they significantly increase the problem’s completion time without a significant increase in problem discrimination.