Graphical revelations: comparing students' translation errors in graphics and logic

We are interested in developing a better understanding of what it is that students find difficult in learning logic. We use both natural language and diagram-based methods for teaching students the formal language of first-order logic. In this paper, we present some initial results that demonstrate that, when we look at how students construct diagrammatic representations of information expressed in natural language (nl) sentences, the error patterns are different from those observed when students translate from nl to first-order logic (fol). In the nl-to-diagram construction task, errors associated with the interpretation of the expression not a small dodecahedron were manifested much more frequently with respect to the object's size than with respect to its shape. In the nl-to-fol task, however, no such asymmetry was observed. We hypothesize a number of possible factors that might be implicated here: differences between the nl-to-diagram and nl-to-fol tasks; the reduced expressivity of diagrams compared to language; scoping errors in participants' nl parsing; and the visuospatial properties of the blocks-world domain. In sum, constructing a diagram requires the student to provide an instantiated representation of the meaning of a natural language sentence; this tests their understanding in a way that translation into first-order logic does not, by ensuring that they are not simply carrying out a symbol manipulation exercise.