Can requests-for-action and commitments-to-act be reliably identified in email messages?

This paper reports on the results of an exploratory annotation task where three coders classified the presence and strength of Requests-for-Action (requests) and Commitments-to-Act (promises) in workplace email messages. The purpose of our annotation task was to explore levels of human agreement to establish whether this is a repeatable task that lends itself to automation. The results from our annotation task suggest that there is relatively high agreement about which sentences embody Requests-for-Action (Κ = 0.78), but poorer agreement about Commitments-to-Act (Κ = 0.54). Analysis of cases of coder disagreement highlighted several areas of systematic disagreement which we believe can be addressed through refining our annotation guidelines. Given this scope for improving agreement, we believe the results presented here are encouraging for our intention to perform largerscale annotation work leading to automation of the detection and classification of Requests-for-Action and Commitments-to-Act in email communication.