Do TODO comments help or hurt when an AI writes your code? Our ICSE 2024 paper finds out.

April 18, 2024

Code-intelligence tools like GitHub Copilot turn natural-language hints into code, which raises a practical question about the notes developers already leave in their source. Comments that record unfinished work, a common form of self-admitted technical debt, are exactly the kind of natural-language hint such a tool might act on, for better or worse.

In this paper (ICSE 2024), David OBrien, Sumon Biswas, Sayem Mohammad Imtiaz, Rabe Abdalkareem, Emad Shihab, and Hridesh Rajan evaluate how prompt engineering and TODO comments interact with Copilot. They examine whether these comments steer the tool toward helpful completions or toward propagating the very debt they describe, and turn the answer into concrete guidance for working with such tools.

This work is part of Modular and Dependable AI; it builds on our study of self-admitted technical debt in machine-learning software. The full paper is available here.