Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow

By: Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim

Abstract

Programmers often consult an online Q&A forum such as Stack Overflow to learn new APIs. This paper presents an empirical study on the prevalence and severity of API misuse on Stack Overflow. To reduce manual assessment effort, we design Maple, an API usage mining approach that extracts patterns from over 380K Java repositories on GitHub and subsequently reports potential API usage violations in Stack Overflow posts. We analyze 217,818 Stack Overflow posts using Maple and find that around 31% of them have potential API usage violations that may produce the symptoms such as program crashes and resource leaks. Such API misuse is caused by three main reasons—missing control constructs, missing or incorrect order of API calls, and incorrect guard conditions. Even the posts that are accepted as correct answers or upvoted by other programmers are not necessarily more reliable than other posts in terms of API misuse. This study result calls for a new human-in-the-loop approach to augment Stack Overflow code snippets and help the user consider better or alternative API usage.

ACM Reference

Zhang, T. et al. 2018. Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow. Proceedings of the 40th International Conference on Software Engineering, ICSE, Gothenburg, Sweden (2018), 886–896.

BibTeX Reference

@inproceedings{ZhangETAL2018,
  author = {Tianyi Zhang and Ganesha Upadhyaya and Anastasia Reinhardt and Hridesh Rajan and Miryung Kim},
  title = {Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow},
  booktitle = {Proceedings of the 40th International Conference on Software Engineering, ICSE, Gothenburg, Sweden},
  pages = {886--896},
  year = {2018},
  publisher = {{ACM}},
  editor = {Michel Chaudron and Ivica Crnkovic and Marsha Chechik and Mark Harman},
  doi = {10.1145/3180155.3180260},
  abstract = {
 Programmers often consult an online Q&A forum such as Stack Overflow to learn new APIs. 
 This paper presents an empirical study on the prevalence and severity of API misuse on Stack Overflow. 
 To reduce manual assessment effort, we design Maple, an API usage mining approach 
 that extracts patterns from over 380K Java repositories on GitHub and subsequently 
 reports potential API usage violations in Stack Overflow posts.
 We analyze 217,818 Stack Overflow posts using Maple and find that around 31% of them 
 have potential API usage violations that may produce the symptoms such as program 
 crashes and resource leaks. Such API misuse is caused by three main 
 reasons---missing control constructs, missing or incorrect order of API calls, and 
 incorrect guard conditions. Even the posts that are accepted as correct answers or 
 upvoted by other programmers are not necessarily more reliable than other posts in 
 terms of API misuse. This study result calls for a new human-in-the-loop approach 
 to augment Stack Overflow code snippets and help the user consider better or 
 alternative API usage.},
}