A Comprehensive Study on Deep Learning Bug Characteristics

By: Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan

Abstract

Deep learning has gained substantial popularity in recent years. Developers mainly rely on libraries and tools to add deep learning capabilities to their software. What kinds of bugs are frequently found in such software? What are the root causes of such bugs? What impacts do such bugs have? Which stages of deep learning pipeline are more bug prone? Are there any antipatterns? Understanding such characteristics of bugs in deep learning software has the potential to foster the development of better deep learning platforms, debugging mechanisms, development practices, and encourage the development of analysis and verification frameworks. Therefore, we study 2716 high-quality posts from Stack Overflow and 500 bug fix commits from Github about five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand the types of bugs, root causes of bugs, impacts of bugs, bug-prone stage of deep learning pipeline as well as whether there are some common antipatterns found in this buggy software. The key findings of our study include: data bug and logic bug are the most severe bug types in deep learning software appearing more than 48% of the times, major root causes of these bugs are Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) showing up more than 43% of the times. We have also found that the bugs in the usage of deep learning libraries have some common antipatterns that lead to a strong correlation of bug types among the libraries.

ACM Reference

Islam, M.J. et al. 2019. A Comprehensive Study on Deep Learning Bug Characteristics. Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE, Tallinn, Estonia (2019), 510–520.

BibTeX Reference

@inproceedings{IslamNguyenPanRajan2019,
  author = {Md Johirul Islam and Giang Nguyen and Rangeet Pan and Hridesh Rajan},
  title = {A Comprehensive Study on Deep Learning Bug Characteristics},
  booktitle = {Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE, Tallinn, Estonia},
  pages = {510--520},
  year = {2019},
  publisher = {{ACM}},
  editor = {Marlon Dumas and Dietmar Pfahl and Sven Apel and Alessandra Russo},
  doi = {10.1145/3338906.3338955},
  abstract = {
  Deep learning has gained substantial popularity in recent years.
  Developers mainly rely on libraries and tools to add deep learning
  capabilities to their software. What kinds of bugs are frequently
  found in such software? What are the root causes of such bugs?
  What impacts do such bugs have? Which stages of deep learning
  pipeline are more bug prone? Are there any antipatterns?
  Understanding such characteristics of bugs in deep learning
  software has the potential to foster the development of better
  deep learning platforms, debugging mechanisms, development practices,
  and encourage the development of analysis and verification frameworks.
  Therefore, we study 2716 high-quality posts from Stack Overflow
  and 500 bug fix commits from Github about five popular deep learning
  libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand
  the types of bugs, root causes of bugs, impacts of bugs, bug-prone
  stage of deep learning pipeline as well as whether there are some
  common antipatterns found in this buggy software.
  The key findings of our study include: data bug and logic bug are
  the most severe bug types in deep learning software appearing more
  than 48% of the times, major root causes of these bugs are Incorrect
  Model Parameter (IPS) and Structural Inefficiency (SI) showing up
  more than 43% of the times.
  We have also found that the bugs in the usage of deep learning
  libraries have some common antipatterns that lead to a strong
  correlation of bug types among the libraries.},
}