What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow

By: Md Johirul Islam, Hoan Anh Nguyen, Rangeet Pan, and Hridesh Rajan

Abstract

Modern software systems are increasingly including machine learning (ML) as an integral component. However, we do not yet understand the difficulties faced by software developers when learning about ML libraries and using them within their systems. To that end, this work reports on a detailed (manual) examination of 3,280 highly-rated posts related to ten ML libraries, namely Tensorflow, Keras, Scikit-learn, Weka, Caffe, Theano, MLLib, Torch, Apache Mahout, and H2O, on Stack Overflow, a popular online technical Q and A forum. We classify these questions into seven typical stages of an ML pipeline to understand the correlation between the library and the stage. We also perform inter- and intra-library analyses to understand broad trends. Our findings reveal the urgent need for software engineering (SE) research in this area. Both static and dynamic analyses are mostly absent and badly needed to help developers find errors earlier. While there has been some early research on debugging, much more work is needed. API misuses are prevalent and API design improvements are sorely needed. Enabling reuse of trained models across libraries needs attention. Last and somewhat surprisingly, a tug of war between providing higher levels of abstractions and the need to understand the behavior of the trained model is prevalent. These findings suggest new paths for SE researchers to help improve the engineering of software that includes ML components.

ACM Reference

Islam, M.J. et al. 2019. What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow. CoRR. abs/1906.11940, (2019).

BibTeX Reference

@article{MdJohirulNguyenRajan2018,
  author = {Md Johirul Islam and Hoan Anh Nguyen and Rangeet Pan and Hridesh Rajan},
  title = {What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow},
  journal = {CoRR},
  volume = {abs/1906.11940},
  year = {2019},
  abstract = {
  Modern software systems are increasingly including machine learning (ML) as 
  an integral component. However, we do not yet understand the difficulties 
  faced by software developers when learning about ML libraries and using them 
  within their systems. To that end, this work reports on a detailed (manual) 
  examination of 3,280 highly-rated posts related to ten ML libraries, 
  namely Tensorflow, Keras, Scikit-learn, Weka, Caffe, Theano, MLLib, Torch, 
  Apache Mahout, and H2O, on Stack Overflow, a popular online technical Q and A forum. We classify 
  these questions into seven typical stages of an ML pipeline to understand the 
  correlation between the library and the stage. We also perform inter- and 
  intra-library analyses to understand broad trends. Our findings reveal the urgent 
  need for software engineering (SE) research in this area. Both static and dynamic 
  analyses are mostly absent and badly needed to help developers find errors 
  earlier. While there has been some early research on debugging, much more work is 
  needed. API misuses are prevalent and API design improvements are sorely 
  needed. Enabling reuse of trained models across libraries needs attention. 
  Last and somewhat surprisingly, a tug of war between providing higher levels 
  of abstractions and the need to understand the behavior of the trained model 
  is prevalent. These findings suggest new paths for SE researchers to help 
  improve the engineering of software that includes ML components.},
}