Skip to main content
SpringerLink
Log in

Multi-task deep learning for fine-grained classification and grading in breast cancer histopathological images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Fine-grained classification and grading of breast cancer (BC) histopathological images are of great value in clinical application. However, automatic classification and grading of BC histopathological images are complicated by (1) small inter-class variance and large intra-class variance exist in BC histopathological images, and (2) features extracted from similar histopathological images with different magnification are quite different. To address these issues, an improved deep convolution neural network model is proposed and the procedure can be divided into three main stages. Firstly, in the representation learning process, multi-class recognition task and verification task of image pair are combined. Secondly, in the feature extraction process, a prior knowledge is built, which is “the variances in feature outputs between different subclasses is relatively large while the variance between the same subclass is small.” Additionally, the prior information that histopathological images with different magnification belong to the same subclass are embedded in the feature extraction process, which contributes to less sensitive with image magnification. The experimental results based on three different histopathological image datasets show that the performance of the proposed method is better than state of the art, with better robustness and generalization ability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Akbar S, Peikari M, Salama S, Nofech-Mozes S, Martel A (2017) Transitioning between convolutional and fully connected layers in neural networks. In: Deep learning in medical image analysis and multimodal learning for clinical decision support (DLMIA)

  2. Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: International conference on pattern recognition (ICPR), pp 2441–2446

  3. Bell S, Bala K (2015) Learning visual similarity for product design with convolutional neural networks. ACM Trans Graph 34(4):98:1–98:10

    Article  Google Scholar 

  4. Chen H, Dou Q, Wang X et al (2016) Mitosis detection in breast cancer histology images via deep cascaded networks. In: Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI), pp 1160–1166

  5. Chollet F (2016) Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357

  6. Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 539–546

  7. Codella N, Moradi M, Matasar M, Sveda-Mahmood T, Smith JR (2016) Lymphoma diagnosis in histopathology using a multi-stage visual learning approach. In: Proceedings of the SPIE, medical imaging 2016: digital pathology

  8. Dimitropoulos K, Barmpoutis P, Zioga C, Kamas A, Patsiaoura K, Grammalidis N (2017) Grading of invasive breast carcinoma through grassmannian VLAD encoding. Plos One 12(9):e0185110

    Article  Google Scholar 

  9. Dozat T (2015) Incorporating nesterov momentum into adam. Technical report, http://cs229.stanford.edu/proj2015/054_report.pdf

  10. Garud H, Karri SPK, Sheet D et al (2017) High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 828–833

  11. Gupta V, Bhavsar A (2017) An integrated multi-scale model for breast cancer histopathological image classification with joint colour-texture features. In: 17th international conference on computer analysis of images and patterns (CAIP), pp 354–366

  12. Gupta V, Bhavsar A (2017) Breast cancer histopathological image classification: is magnification important?. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 769–776

  13. Gupta V, Singh A, Sharma K et al (2017) Automated classification for breast cancer histopathology images: is stain normalization important?. In: Computer assisted and robotic endoscopy and clinical image-based procedures (CLIP) 10550, pp 160–169

  14. Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction by learning an invariant mapping. In: Computer vision and pattern recognition (CVPR), pp 1735–1742

  15. Han Z, Wei B, Zheng Y, Yin Y, Li K, Li S (2017) Breast cancer multi-classification from histopathological images with structured deep learning model. Sci Rep 7(1):4172

    Article  Google Scholar 

  16. Janowczyk A, Madabhushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7(1):29

    Article  Google Scholar 

  17. Kowal M, Filipczuk P, Obuchowicz A et al (2013) Computer-aided diagnosis of breast cancer based on fine needle biopsy microscopic images. Comput Biol Med 43 (10):1563–1572

    Article  Google Scholar 

  18. Lan R, Zhou Y (2016) Quaternion-michelson descriptor for color image classification. IEEE Trans Image Process 25(11):5281–5292

    Article  MathSciNet  Google Scholar 

  19. Lan R, Zhou Y, Tang Y (2016) Quaternionic local ranking binary pattern: a local descriptor of color images. IEEE Trans Image Process 25(2):566–579

    Article  MathSciNet  Google Scholar 

  20. Lan R, Zhou Y, Tang Y (2017) Quaternionic weber local descriptor of color images. IEEE Trans Circuits Syst Video Techn 27(2):261–274

    Article  Google Scholar 

  21. Lan R, Zhong S, Liu Z et al (2018) A simple texture feature for retrieval of medical images. Multimed Tools Appl 77(9):10853–10866

    Article  Google Scholar 

  22. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444

    Article  Google Scholar 

  23. Li P, Wang D, Wang L, Lu H (2018) Deep visual tracking: review and experimental comparison. Pattern Recogn 76:323–338

    Article  Google Scholar 

  24. Li Y, Lu H, Li KC et al (2018) Non-uniform de-scattering and de-blurring of underwater images. Mob Netw Appl 23(4):352–362

    Article  Google Scholar 

  25. Lu H, Zhang L, Serikawa S (2012) Maximum local energy: an effective approach for multisensor image fusion in beyond wavelet transform domain. Comput Math Appl 64(5):996–1003

    Article  Google Scholar 

  26. Lu H, Li B, Zhu J et al (2017) Wound intensity correction and segmentation with convolutional neural networks. Concurr Comput Pract Experience 29(6):3927

    Article  Google Scholar 

  27. Lu H, Li Y, Chen M et al (2018) Brain intelligence: go beyond artificial intelligence. Mob Netw Appl 23(2):1–8

    Google Scholar 

  28. Pan X, Li L, Yang H et al (2017) Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing 229:88–99

    Article  Google Scholar 

  29. Pan X, Yang D, Li L et al (2018) Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks. World Wide Web-internet & Web Information Systems 21(6):1721–1743

    Google Scholar 

  30. Song Y, Zou J, Chang H et al (2017) Adapting fisher vectors for histopathology image classification. In: International symposium on biomedical imaging (ISBI), pp 600–603

  31. Song Y, Chang H, Huang H, Cai W (2017) Supervised intra-embedding of fisher vectors for histopathology image classification. In: Medical image computing and computer-assisted intervention (MICCAI)

  32. Spanhol FA, Cavalin P, Oliveira LS, Petitjean C, Heutte L (2017) Deep features for breast cancer histopathological image classification. In: IEEE international conference on systems, man, and cybernetics (IEEE SMC), pp 1868–1873

  33. Spanhol FA, Oliveira LS, Petitjean C et al (2016) A dataset for breast cancer histopathological image classification. IEEE Trans Biomed Eng 63(7):1455–1462

    Article  Google Scholar 

  34. Spanhol FA, Oliveira LS, Petitjean C, Heutte L (2016) Breast cancer histopathological image classification using convolutional neural networks. In: 2016 international joint conference on neural networks, pp 2560–2567

  35. Stewart B, Wild CP (2014) World cancer report 2014. International Agency for Research on Cancer, World Health Organization

  36. Sun Y, Chen Y, Wang X et al (2014) Deep learning face representation by joint identification-verification. In: International conference on neural information processing systems (ICNIPS), pp 1988–1996

  37. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S et al (2015) Going deeper with convolutions. In: Computer vision and pattern recognition (CVPR)

  38. Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1701–1708

  39. Wang D, Lu H, Li X (2011) Two dimensional principal components of natural images and its application. Neurocomputing 74(17):2745–2753

    Article  Google Scholar 

  40. Wang P, Hu X, Li Y, Liu Q, Zhu X (2016) Automatic cell nuclei segmentation and classification of breast cancer histopathology images. Signal Process 122:1–13

    Article  Google Scholar 

  41. Wang D, Khosla A, Gargeya R et al (2016) Deep learning for identifying metastatic breast cancer. arXiv:1606.05718

  42. Wei B, Han Z, He X, Yin Y (2017) Deep learning model based breast cancer bistro pathological image classification. In: 2017 IEEE 2nd international conference on cloud computing and big data analysis (ICCCBDA)

  43. Xiao T, Xu Y, Yang K et al (2015) The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 842–850

  44. Zhang S, Metaxas D (2016) Large-scale medical image analytics: recent methodologies, applications and future directions. Med Image Anal 33:98–101

    Article  Google Scholar 

  45. Zhang Y, Zhang B, Coenen F et al (2014) One-class kernel subspace ensemble for medical image classification. EURASIP J Adv Signal Process 2014:17. https://doi.org/10.1186/1687-6180-2014-17

    Article  Google Scholar 

  46. Zhang X, Su H, Yang L, Zhang S (2015) Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  47. Zhang X, Zhou F, Lin Y, Zhang S (2016) Embedding label structures for fine-grained feature representation. In: 29th IEEE conference on computer vision and pattern recognition (CVPR), pp 1114–1123

  48. Zhi W, Yueng HWF, Chen Z et al (2017) Using transfer learning with convolutional neural networks to diagnose breast cancer from histopathological images. In: 24th international conference on neural information processing (ICNIP)

  49. Zhu X, Li X, Zhang S (2016) Block-row sparse multiview multilabel learning for image classification. IEEE Trans Cybern 46(2):450–461

    Article  Google Scholar 

  50. Zhu X, Suk HI, Lee SW et al (2016) Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans Bio-Med Eng 63(3):607–618

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Spanhol et al. [33], Dimitropoulos et al. [8], and Dr. Andrew Janowczyk et al. [16] for publishing the datasets. We would like to express our gratitude to anonymous reviewers and editor for their helpful comments. This research was supported in part by the National Natural Science Foundation of China (Grant Nos. 21365008, 61462018, 61762026 and 61562013), and Natural Science Foundation of Guangxi Province (No. 2017GXNSFDA198025 and 2017GXNSFAA198278). The authors declare that they have no conflict of interest.

Author information

Authors and Affiliations

  1. School of Automation, Beijing University of Posts and Telecommunications, No. 10, Xitucheng Road, Haidian District, Beijing, China

    Lingqiao Li, Xipeng Pan, Huihua Yang, Zhongming Li, Zhiwei Cao & Longhao Zhang

  2. School of Computer Science and Information Security, Guilin University of Electronic Technology, No. 1 Jinji Road, Qixing District, Guilin, China

    Lingqiao Li, Huihua Yang, Zhenbing Liu & Yongxian Fan

  3. School of Computing and Information Systems, University of Melbourne, Melbourne, Australia

    Yubei He

Authors
  1. Lingqiao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xipeng Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huihua Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenbing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yubei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhongming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yongxian Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhiwei Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Longhao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Huihua Yang or Zhenbing Liu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Lingqiao Li and Xipeng Pan are contributed equally

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, L., Pan, X., Yang, H. et al. Multi-task deep learning for fine-grained classification and grading in breast cancer histopathological images. Multimed Tools Appl 79, 14509–14528 (2020). https://doi.org/10.1007/s11042-018-6970-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6970-9

Keywords

Search

Navigation