Hello everyone, my name is Haruto Tanaka, Data Scientist at HACARUS.
Recently, I have had the opportunity to research the segmentation of histopathological images (automatic region detection). In today’s article, I would like to introduce you to HistoSegNet, a deep learning model that is capable of detecting pixel-level segmentation through semi-supervised learning.
Before we begin, let’s talk about what semi-supervised learning is. Semi-supervised learning is a method where previously collected incomplete or inaccurate data is studied by a deep learning model. In particular, when it comes to the segmentation, learning is conducted by attaching a category label to an image and then the image is divided into segments.
For regional imaging segmentation, it can take a long time to prepare the labels associated with the various pixel patterns. However, by simply adding category related data to the image, the semi-supervised learning method is able to greatly reduce this burden.
One problem that has faced conventional semi-supervised learning models is their low general performance. I will go into more detail later, but with HistoSegNet its deduction process is divided into the two stages of patch and pixel level imaging. The problem is then solved by the software independently after the processing is completed.
HistoSegNet is a model used to segment histopathological images. Through the use of previously obtained segmentation results, it is possible to study the classification models associated with the tissues we are interested in. Using these results, it is possible to greatly expand the scope of histopathological analysis and offer support during histopathological examinations.
Now, let us get started by introducing HistoSegNet! As mentioned before, the inference process is divided into two stages. This article is a first of the 2-part series, where I will explain HistoNet, the model responsible for handling the patch-level inference stage.
Contents:
- Challenges of Histopathological Examinations
- HistoSegNet
- Patch Division
- HistoNet
- Summary
- References
Challenges of Histopathological Examinations
Normally, the images used in histopathological examinations are analyzed by a professional pathologist. However, as of 2020, it had been recorded that there are only around 2000 pathologists practicing in Japan.
This is a significantly small number compared to over 25 million histopathological diagnoses, 300,000 on the spot diagnoses performed during surgery, 30 million cytodiagnosis consultations and diagnoses, and at least 1500 cases that depend on the condition of the patient [2].
Analyzing such a large number of images increases the risk of human error and misdiagnosis. By creating a support system to assist during examinations, it will be possible to alleviate the burdens placed on Japan’s pathologists and reduce the risk of misdiagnosis. HistoSegNet is one of the potential tools that pathologists can use to improve the efficiency and accuracy of their diagnoses.
HistoSegNet
HistoSegNet is a model that was announced at the International Conference of Computer Vision (ICCV) back in 2019. This model consists of the 6 steps outlined below (figure 1).
- Step 1: Splitting the image into “patch” segments
- Step 2: Narrowing down the types of tissues that exist within the patch (performed by HistoNet)
- Step 3: Predict the class (in this case, the organization) that each pixel belongs to.
- Step 4: Making adjustments to the predicted values from cases such as overlapping patch segments and background images (Inter-HTT Adjustments).
- Step 5: Coloring patches for visualization
- Step 6: Restoring the shape of the original colored image from the colored patch.
Figure 1: HistoSegNet Outline (cited from Reference 1, Figure 2)
When you process an image using HistoSegNet, you get two types of outputs: morphological segmentation and functional segmentation. As shown in figure 2, looking at the morphological segmentation, the bright red region (ROI-3) corresponds to the smooth muscle while the dark red region (ROI-1) corresponds to the loose connective tissue.
Next, when looking at the functional segmentation, the dark blue region represents the exocrine glands and the light blue region represents the morphological classification(形態学的分類).
- Morphological Classification: Classification by the type of cells that make up the tissue.
Examples: smooth muscle, stratified cuboidal epithelium, neuropil, etc. - Functional Classification: Classification according to the functionality of the tissue.
Examples: Secretory glands, blood vessels, etc.
Figure 2: HistoSegNet inputs and outputs (cited from Reference 1, Figure 10).
Left: Histopathological imaging of the digestive system (input). Center: Morphological Classification (output). Right: Functional Classification (output).
As mentioned above, the general performance of HistoSegNet was found to be significantly higher than other conventional learning methods (Table 1.)
Table 1: A comparison measuring the accuracy between HistoSegNet and other methods (cited from Reference 1, Figure 11). In the table, “morph” indicates the results for the morphological classification while “func” indicates the results for the functional classification.
Table 1: A comparison measuring the accuracy between HistoSegNet and other methods (cited from Reference 1, Figure 11). In the table, “morph” indicates the results for the morphological classification while “func” indicates the results for the functional classification.
Looking at table 1, HistoSegNet outperforms both morphological and functional classifications by about .1mIoU when comparing the accuracy for analyzing unknown images. For this test, a comparison of Seed, Expand, and Constrain (SEC) and Deep Seeded Region Growing (DSRG) using the mIoU (mean IoU) with a value of 1 being the best.
In other words, HistoSegNet can highlight diagnostically relevant areas in histopathological images with a greater degree of accuracy compared to the other two models previously mentioned.
By automating the search for the diagnosis site during a pathological diagnosis, it is possible to decrease the burden placed on pathologists. Now that we have an overview of HistoSegNet, let us take a look at what is actually happening.
The First Step: Patch Division
The preliminary process for step one is the splitting of the histopathological image into patches. The average size for an image used for HistoSegNet calibration is around 14,000px by 14,000px, which is extremely large compared to images used by deep learning methods (around 224px by 224px) [4].
When using an image of this size, the number of parameters required will quickly exhaust the system’s memory. For this reason, it is divided into several smaller images (patches), which are used for the calibration instead.
While assembling the patches into a cohesive image, the images are arranged so that 25% of the boundaries overlap with adjacent patches (Figure 4, left). This practice is performed to eliminate the occurrence of “boundary artifacts.”
Boundary artifacts occur when the edges of the patches deteriorate when restoring the image to its original state. The image on the right in Figure 4 shows an example of boundary artifacts appearing in an image during the restoration process.
Figure 4: An example of reassembling patches (Left) (modified image of Figure 1 from Reference 1) and an example of boundary artifacts (right) (cited from Figure 1C of Reference 4)
The Second Step: Narrowing down the types of tissues that exist within the patch (performed by HistoNet)
As previously mentioned, HistoNet is a model that handles the patch-level inference stage during step 2. HistoNet’s architecture is similar to VGG16, a popular model used for image classification, shown in Figure 5. VGG16 is used in a wide range of classification tasks, from animal classification to COVID-19 morbidity levels based on CT scans of a person’s lungs.
Figure 5. HistoNet (top) and VGG16 (Bottom) architecture (cited from Figure 3 of Reference 1)
HistoNet is able to predict which of the 51 different types of tissues are included within the patch by assigning them a value of either 1 or 0. If the tissue is predicted to exist, it assigns a value of 1 and a value of 0 otherwise.
This process is repeated once again using 31 types of tissues. After this process, the model makes a final prediction of the tissues found within the patch. Figure 6 shows a real life example of an experiment in which a sample image is input and tested.
Figure 6: Test patch (input)(left), Ground Truth for the patch (label)(Center), and model prediction results (right).
As shown in figure 6, the results obtained by the model match the Ground Truth for the test image. By the way, some of the tissues classified in this patch included the simple squamous epithelium, dense connective tissue, smooth muscle, and many more.
By using this model to narrow down the types of tissue located in the patch, it is possible to improve the accuracy of the segmentation because it is possible to focus only on the tissues that are likely to exist within the patch in the following steps.
Summary
At first, it might seem like a difficult task to classify the labels of a patch down to every pixel for 31 separate types of labels. However, by narrowing down the label count to 7 types using HistoNet, it becomes significantly more manageable. After reading this article, the main points can be summarized as the following:
As a part of the pre-processing stage, a high resolution histopathological image is first divided into patches. This is done due to the large image size and smaller patches allows for a more manageable processing time.
HistoNet is then used to narrow down the tissue typing candidates included in each patch in order to improve the accuracy of subsequent segmentation.
HistoSegNet and other Deep Learning models have the potential to provide us insight into many topics that we currently do not understand. Although Deep Learning models are considered a black box, HistoSegNet may detect many traits that pathologists may not be aware of.
In addition to providing an automated process for image analysis, this technology also opens up new possibilities for pathological diagnosis in the future.
If you are interested in this AI technology and its potential for histopathological analysis or are interested in automating this process for yourself, please feel free to contact us. We are also open to working with like minded individuals who are interested in similar medical data analysis technologies.
Today, I talked about the first two steps for the HistoSegNet model. Next time, I would like to introduce the last four steps which includes Grad-CAM (Step 3), a process that makes predictions at a pixel level. I will also talk about the post-processing steps involving coloring and restoration from the patches into its original image (Steps 4-6).
I look forward to presenting these steps in my next article!
References
[1] Z.-H. Zhou, “A brief introduction to weakly supervised learning”, Natl Sci Rev, vol. 5, no. 1, pp. 44–53, 2017, doi: 10.1093/nsr/nwx106. [2] Toshiaki Manabe and Ryoji Kushima, “AIと医療09 遠隔病理診断ネットワークにおけるAIによる診断補助導入の可能性と展望”, あいみっく, vol. 41, no. 2, pp. 32-37, 2020, https://www.palana.or.jp/pdf/2020071303.pdf [3] L. Chan, M. S. Hosseini, C. Rowsell, K. N. Plataniotis, and S. Damaskinos, “HistoSegNet: Semantic Segmentation of Histological Tissue Type in Whole Slide Images”, 2019 Ieee Cvf Int Conf Comput Vis Iccv, vol. 00, pp. 10661–10670, 2019, doi: 10.1109/iccv.2019.01076. [4] M. S. Hosseini et al., “Atlas of Digital Pathology: A Generalized Hierarchical Histological Tissue Type-Annotated Database for Deep Learning”, 2019 Ieee Cvf Conf Comput Vis Pattern Recognit Cvpr, vol. 00, pp. 11739–11748, 2019, doi: 10.1109/cvpr.2019.01202. [5] R. Liu and J. Jia, “Reducing boundary artifacts in image deconvolution”, 2008 15th Ieee Int Conf Image Process, pp. 505–508, 2008, doi: 10.1109/icip.2008.4711802. [6] V. Shah, R. Keniya, A. Shridharani, M. Punjabi, J. Shah, and N. Mehendale, “Diagnosis of COVID-19 using CT scan images and deep learning techniques”, medRxiv, p. 2020.07.11.20151332, Jan. 2020, doi: 10.1101/2020.07.11.20151332.
