Introduction:
The complexity of histopathological images remains a challenging issue in cancer diagnosis. A pathologist analyses immunohistochemical images to detect a colour-based stain, which is brown for positive nuclei with different intensities and blue for negative nuclei. Several issues emerge during the eyeballing tissue slide analysis, such as colour variations caused by stain inhomogeneity, non-uniform illumination, irregular cell shapes, and overlapping cell nuclei. To overcome those problems, an automated computer-aided diagnosis system is proposed to segment and quantify digestive neuroendocrine tumours.

Material and methods:
We present a novel pre-processing approach based on colour space assessment. A criterion called pertinence degree is introduced to select the appropriate colour channel, followed by contrast enhancement. Subsequently, the adaptive local threshold technique that uses the modified Laplacian filter is applied to minimize the implementation complexity, highlight edges, and emphasize intensity variation between cells across the slide. Finally, the improved watershed algorithm based on the concave vertex graph is applied for cell separation.

Results:
The performance of the algorithms for nucleus segmentation is evaluated according to both the object-level and pixel-level criteria. Our approach increases segmentation accuracy, with the F1-score equal to 0.986. There is significant agreement between the applied approach and the expert’s ground truth segmentation.

Conclusions:
The proposed method outperformed the state-of-the-art techniques based on recall, precision, the F1-score, and the Dice coefficient.