Temporal refinement of 3D CNN semantic segmentations on 4D time-series of undersampled tomograms using hidden Markov models

Bellos, Dimitrios; Basham, Mark; Pridmore, Tony; French, Andrew P.

Abstract

Recently, several convolutional neural networks have been proposed not only for 2D images, but also for 3D and 4D volume segmentation. Nevertheless, due to the large data size of the latter, acquiring a sufficient amount of training annotations is much more strenuous than in 2D images. For 4D time-series tomograms, this is usually handled by segmenting the constituent tomograms independently through time with 3D convolutional neural networks. Inter-volume information is therefore not utilized, potentially leading to temporal incoherence. In this paper, we attempt to resolve this by proposing two hidden Markov model variants that refine 4D segmentation labels made by 3D convolutional neural networks working on each time point. Our models utilize not only inter-volume information, but also the prediction confidence generated by the 3D segmentation convolutional neural networks themselves. To the best of our knowledge, this is the first attempt to refine 4D segmentations made by 3D convolutional neural networks using hidden Markov models. During experiments we test our models, qualitatively, quantitatively and behaviourally, using prespecified segmentations. We demonstrate in the domain of time series tomograms which are typically undersampled to allow more frequent capture; a particularly challenging problem. Finally, our dataset and code is publicly available.

Citation

Bellos, D., Basham, M., Pridmore, T., & French, A. P. (2021). Temporal refinement of 3D CNN semantic segmentations on 4D time-series of undersampled tomograms using hidden Markov models. Scientific Reports, 11(1), Article 23279. https://doi.org/10.1038/s41598-021-02466-x

• Journal Article Type: Article
• Acceptance Date: Nov 9, 2021
• Online Publication Date: Dec 2, 2021
• Publication Date: Dec 1, 2021
• Deposit Date: Nov 19, 2021
• Publicly Available Date: Dec 2, 2021
• Journal: Scientific Reports
• Electronic ISSN: 2045-2322
• Publisher: Nature Publishing Group
• Peer Reviewed: Peer Reviewed
• Volume: 11
• Issue: 1
• Article Number: 23279
• DOI: https://doi.org/10.1038/s41598-021-02466-x
• Public URL: https://nottingham-repository.worktribe.com/output/6737501
• Publisher URL: https://www.nature.com/articles/s41598-021-02466-x