Learning a test oracle towards automating image segmentation evaluation

Image segmentation is the act of extracting particular structures of interest from an image. A lot of time and effort is spent in order to evaluate image segmentation algorithms. If the image segmentation algorithm does not provide accurate enough results or in verification and validation terms fails, the technical expert needs to modify it and rerun the whole test suite to verify the revised image segmentation algorithm. This process is repeated as the image segmentation algorithm evolves to its final acceptable version where the test suite passes. This evaluation process is mostly done manually at the moment and is therefore very time consuming, requiring the presence of reliable experts. In this thesis, a solution is proposed that uses machine learning techniques to semi-automate this evaluation process. During the initial learning phase, the expert is required to evaluate segmentations manually. The similarity between the segmentations produced by the initial versions of the segmentation algorithm is found by applying a set of comparison measures to pairs of segmentations from the same subject. This information is used by different machine learning algorithms to devise a classifier that can classify a pair of segmentations as being diagnostically consistent or inconsistent. In a second phase, once a valid classifier is learnt, a segmentation produced by any new version of the image segmentation algorithm under test will be automatically deemed correct or incorrect depending on whether it is diagnostically consistent with the segmentations previously deemed correct. In this second phase, there is no need for any intervention from human experts. To demonstrate the performance of the approach, we have applied the solution to the evaluation of the left heart ventricle segmentation and have gotten very promising results. This solution also helps find the best performing machine learning techniques and the similarity measures with the most discriminating power in the context of the application.