Predicting saliency by learning inpainting error

Abstract

Understanding which parts of an image are most salient is an ongoing research problem in the field of computer vision. In this thesis, we consider the problem of saliency prediction insofar as this can benefit from learning image inpainting error. We address this problem by presenting a novel approach to predict saliency maps. The first model learns which parts of the image are most difficult to predict by applying an inpainting algorithm on a regular grid and measuring the error in the resulting prediction. A convolutional neural network is trained to predict the degree of error subject to this inpainting process. A second network uses transfer learning from the first network to predict saliency maps by taking advantage of image inpainting error. We demonstrate that saliency prediction can benefit from first learning image inpainting error. We evaluate and compare our results both by considering image inpainting error, and also through an ablation study we consider a comparison to direct training on the saliency data without transfer learning. We then evaluate our results with the previous state of the art models. We test our networks on two well-known saliency datasets including CAT2000, and SALICON.