Salient object detection is a problem that has been considered in detail and many solutions proposed. In this thesis, we argue that work to date has addressed a problem that is relatively ill-posed. Specifically, there is not universal agreement about what constitutes a salient object when multiple observers are queried which implies a relative rank exists on salient objects. In this thesis, we solve this more general problem that considers relative rank. A novel deep learning solution is proposed based on a hierarchical representation of relative saliency and stage-wise refinement to address both of the saliency ranking and subitizing tasks. We also present methods for deriving suitable ranked salient object instances to generate a large scale dataset for saliency ranking, along with metrics suitable to measuring success in a relative object saliency landscape. Our approach exceeds performance of any prior work across all metrics considered (both traditional and newly proposed).