In image compression, shape-adaptive (SA) transform coding has been shown to outperform traditional block-based transform coding by better capturing the underlying textures of image regions. However, constructing optimal transforms for arbitrarily shaped image segments remains a challenging task. This thesis proposes a novel method for developing SA transforms by utilizing a non-causal finite lattice homogeneous Gauss-Markov random field (GMRF) model. The Karhunen-Loeve transform (KLT) of the GMRF model is employed as a compact, parameterized representation of the KLT for arbitrarily shaped image segments, with the model parameters effectively characterizing the image texture. This approach offers a significant advantage over existing SA transforms in the literature, as it allows for the optimization of the transform for each image segment. Specifically, if the GMRF model accurately describes the segment's texture, the resulting KLT will be the optimal transform for that segment.

A key aspect of the proposed method is the introduction of a modified version of asymmetric Neumann boundary conditions, which are designed to maintain a high transform coding gain, even for arbitrarily shaped image segments. This enhancement ensures that the transform coding process is robust and effective across diverse image shapes and textures.

Additionally, by employing a set of GMRF parameters estimated from a training set of images, the aforementioned approach is extended to enable joint content and shape adaptive transform (JCSAT) coding. This approach allows for the simultaneous adaptation to both the content and shape of image segments, leading to more efficient compression.

Experimental results demonstrate that JCSAT coding surpasses the performance of other known SA transforms, as well as the widely used 2D-discrete cosine transform (2D-DCT), in terms of energy compaction efficiency and coding gain. This research makes an important contribution to the adaptive transform coding of images, providing a more flexible and optimal solution for coding arbitrarily shaped image regions in image and video compression applications.