Constrained quantization in the transform domain with applications in arbitrarily-shaped object coding

In any block-based transform coding of image/video signals, it is well-known that the mean square error (MSE) distortion measured in the pixel domain is exactly equal to the MSE distortion resulted from quantization in the transform domain if the involved transform matrix is unitary. However, such a property no longer exists if the pixel-domain distortion is measured only on a selected part of pixels within one image block. This provides us an opportunity of dynamically shaping the quantization errors so as to make the selected pixels (much) better than the unselected ones. In this paper, we first develop a reversed iterative algorithm to guide us to perform a highly constrained quantization so that the coding quality of the selected pixels in each image block is significantly higher than what can be achieved by using the normal quantization. Then, we apply this intelligent quantization in one practical scenariocoding of arbitrarily-shaped image blocks in MPEG-4, showing remarkable improvements in comparison with the original MPEG-4.