Coding Gain Optimized 8-Point DST with Fast Algorithm for Intra-frames in Video Coding

Cilasun M. H., KAMIŞLI F.

European Signal Processing Conference (EUSIPCO), Rome, Italy, 3 - 07 August 2018, pp.156-160 identifier

  • Publication Type: Conference Paper / Full Text
  • City: Rome
  • Country: Italy
  • Page Numbers: pp.156-160
  • Middle East Technical University Affiliated: Yes


In modern video codecs, such as HEVC and VP9, intra-frame blocks are decorrelated using DCT or DST. The optimal separable transform for intra prediction residual blocks has been determined to be a hybrid transform composed of the DCT and the odd type-3 DST (ODST-3), independent of block size. However, the ODST-3 has no fast algorithm like the DCT. Hence its use in HEVC and VP9 has been limited to only 4x4 blocks. For larger blocks such as 8x8 or 16x16, HEVC replaces the optimal ODST-3 with the conventional DCT while VP9 replaces it with the even type-3 DST (EDST-3), both of which have fast algorithms. The EDST-3 has better coding gain than the DCT but it has still a coding gain loss with respect to the optimal ODST-3. This paper attempts to optimize some parameters of the EDST-3 to reduce this coding gain loss while still retaining the fast algorithm. In particular, the 8-point EDST-3 is represented as a cascade of Givens rotations and some rotation angles are optimized to reduce the coding gain loss with respect to the optimal 8-point ODST-3. By replacing only the 8-point EDST-3 in VP9 with this optimized transform, while leaving other transforms with different sizes unchanged, average Bjontegaard-Delta bitrate savings of -0.13% are achieved with respect to the standard VP9 codec.