Researcher
Image and Video Coding Group
Video Coding
email: meyer@ient.rwth-aachen.de
A central part of image and video coding is the intra-prediction, which attempts to predict the content of an image block based on the already decoded environment as accurately as possible. In this way, redundancy can be removed from the image and the information content of the prediction difference can be reduced. In current standards, the adjacent pixels of the block are either repeated in a certain direction or summed up and weighted.
In recent years, machine learning techniques, especially neural networks, have achieved great success in both image analysis and classification, as well as in general prediction problems. Therefore, an attempt is made to use these methods also for the intra-prediction of video signals. First, networks must be trained to analyze the already encoded environment and generate a prediction of the block that is to be encoded. These must then be integrated into existing video codecs. Initial experiments have already shown that this can increase coding efficiency. Based on these initial results, it is now being investigated which networks and training methods are optimal for this use case in order to further increase the coding gain while also reducing the computational complexity in order to be able to decode in real time. |
One of the key points that need to be examined is the network architecture, as it not only has a significant influence on the quality of the prediction, but also determines the required computational complexity. A particularly interesting aspect is whether it is worthwhile to have such a network generate multiple predictions, of which the best can then be used, but must also be signaled. In addition, the optimal integration of such networks into the overall system of a video codec is an important aspect. In order limit the amount of additionally transmitted information as much as possible, it is particularly important to be able to estimate as precisely as possible in which cases the network-based prediction brings improvements and for which it is unsuitable. At the same time, however, the integration also determines what information and image areas are available for the prediction of a block.
M. Meyer, Neural Networks for Video Intra Prediction, vol. 25 of Aachen Series on Multimedia and Communications Engineering. Aachen: Shaker Verlag, Apr. 2024. [ bib ]
T. Meyer, M. Meyer, D. Mehlem, and C. Rohlfing, “Attribute-aware partitioning for graph-based point cloud attribute coding,” in Proc. of International Picture Coding Symposium PCS '22, (San Jose, USA), pp. 121–125, IEEE, Piscataway, Dec. 2022. [ bib | DOI | .pdf ]
J. Schneider, D. Mehlem, M. Meyer, and C. Rohlfing, “Sparse Coding-based Intra Prediction in VVC,” in Picture Coding Symposium (PCS'21), (Bristol, UK), IEEE, Piscataway, June 2021. [ bib | .pdf ]
M. Meyer, J. Wiesner, and C. Rohlfing, “Optimized convolutional neural networks for video intra prediction,” in Proc. of IEEE International Conference on Image Processing ICIP '20, IEEE, Piscataway, Oct. 2020. [ bib | more… | .pdf ]
M. Meyer, J. Wiesner, J. Schneider, and C. Rohlfing, “Convolutional neural networks for video intra prediction using cross-component adaptation,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP '19, pp. 1607–1611, IEEE, Piscataway, May 2019. [ bib | more… | .pdf ]
C. Heithausen, M. Meyer, M. Bläser, and J.-R. Ohm, “Temporal Prediction of Motion Parameters with Interchangeable Motion Models,” in Proc. of IEEE Data Compression Conference DCC '17, (Snowbird, Utah, USA), IEEE, Piscataway, Apr. 2017. [ bib ]
M. Meyer and C. Rohlfing, “Ahg11-related: Investigation on cnn-based intra prediction,” Doc. JVET-U0106, Joint Video Exploration Team (on Future Video coding) of ITU-T VCEG and ISO/IEC MPEG, online, Jan. 2021. [ bib | https ]
M. Meyer, A. Tourapis, and D. Singer, “Request for 4:4:4 support in the JEM software,” Doc. JVET-F0076, Joint Video Exploration Team (on Future Video coding) of ITU-T VCEG and ISO/IEC MPEG, Hobart, AUS, Apr. 2017. [ bib | http ]
A. Tourapis, M. Meyer, and D. Singer, “A new tool for color gamut analysis of MPEG video content,” Doc. JVET-E0105, Joint Video Exploration Team (on Future Video coding) of ITU-T VCEG and ISO/IEC MPEG, Geneva, CH, Jan. 2017. [ bib | http ]
M. Meyer, A. Tourapis, D. Singer, and Y. Su, “Ahg 7: Content colour volume SEI message - observations and findings,” Doc. JCTVC-Z0043, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Geneva, CH, Jan. 2017. [ bib | http ]
NOTICE FOR IEEE PUBLICATIONS: © IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: + Intl. 908-562-3966.