Christian Timmerer’s research while affiliated with Alpen-Adria-Universität Klagenfurt and other places

HTTP adaptive streaming (HAS) has emerged as a prevalent approach for over-the-top (OTT) video streaming services due to its ability to deliver a seamless user experience. A fundamental component of HAS is the bitrate ladder, which comprises a set of encoding parameters ( e.g. , bitrate-resolution pairs) used to encode the source video into multiple representations. This adaptive bitrate ladder enables the client’s video player to dynamically adjust the quality of the video stream in real-time based on fluctuations in network conditions, ensuring uninterrupted playback by selecting the most suitable representation for the available bandwidth. The most straightforward approach involves using a fixed bitrate ladder for all videos, consisting of pre-determined bitrate-resolution pairs known as one-size-fits-all . Conversely, the most reliable technique relies on intensively encoding all resolutions over a wide range of bitrates to build the convex hull , thereby optimizing the bitrate ladder by selecting the representations from the convex hull for each specific video. Several techniques have been proposed to predict content-based ladders without performing a costly, exhaustive search encoding. This paper provides a comprehensive review of various convex hull prediction methods, including both conventional and learning-based approaches. Furthermore, we conduct a benchmark study of several handcrafted- and deep learning (DL)-based approaches for predicting content-optimized convex hulls across multiple codec settings. The considered methods are evaluated on our proposed large-scale dataset, which includes 300 UHD video shots encoded with software and hardware encoders using three state-of-the-art video standards, including AVC/H.264, HEVC/H.265, and VVC/H.266, at various bitrate points. Our analysis provides valuable insights and establishes baseline performance for future research in this field. Dataset URL: https://nasext-vaader.insa-rennes.fr/ietr-vaader/datasets/br_ladder

HTTP Adaptive Streaming (HAS) is a widely adopted method for delivering video content over the Internet, requiring each video to be encoded at multiple bitrates and resolution pairs, known as representations, to adapt to various network conditions and device capabilities. This multi-bitrate encoding introduces significant challenges due to the computational and time-intensive nature of encoding multiple representations. Conventional approaches often encode these videos independently without leveraging similarities between different representations of the same input video. This paper proposes an accelerated multi-resolution encoding strategy that utilizes representations of lower resolutions as references to speed up the encoding of higher resolutions when using Versatile Video Coding (VVC); specifically in VVenC, an optimized open-source software implementation. For multi-resolution encoding, a mid-bitrate representation serves as the reference, allowing interpolated encoded partition data to efficiently guide the partitioning process in higher resolutions. The proposed approach uses shared encoding information to reduce redundant calculations, optimizing partitioning decisions. Experimental results demonstrate that the proposed technique achieves a reduction of up to 17% compared to medium preset in encoding time across videos of varying complexities with minimal BDBR/BDT of 0.12 compared to the fast preset.

In response to the growing demand for high-quality videos, Versatile Video Coding (VVC) was released in 2020, building on the hybrid coding architecture of its predecessor, HEVC, achieving about 50% bitrate reduction for the same visual quality. It introduces more flexible block partitioning, enhancing compression efficiency at the cost of increased encoding complexity. To make efficient use of VVC in practical applications, optimization is essential. VVenC, an optimized open-source VVC encoder, introduces multiple presets to address the trade-off between compression efficiency and encoder complexity. Although an optimized set of encoding tools has been selected for each preset, the rate-distortion (RD) search space in the encoder presets still poses a challenge for efficient encoder implementations. In this paper, we propose Early Termination using Reference Frames (ETRF), which improves the trade-off between encoding efficiency and time complexity and positions itself as a new preset between medium and fast presets. The CTU partitioning map of the reference frames in lower temporal layers is employed to accelerate the encoding of frames in higher temporal layers. The results show a reduction in the encoding time of around 21% compared to the medium preset. Specifically, for videos with high spatial and temporal complexities, which typically require longer encoding times, the proposed method achieves a better trade-off between bitrate savings and encoding time compared to the fast preset.

As multimedia services such as video streaming, video conferencing, virtual reality (VR), and online gaming continue to expand, ensuring high perceptual visual quality becomes a priority to maintain user satisfaction and competitiveness. However, multimedia content undergoes various distortions during acquisition, compression, transmission, and storage, resulting in the degradation of experienced quality. Thus, perceptual visual quality assessment (PVQA), which focuses on evaluating the quality of multimedia content based on human perception, is essential for optimizing user experiences in advanced communication systems. Several challenges are involved in the PVQA process, including diverse characteristics of multimedia content such as image, video, VR, point cloud, mesh, multimodality, etc., and complex distortion scenarios as well as viewing conditions. In this paper, we first present an overview of PVQA principles and methods. This includes both subjective methods, where users directly rate their experiences, and objective methods, where algorithms predict human perception based on measurable factors such as bitrate, frame rate, and compression levels. Based on the basics of PVQA, quality predictors for different multimedia data are then introduced. In addition to traditional images and videos, immersive multimedia and generative artificial intelligence (GenAI) content are also discussed. Finally, the paper concludes with a discussion on the future directions of PVQA research.

Reversible data hiding in encrypted images (RDH-EI) has gained widespread attention due to its potential applications in secure cloud storage. However, the security challenges of RDH-EI in cloud storage scenarios remain largely unexplored. In this paper, we present a counterfeiting attack on RDH-EI schemes that utilize block-permutation and Co-XOR (BPCX) encryption. We demonstrate that ciphertext images generated by BPCX-based RDH-EI are easily tampered with to produce a counterfeit decrypted image with different contents imperceptible to the human eye. This vulnerability is mainly because the block permutation key information of BPCX is susceptible to known-plaintext attacks (KPAs). Taking ciphertext images in telemedicine scenarios as an example, we describe two potential counterfeiting attacks, namely fixed-area and optimal-area attacks. We show that the quality of forged decrypted images depends on the accuracy of the estimated block-permutation key under KPA conditions. To improve the invisibility of counterfeit decrypted images, we analyze the limitations of existing KPA methods against BPCX encryption for $2\times 2$ block sizes and propose a novel diagonal inversion rule specifically designed for image blocks. This rule further enhances the accuracy of the estimated block-permutation key. The experiments show that, compared to existing KPA methods, the accuracy of the estimated block-permutation key in the UCID dataset increases by an average of 11.5%. In the counterfeiting attack experiments on Camera's encrypted image, we successfully tampered with over 80% of the pixels in the target area under the fixed-region attack. Additionally, we achieved a tampering success rate exceeding 90% in the optimal-region attack.

Energy efficiency has become a crucial aspect of today's IT infrastructures, and video (streaming) accounts for over half of today's Internet traffic. This column highlights open-source tools, datasets, and solutions addressing energy efficiency in video streaming presented at ACM Multimedia Systems 2024 and its co-located workshop ACM Green Multimedia Systems.

The 146th MPEG meeting was held in Rennes, France from 22-26 April 2024, and the official press release can be found here. It comprises the following highlights: •AI-based Point Cloud Coding: Call for proposals focusing on AI-driven point cloud encoding for applications such as immersive experiences and autonomous driving. •Object Wave Compression: Call for interest in object wave compression for enhancing computer holography transmission. •Open Font Format: Committee Draft of the fifth edition, overcoming previous limitations like the 64K glyph encoding constraint. •Scene Description: Ratified second edition, integrating immersive media objects and extending support for various data types. •MPEG Immersive Video (MIV): New features in the second edition, enhancing the compression of immersive video content. •Video Coding Standards: New editions of AVC, HEVC, and Video CICP, incorporating additional SEI messages and extended multiview profiles. •Machine-Optimized Video Compression: Advancement in optimizing video encoders for machine analysis. •MPEG-I Immersive Audio: Reached Committee Draft stage, supporting high-quality, real-time interactive audio rendering for VR/AR/MR. •Video-based Dynamic Mesh Coding (V-DMC): Committee Draft status for efficiently storing and transmitting dynamic 3D content. •LiDAR Coding: Enhanced efficiency and responsiveness in LiDAR data processing with the new standard reaching Committee Draft status.

The 145th MPEG meeting was held online from 22-26 January 2024, and the official press release can be found here. It comprises the following highlights: •Latest Edition of the High Efficiency Image Format Standard Unveils Cutting-Edge Features for Enhanced Image Decoding and Annotation •MPEG Systems finalizes Standards supporting Interoperability Testing •MPEG finalizes the Third Edition of MPEG-D Dynamic Range Control •MPEG finalizes the Second Edition of MPEG-4 Audio Conformance •MPEG Genomic Coding extended to support Transport and File Format for Genomic Annotations •MPEG White Paper: Neural Network Coding (NNC) - Efficient Storage and Inference of Neural Networks for Multimedia Applications This column will focus on the High Efficiency Image Format (HEIF) and interoperability testing. As usual, a brief update on MPEG-DASH et al. will be provided.

Live video streaming’s growing demand for high-quality content has resulted in significant energy consumption, creating challenges for sustainable media delivery. Traditional adaptive video streaming approaches rely on the over-provisioning of resources leading to a fixed bitrate ladder, which is often inefficient for the heterogeneous set of use cases and video content. Although dynamic approaches like per-title encoding optimize the bitrate ladder for each video, they mainly target video-on-demand to avoid latency and fail to address energy consumption. In this paper, we present LiveESTR, a method for building a quality- and energy-aware bitrate ladder for live video streaming. LiveESTR eliminates the need for exhaustive video encoding processes on the server side, ensuring that the bitrate ladder construction process is fast and energy efficient. A lightweight model for multi-label classification, along with a lookup table, is utilized to estimate the optimized resolution-bitrate pair in the bitrate ladder. Furthermore, both spatial and temporal resolutions are supported to achieve high energy savings while preserving compression efficiency. Therefore, a tunable parameter λ and a threshold τ are introduced to balance the trade-off between compression/quality and energy efficiency. Experimental results show that LiveESTR reduces the encoder and decoder energy consumption by 74.6% and 29.7%, with only a 2.1% increase in Bjøntegaard Delta Rate (BD-Rate) compared to traditional per-title encoding. Furthermore, it is shown that by increasing λ to prioritize video quality, LiveESTR achieves 2.2% better compression efficiency in terms of BD-Rate while still reducing decoder energy consumption by 7.5%.

The 142nd MPEG meeting was held as a face-to-face meeting in Antalya, Türkiye, and the official press release can be found here and comprises the following items: •MPEG issues Call for Proposals for Feature Coding for Machines •MPEG finalizes the 9th Edition of MPEG-2 Systems •MPEG reaches the First Milestone for Storage and Delivery of Haptics Data •MPEG completes 2nd Edition of Neural Network Coding (NNC) •MPEG completes Verification Test Report and Conformance and Reference Software for MPEG Immersive Video •MPEG finalizes work on metadata-based MPEG-D DRC Loudness Leveling The press release text has been modified to match the target audience of ACM SIGMM and highlight research aspects targeting researchers in video technologies. This column focuses on the 9th edition of MPEG-2 Systems, storage and delivery of haptics data, neural network coding (NNC), MPEG immersive video (MIV), and updates on MPEG-DASH.