Predição de Light Fields utilizando técnicas de aprendizado profundo

Abstract

The increasing use of multiple cameras has led researchers to revisit the theory of Light Fields, which captures spatial and angular information, increasing the amount of stored data and requiring advanced compression techniques. Some approaches compress Light Fields by reorganizing them into pseudo-temporal sequences or using JPEG-Pleno, but the lack of block-level prediction or conversion to 3D may reduce compression efficiency, creating opportunities to explore block-level predictions that better leverage the 4D structure. Furthermore, recent research has achieved promising results by employing machine learning algorithms, such as convolutional neural networks, to perform prediction in both image and video encoders. However, there are numerous training techniques and convolutional neural network architec tures, and their performance varies significantly depending on the task and data type. Thus, this work proposes a method for training convolutional neural networks capable of adapting intra prediction in video encoders to exploit the angular and spatial redundancies present in Light Fields. All evaluation and development steps throughout the workflow were meticulously analyzed, providing a detailed explanation of the objectives of each technique, as well as their failures and successes. The EVC reference software was used to evaluate various architectures, such as autoencoders, Highway networks, and Residual networks, under different kernel configurations and hyperparameters, including data augmentation and learning rate decay methods. Another experiment conducted was the comparison of the SATD metric with MSE and SAD as loss functions during training. Additionally, structured and unstructured pruning techniques were evaluated to improve the efficiency of the trained models. At the end of the experiments, the resulting predictors consisted of approximately 1.3M parameters and, when validated, achieved a BD-Rate of -40.95% for the HM encoder and -46.89% for the VTM encoder. When validated by predicting the encoding of a second dataset, they achieved an efficiency improvement of -30.09% in the VTM encoder. Furthermore, the predictors proved to be competitive with the state of the art in Light Field compression, outperforming related works by approximately -20%.