Replicating noise in video : a comparison between physics-based and deep learning models for simulating noise

Sammanfattning: Algorithms that track objects in video following Newtonian physics can often be affected by noise in the data. Some types of noise might be hard or expensive to capture, so to be able to augment or generate a new data set from models replicating a certain type of noise can be useful. Recent research into unsupervised learning of video sequences using Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) combined with convolutions (ConvLSTM) gives hope of a deep learning model which can when trained on a certain type of noise, replicate the properties of that noise without copying the exact data. This thesis takes two data sets of different noises (rain and moving insects) and attempts to replicate these. For comparison, two models are created for each noise. One is physicsbased and focuses on creating a specific model of a noise for simulation and generation. The other is a deep learning network trained on data captured in real-life representing each noise. Generated sequences from the models are then evaluated using different measurements and compared to a validation set, using both established techniques such as Fréchet Inception Distance (FID) and new ones created to show the difference for this type of sparse data. The result shows that it is difficult to measure such a sparse data set using existing techniques. FID scores for insect models compared to a validation set, are almost equal (103 ≈ 107). However, this is not consistent with a visual inspection of the data, which shows the deep learning model performing worse. Similar results can be seen for the rain models, which makes FID scores difficult to interpret since it does not match a visual inspection. New measurements techniques show the difference between data sets created with a physics-based model and a deep learning model, but their generalization are questioned. The conclusion is that the physics-based models perform better than the deep learning models, however, they do not generalize as well and takes a considerable effort to produce.