Modeling and Information Processing
SR - Super-resolution
Modeling images and evaluating performance
The purpose of super-resolution is to improve the resolution of a sequence of images from a moving sensor. This mainly involves countering the image aliasing caused by the sub-sampling of the focal plane relative to the optical cut-off frequency [1]. What these techniques have in common is that they combine several images using the sub-pixel variations of view taken to separate the aliased frequencies.
The aim of the work done at the DTIM is to quantitatively evaluate these techniques' performances in order to define, in the long run, the macroscopic conditions in which super-resolution (SR) has advantages.
Modeling the performance of super-resolution is a relatively unexplored field [2] in which, to our knowledge, no satisfactory work has been done. The difficulty stems from the interlacing of two estimation problems, one radiometric (the super-resolution itself) and the other geometric (estimating the inter-image offsets). Any general solution proposed must solve the following points:
- lack of knowledge about the impact of the quality of geometric estimate on the radiometric estimate,
- the fact that the qualities of the two types of estimates depend on the scene observed,
- the fact that there are numerous concurrent techniques for the two problems.
The work done up to now at Onera in this context has defined and calculated simple and intuitive performance indicators, so as to answer the original SR question: "how many images do I need to double the resolution?". To do this, we quantitatively compared the performance of SR processing using a variable number of images coming from a single unit resolution sensor (sampling frequency 1) and the performances of a sensor with better resolution (i.e. sampling frequency > 1) without SR. In these studies, the performance measurement used is the mean quadratic error in estimating a texture on the sensor input.
Recent work has consisted of extending this work in two directions:
- consideration of performance measurements linked to the accuracy of primitive location,
- consideration of model errors making the processing sub-optimal.
Sub-pixel primitive location
This part of the work was done in collaboration with the L2TI ( Paris XIII).
Our work consisted of modeling a multi-temporal contour location algorithm in a way that allowed us to evaluate the mean quadratic error of contour location as a function of the contour and sensor characteristics. In particular, we were interested in the change in this error when the number of signals available increased or when the resolution of the sensor increased. An example is given in Figure 1.
Figure 1 - The combination of these two curves gives a representation of the increase in resolution as a function of the number of signals used.
Taking model error into account
Any error in modeling results in sub-optimal processing. Our techniques allow us to evaluate the corresponding optimal loss to a certain extent. We examined the case of bias in the resetting in SR. This case is illustrated by Figure 2.
Figure 2 - The green curve represents the mean quadratic error in estimating the signal as a function of the resetting bias, minimal when the bias is null. The dotted blue curve represents the estimating error obtained when a single signal is used. We define the maximum tolerated bias as the bias value such that the MQE rejoins the blue curve, rendering the SR useless.
Publications
[1] R. R. Schultz et R. R. Stevenson,Extraction of high-resolution frames from video sequences, IEEE Trans. Image Processing, 5 (6) : 996-1011, juin 1996.
[2] S. Baker et T. Kanade, Limits on Super-Résolution and how to break them, IEEE Trans. Pattern Analysis and Machine Intelligence, 6 (12) : 1621-1633, 2002.
[3] R. Mardia, The link between Kriging and thin-plate splines, in Probality, Statistics and Optimization, Ed. F.P. Kelly, pp326-339, 1994, Wiley
