We are working on the digital restoration of missing parts in damaged paintings. This is part of an ongoing project on the virtual restoration of the 3600 years old wall paintings excavated in the pre-historic Aegean settlement in Akrotiri, Thera, Greece.

As part of this work we have acquired hundreds of photographs of several wall paintings preserved at the Akrotiri Thera excavation site, which we have used to create ultra high resolution mosaics of each composition. For filling in the missing parts we have developed digital image inpainting techniques, relying on either physical diffusion processes or statistical wavelet-domain image models.

Our interest in image inpainting is motivated by the problem of digital restoration of the ancient wall-paintings discovered in the pre-historic settlement of Akrotiri, Thera, some representative examples of which are shown in Figure 2.

The Theran wall paintings are famous for their thematic gamut, their artistic value, and the abundant and diversified information they yield about the Aegean world more than 3500 years ago. Restoring these unique 17th century B.C. wall-paintings is a particularly challenging and time consuming process, requiring the painstaking effort of several specially trained conservators working under expert archaeological guidance. The restoration process comprises, among others, gathering the plaster fragments from the archaeological site, cleaning and restoring each fragment in the lab, and reassembling each composition from its constituent fragments. The main goal of our work is to virtually restore the reassembled wall paintings by filling in the missing parts.

Figure 2. Overview of the wall paintings we photographed. Their height typically ranges from 1.50 to 2.75 meters. Since they are too large to be captured as a whole in sufficient resolution, we acquired several photographs for each wall painting, each snapshot focusing on a part of the composition.

For digitally restoring the art-work we need to capture it as closely as possible in high-quality digital photographs. A significant difficulty with the Theran wall paintings is that they are very large, each composition typically covering an area of several square meters. Their big size makes it impossible to photograph them in sufficient detail in a single snapshot using conventional equipment.

Our approach has been to divide each composition into several small overlapping patches and photograph each of them in sufficient detail by placing the camera at a quite close distance. Using interest point detection and description methods we have been able to semi-automatically align all photographs corresponding to a particular wall painting as shown in Figure 3. Similar alignment techniques are very popular for creating panoramic images of scenery. However in our case several factors differ, for example, the geometry of the painting surface is flat and the camera position is non-constant. The image alignment step is followed by image blending, yielding the final ultra high resolution mosaic which captures the whole composition in sufficient detail.

Figure 3. We align all photographs of each wall painting and stitch them to create a single high resolution mosaic covering the whole composition. The 250 M-pixel mosaic in this figure captures the 5 sq. meter “Crocus Gatherer & Potnia” wall painting with resolution 200 pixels/inch.

The resulting mosaics are typically hundreds of M-pixel large and their file size makes them difficult to manipulate and share. To make it easier to use them we have set up a system based on IIPImage which allows viewing coarse scale instances or zooming into details of the mosaics through a web interface, as shown in Figure 4. The viewer is accessible here (requires authentication).

Figure 4. Zooming in the image mosaics with a web-based image viewer.

We have developed digital image inpainting techniques for filling in the missing parts of the wall paintings. We have explored two broad classes of techniques, the one based on physical diffusion processes and the other based on statistical wavelet-domain image modeling.

In diffusion-based techniques for image inpainting one tries to propagate structure and color from the known image areas into the missing parts. To ensure good edge continuation of image edges the diffusion process is designed to be of anisotropic nature and formulated as a nonlinear anisotropic diffusion partial differential equation (PDE). For efficiently solving numerically the resulting PDE we employ multigrid techniques, which are closely related to the algorithms we have previously developed for solving similar PDEs arising in the context of image segmentation (see here). An indicative inpainting result of this technique is shown in Figure 1.

We are also working on another class of techniques which considers image inpainting as a missing data inverse problem. For solving it in a Bayesian framework one needs to adopt an appropriate probabilistic prior model to regularize the inversion process. Multiscale image descompositions such as the wavelet transform present several advantages as an image representation on which one can build sufficiently accurate yet tractable prior image models. In our research we have combined an over-complete complex-wavelet image representation, which ensures good shift invariance and directional selectivity with a discrete-state/continuous-observation hidden Markov tree model for the wavelet coefficients, which captures key statistical properties of natural image wavelet responses, such as heavy-tailed histograms and persistence of large wavelet coefficients across scales. We have explored alternative deterministic and Markov chain Monte Carlo algorithms for image inpainting under this multi-scale image model. An inpainting example using our technique is shown in Figure 5. We are currently working into extending our transform domain inpainting model in two main directions: (a) using multiple dictionaries for compactly representing heterogeneous image layers (e.g. texture) and (b) integrating into our probabilistic model constraints for the phase of the complex wavelet transforms.

Figure 5. Image inpainting example using the wavelet domain hidden Markov tree model. (a) Left: the missing parts are annotated. (b) Right: the digitally restored result.

CVSP group people involved:

• George Papandreou
• Petros Maragos

• G. Papandreou,
  Image Analysis and Computer Vision: Theory and Applications in the Restoration of Ancient Wall Paintings,(in greek)
  ECE School PhD Thesis, 2009. Thesis supervisor: Prof. P. Maragos.
  [pdf] [bib]
• G. Papandreou, P. Maragos, and A. Kokaram,
  Image Inpainting with a Wavelet Domain Hidden Markov Tree Model,
  Proc. IEEE Int’l Conference on Acoustics, Speech, and Signal Processing (ICASSP-2008), pp. 773-776, Las Vegas, NV, U.S.A., Mar.-Apr. 2008.
  [pdf] [bib]
• G. Papandreou and P. Maragos,
  Multigrid Geometric Active Contour Models,
  IEEE Transactions on Image Processing, vol. 16, no. 1, pp. 229-240, Jan. 2007.
  [pdf] [bib] [software] [Read more…]