Image Inpainting: From Pixel Propagation to Controllable Generation

Image inpainting fills missing or selected regions while keeping the result consistent with the visible image. Its capabilities have expanded through four approaches: extending nearby pixels, borrowing existing textures, learning image semantics, and generating content according to user intent. These approaches continue to coexist because they address different problems. This guide covers the representative research from 1999 to 2025 — the main approaches, the problems they solve, and how to evaluate them.

中文版：图像修复的完整脉络（中文）。

1. What does image inpainting repair?

This page focuses on image completion with a known mask. Inputs typically include an image I, a mask M, and optional text, structural guidance, or reference images. Here M=1 means the region to fill and M=0 means the region to preserve. If pixels outside the mask must remain unchanged, the original image and prediction Î can be composited in pixel space:

I_out = (1 − M) ⊙ I + M ⊙ Î

This describes output compositing; it does not mean every model's raw output preserves known pixels. A feathered mask creates a blending band, so the permitted area of change must be defined explicitly.

Completion often has multiple valid answers. A model may produce a natural-looking face without recovering the original face. Evaluation should distinguish similarity to ground truth, visual plausibility, and compliance with the intended edit.

2. How the approaches connect

The diagram below maps how ideas and problems relate. Arrows do not establish direct inheritance between every paper, and newer approaches do not simply replace older ones.

For any method, ask six questions: What gets filled first? Where does content come from? How are candidates selected? What maintains global structure? Can there be multiple answers? How does the user control the result?

3. 1999–2004: Pixel continuation and texture synthesis

Early research addressed two different needs: scratches require nearby colors and contours to continue, while larger missing areas require suitable texture material.

Structure propagation emphasizes contour and isophote continuity. Bertalmío and colleagues' 2000 work applied this idea to digital inpainting. Telea's 2004 method used fast marching to estimate pixels progressively from the boundary inward. These methods establish the foundations of small-region repair, but local propagation cannot infer arbitrary missing semantic content. Bertalmío 2000, Telea 2004

Texture synthesis seeks reusable material. Efros and Leung (1999) sampled pixels using similar neighborhoods. Image Quilting (2001) assembled patches and handled their overlap seams. Both contributed the idea of searching and combining existing texture. Non-parametric texture synthesis, Image Quilting

The shared limitation is that usable local color and texture do not establish the correct scene semantics. Also distinguish early PDE information diffusion from modern generative diffusion: the latter samples content through a learned denoising process.

4. 2003–2015: Exemplar patches combine structure and texture

Criminisi and colleagues' 2003/2004 work combined fill order with patch copying: identify the hole boundary, prioritize boundary patches, find a similar patch in the known region, copy its pixels into the unknown area, and update the mask and confidence. Criminisi 2003, Criminisi 2004

The classic priority is P(p) = C(p) · D(p). Confidence C describes the amount of reliable information, while the data term D emphasizes structure entering the hole. This framework addresses both fill order and the source of filling material.

Incorrect matches can propagate, similar colors can hide incorrect structure, and missing source material limits semantic recovery. Later work tackled these issues separately.

PatchMatch is primarily a matching algorithm. A complete inpainting system also needs reconstruction, mask handling, and consistency constraints.

5. 2016–2019: Learning to generate missing semantics

When the source image contains no usable eye, window, or animal contour, completion needs information beyond that image. Deep learning supplies priors learned from training examples.

Context Encoders (2016) predicted missing content with an encoder–decoder and reconstruction and adversarial objectives. Globally and Locally Consistent Image Completion (2017) used two discriminators to assess overall coherence and local realism. Context Encoders, Global and local consistency

Contextual Attention (2018) explicitly borrowed deep features from visible regions. This connects learned generation with exemplar matching in feature space. Contextual Attention

The capability expanded from copying available material to combining visible evidence with learned priors. Outputs consequently depend on the training distribution and may look natural while changing the original object's shape.

6. 2018–2020: Masks, structure, detail, and multiple answers

Standard convolution processes valid pixels alongside placeholder values inside holes. This creates a need to distinguish trustworthy information.

These branches address input validity, shape, detail, and uncertainty. They can be combined rather than treated as mutually exclusive categories.

7. 2021–2023: Global context, large holes, and high resolution

Attractive texture can coexist with misaligned windows and broken lines. Local features alone may not adequately constrain the entire scene.

Older concerns such as structure-first reconstruction and borrowing visible material persist through new representations. High resolution also requires more than a large output file: check internal resizing, retained detail, long-range consistency, and seams. Of these convolutional networks, LaMa has become the default open model — and the strongest one that fits a browser: try it in our LaMa inpainting tool.

8. 2022–2025: Diffusion expands completion into controlled editing

Generative diffusion produces content through iterative denoising. Inpainting requires the sampling procedure or network to use known image regions, masks, and other conditions.

RePaint (2022) constrained sampling from a pretrained unconditional diffusion model and used resampling to improve compatibility with known content. Latent Diffusion Models (2021 preprint / CVPR 2022) moved diffusion into a compressed latent space to reduce computation and support conditional generation, including inpainting. RePaint, LDM

Later work focused on expressing intent, injecting conditions, and reducing cost.

Removing a person calls for background completion; adding a chair calls for a specified object. Identical masks can therefore represent different tasks. Evaluation must consider contextual consistency, instruction adherence, and known-region preservation. Diffusion's practical value depends on the task, latency, control, detail requirements, and cost of failure.

9. Video inpainting: more evidence and temporal constraints

A background hidden in one frame may be visible in another. Video completion should exploit that evidence before generating areas without available observations.

The field progressed from spatiotemporal patch matching to optical-flow propagation, cross-frame attention, and combinations of propagation and Transformers. ProPainter (ICCV 2023) combines flow completion, pixel and feature propagation, and a mask-guided sparse Transformer. ProPainter

Natural-looking individual frames can still flicker or drift during playback. Evaluation must also cover flow errors, changing occlusions, scene cuts, and long-term temporal consistency.

10. Choose an approach by the task

A practical workflow aligns image orientation and mask dimensions, confirms mask polarity, selects the intended edit, runs completion, preserves or blends known pixels, inspects boundaries and structure, and exports the result. Missing shadows or reflections in the mask can leave visible traces even when the algorithm runs correctly.

11. Evaluate the hole, known region, and boundary separately

Whole-image averages can hide errors because unchanged pixels dominate the result.

A useful experiment fixes images, mask shapes and areas, resolution, and model settings; includes small holes, long lines, repeated structures, faces, and large objects; records regional metrics and visual outputs; retains failures; and documents seeds, sample counts, and candidate-selection rules for stochastic methods. These dimensions form a synthesized evaluation framework — they should not collapse into an unexplained quality score. RePaint also discusses the limits of comparing diverse outputs with one ground-truth image.

12. A reading path — and where this site fits

Read by the problem being solved:

• Pixel continuation: Bertalmío and Telea.
• Sources of texture: Efros and Leung, Image Quilting, and Criminisi.
• Reliable search and composition: structure propagation, global optimization, PatchMatch, Patch Sparsity, and Patch Offsets.
• Missing source material: Context Encoders and global/local consistency.
• Visible feature reuse: Contextual Attention, Partial/Gated Convolution, and PEN-Net.
• Shape and uncertainty: EdgeConnect and Pluralistic Image Completion.
• Global relationships and information loss: LaMa, MAT, ZITS, and PUT.
• Intent and generation cost: RePaint, LDM, PowerPaint, BrushNet, and TurboFill.
• Cross-frame evidence: video completion and ProPainter.

The closest branch to the algorithm this site runs is the exemplar chain: Criminisi → candidate representation and scoring → search efficiency and global consistency → deep feature borrowing. Our implementation is the 2016 feature-priority refinement of Criminisi's method — a classical, non-generative technique that only borrows real pixels from your image, which is why it runs privately in your browser. Where this branch ends — semantic gaps with nothing to borrow from — is exactly where the learned and diffusion approaches above take over.

The field's development connects three requirements: use visible evidence effectively, introduce plausible priors where evidence is missing, and align completion with user intent. Modern models extend these capabilities while classical methods remain useful foundations for understanding structure, texture, matching, and error propagation.