The Inpainting Algorithm Behind This Tool

The tool on this site is not a neural network — it runs a patch-matching inpainting algorithm in your browser. Its lineage is two papers: Criminisi's 2003 exemplar-based method [1], and the 2016 feature-priority refinement by Li, Wang, Liu & Wang [2] that fixes the classic method's most damaging failure mode. This page traces that path, using figures from the 2016 paper (reproduced with translated sub-captions).

2003 — Criminisi's exemplar-based inpainting

Earlier diffusion-based methods smeared pixels inward like heat. Criminisi, Perez and Toyama instead treated the hole as a texture synthesis problem: fill it one patch at a time, copying real pixels from elsewhere in the image [1]. Each patch on the fill front gets a priority:

P(p) = C(p) × D(p)

where C(p) — the confidence term — measures how much reliable, already-known data surrounds the patch, and D(p) — the data term — rewards patches sitting on strong continuous structures (edges, contours). The highest-priority patch is filled first: the algorithm searches the known region for the most similar patch and copies its missing pixels. Confidence is updated, and the loop repeats until the hole is gone.

This one idea — fill in the right order, borrow real pixels — still powers classic inpainting today. But it has a flaw that gets worse as the hole grows.

The framework both versions share

Both papers describe the image the same way: Ω is the target (missing) region, Φ the known source region, δΩ the fill front, and Ψ_p the patch centered on a front pixel p, whose surface normal n_p meets the isophote (perpendicular to the gradient, ∇I_p^⊥) in the data term:

Where the classic method breaks

Every filled pixel gets the confidence of the patch that filled it — always less than 1. So as filling progresses, confidence values on the front keep decaying toward zero, and since P(p) = C(p) × D(p), the priority formula eventually stops caring about the data term at all. The paper calls this the effect-loss problem [2]: the fill order degenerates, errors accumulate, and structures erode. In the worst case no suitable match is found at all and the result visibly distorts.

2016 — the feature-priority fix

Li, Wang, Liu and Wang published the fix in the Journal of Computer-Aided Design & Computer Graphics [2]. The fill loop and the search stay the same; only the priority changes:

P(p) = (C(p) − λ + w·|∇I(p)|) × D(p)

• C(p) − λ subtracts a constant (λ = 0.35 in our implementation) from the confidence term, so it no longer collapses toward zero — the priority formula keeps working deep into the hole.
• w·|∇I(p)| adds the normalized gradient magnitude at the front pixel itself: patches sitting on a strong, identifiable feature (a wire, a roof line, a boundary) get filled earlier, while there is still reliable structure to match against. That cuts the error rate of the matching step.

The paper also analyzes how a wrong best-match patch triggers chains of consecutive wrong fills, and shows the mechanism for locating the best source patch Ψ_q in Φ — the step both versions share:

Real object removal, side by side

The paper's most convincing experiment removes a bungee jumper over water — one of the classic test images from the 2003 paper:

This is the exact method this site runs — ported line for line from the paper's formulation, with the Criminisi version kept as a baseline for verification. Watch it work on your own photo:

What neither version can do

Both algorithms are non-generative: they only ever borrow real pixels from your image. Repetitive texture (grass, walls, water) and thin structures fill cleanly; a hole over a face or a chaotic crowd has nothing to borrow from, and that is the job for generative AI models — at the cost of uploading your photo and a plausible-but-invented result. For where this branch sits in the wider field, see the complete map of inpainting techniques from 1999 to 2025.