Originally published at https://bestlatestapps.com/blog/background-removal-in-the-browser-u2net-onnx/.
How my background remover turns a 320×320 saliency mask into a full-resolution cut-out with soft edges, entirely in a Web Worker, with WebGPU when it's there and WebAssembly when it isn't
Background removal is one of those features that feels like it needs a server: a big model, a GPU, an upload. For a lot of photos, it doesn't. A small segmentation network can run in a browser tab, and most of the quality people notice, clean edges and hair that doesn't look cut out with scissors, comes from what you do after the network has run.
This article walks through the background remover on my site using its real source: p2-aiimage.js (the page), p2-aiimage-worker.js (inference) and p2-aiimage-core.js (the image maths). Snippets are taken or trimmed from those files. I'll be clear where something is a heuristic, and about the kinds of photo it handles badly.
1. The constraints that shaped everything
Running a model in a page means paying for it in download size, memory and the main thread's responsiveness. The tool's choices follow from that:
- The model is U2-Net's small variant,
u2netp, by Xuebin Qin and colleagues, released under Apache-2.0. The ONNX file the page downloads is about 4.4 MB. The full-size U2-Net is far larger; for a free web tool, the small one is the right trade. - The runtime is ONNX Runtime Web (MIT licence), served from the site as gzipped WebAssembly: about 3.6 MB for the CPU build, or about 6.8 MB for the WebGPU build. Both numbers come from the files the page serves, and the page states them.
- Both are downloaded once and kept in the browser's Cache Storage, so the second photo starts immediately.
- All inference runs in a Web Worker, so the page stays scrollable and the Stop button stays responsive while the model works.
- The photo stays in the tab. Its pixels are copied to the worker with
postMessage; the tool's code doesn't upload them. The only network requests the tool makes are for the model and runtime files themselves.
2. Choosing WebGPU or WebAssembly, and falling back
Before starting the worker, the page asks whether the browser has a usable GPU adapter:
`` javascript
function chooseEp() {
if (!navigator.gpu || typeof navigator.gpu.requestAdapter !== "function") return Promise.resolve("wasm");
return navigator.gpu.requestAdapter().then(function (adapter) {
return adapter ? "webgpu" : "wasm";
}, function () { return "wasm"; });
}
CODEBLOCK0 javascript
function gunzip(bytes) {
if (typeof DecompressionStream !== "function") return Promise.reject(new Error("runtime"));
var stream = new Blob([bytes]).stream().pipeThrough(new DecompressionStream("gzip"));
return new Response(stream).arrayBuffer().then(function (buf) {
var out = new Uint8Array(buf);
if (out.length < 8 || out[0] !== 0 || out[1] !== 97 || out[2] !== 115 || out[3] !== 109) throw new Error("bad-wasm");
return out;
});
}
CODEBLOCK1 javascript
return createImageBitmap(file, { imageOrientation: "from-image", premultiplyAlpha: "none" }).catch(function () {
return createImageBitmap(file, { imageOrientation: "from-image" });
}).catch(function () { return createImageBitmap(file); }).catch(function () { return viaImg(); });
CODEBLOCK2 javascript
var MEAN = [0.485, 0.456, 0.406], STD = [0.229, 0.224, 0.225];
// U2-Net input: rect of an RGBA image squeezed to size×size, scaled by its maximum, ImageNet mean and std.
function u2Input(rgba, w, h, rect, size) {
var px = resize(rgba, w, h, 4, 3, rect, size, size), n = size * size, max = 1e-6, i;
for (i = 0; i < px.length; i++) if (px[i] > max) max = px[i];
var out = new Float32Array(3 * n);
for (i = 0; i < n; i++) {
for (var c = 0; c < 3; c++) out[c * n + i] = (px[i * 3 + c] / max - MEAN[c]) / STD[c];
}
return out;
}
CODEBLOCK3 javascript
return inferMask(sess, rgba, w, h, null).then(function (full) {
var box = A.maskBox(full, 320, 320, 0.2);
var crop = box ? A.cropAround(box, 320, 320, w, h) : null;
var zoomP = crop && A.worthZoom(crop, w, h)
? (say("Refining the outline…", 0, 0, "run"), inferMask(sess, rgba, w, h, crop))
: Promise.resolve(null);
// ...
CODEBLOCK4 javascript
// Guided-filter coefficients that only act in the uncertain band around the coarse edge.
function refineCoeffs(I, p, w, h, opts) {
var n = w * h, box = boxer(w, h);
var r = opts.r || Math.max(2, Math.round(Math.max(w, h) / 100));
var g = guided(I, p, w, h, r, opts.eps || 1e-3, box), lo = new Float32Array(n), hi = new Float32Array(n), i;
for (i = 0; i < n; i++) { lo[i] = p[i] > 0.02 ? 1 : 0; hi[i] = p[i] > 0.98 ? 1 : 0; }
box(lo, r, lo); box(hi, r, hi);
for (i = 0; i < n; i++) {
var wt = Math.min(1, lo[i] * 8) * Math.min(1, (1 - hi[i]) * 8);
g.a[i] *= wt; g.a[i + n] *= wt; g.a[i + 2 * n] *= wt;
g.b[i] = wt * g.b[i] + (1 - wt) * p[i];
}
return g;
}
CODEBLOCK5 javascript
upRows(planes, w, h, W, H, function (y, rows) {
var ar = rows[0], ag = rows[1], ab = rows[2], bb = rows[3];
for (var x = 0, p = y * W * 4; x < W; x++, p += 4) {
var r = rgba[p] / 255, g = rgba[p + 1] / 255, b = rgba[p + 2] / 255;
var a = finishAlpha(ar[x] * r + ag[x] * g + ab[x] * b + bb[x]) * (rgba[p + 3] / 255), a8 = Math.round(a * 255);
// ...
}
});
CODEBLOCK6 javascript
if (a8 <= 0) { rgba[p] = rgba[p + 1] = rgba[p + 2] = rgba[p + 3] = 0; continue; }
if (fg && a8 < 255) {
var f0 = rows[4][x], f1 = rows[6][x], f2 = rows[8][x], u = 1 - a;
rgba[p] = clamp(f0 + a * (r - a * f0 - u * rows[5][x]), 0, 1) * 255;
// same for green and blue
}
rgba[p + 3] = a8;
`
Fully transparent pixels also have their colour zeroed. That isn't cosmetic: it means the PNG doesn't secretly contain the old background in invisible pixels, which some editors reveal when you change blend modes.
These steps are pure functions in the core file and have Node tests, including one that runs the actual u2netp model on a real portrait and asserts that the background corner comes out nearly transparent and the subject's body comes out nearly opaque.
------------------------------------------------------------------------
## 10. New backgrounds without re-running the model
Once the cut-out exists, the background options (Transparent, White, Black, a colour, or Blur) are all drawn from it on the page. Changing them never re-runs the model.
Blur is the interesting one. If you just blur the original photo, the subject smears into the background and leaves a halo. The core first fills the subject's area from its surroundings, using a normalised convolution (a background-weighted blur divided by the blurred weights), and only then blurs the result, three box-blur passes in a row, which approximates a Gaussian. The sharp cut-out is drawn on top. It runs on a copy of at most 720 pixels on the long side and is scaled up, since a blurred background doesn't need detail.
Saving a JPG with a transparent background flattens onto white, and the status line says so; the PNG keeps the transparent edges.
------------------------------------------------------------------------
## 11. Where it struggles, honestly
U2-Net is a *salient object* model: it finds the thing a person would look at first. That's ideal for one person or one product, and it explains the failures the page warns about:
- **Crowds and scenes without one subject** can keep only the most prominent region.
- **Openings**, such as the gap in a mug handle or under an arm, can keep a little background.
- **Very faint results** are detected: if under 1% of pixels end up mostly opaque, the status line suggests a photo with a single clear subject instead of silently handing you an almost empty PNG.
- **Speed varies a lot by device.** The status line reports how long the run took, and says "graphics chip" or "processor". That label is an estimate from how fast the first inference ran (under 400 ms counts as the graphics chip), not a readout from the runtime. I don't quote timings here because they depend entirely on the hardware.
------------------------------------------------------------------------
## 12. Shipping on-device ML on the web: a checklist
1. **Pick the smallest model that's good enough**, then spend effort on post-processing.
2. **Run inference in a worker**, and use a job token so a cancelled or superseded run never paints over a newer one.
3. **Prefer WebGPU when an adapter exists**, and fall back to WebAssembly at every failure point.
4. **Verify downloaded binaries** (the \0asm` magic) and cache them in a versioned cache.
- Respect EXIF orientation and fit large photos to a memory budget before processing.
- Match the model's training preprocessing exactly.
- Refine edges with a guided filter only where the mask is uncertain, and upsample coefficients, not the alpha.
- Estimate foreground colours at soft edges, and zero the colour of fully transparent pixels.
Wrapping up
The network is the smallest part of this tool by line count. A 4.4 MB model gives a rough idea of where the subject is; a zoomed second pass, a guided filter confined to the uncertain band, coefficient upsampling and foreground colour estimation turn that rough idea into a cut-out you'd put in a presentation. All of it runs in a worker in your browser tab, on the GPU when there is one.
Try it, or browse the related tools:
- Background remover: https://bestlatestapps.com/tools/background-remover/
- Upscale image (same worker, Real-ESRGAN): https://bestlatestapps.com/tools/upscale-image/
- Passport photo maker: https://bestlatestapps.com/tools/passport-photo-maker/
- All of my free web tools: https://bestlatestapps.com/tools/
- My portfolio: https://bestlatestapps.com/alok/
Portraits with clear subjects work best today; busy scenes and fine, low-contrast edges are where I expect the next improvements to come from.
— Alok Kumar Singh