Prism

Feed-forward multi-view inverse rendering — joint neural-SDF geometry, physically-based materials, and lighting from sparse views.

2026 Python · PyTorch github.com/osbo/prism

Prism

Feed-forward multi-view inverse rendering: joint neural-SDF geometry, Cook-Torrance GGX materials, and a point-light + ambient illumination model recovered from sparse RGB views in a single forward pass.

Unlike radiance-field-only pipelines, Prism keeps geometry and appearance physically separable by coupling an implicit surface to differentiable NeuS-style rendering, then training the surface, the BRDF, and the light against the input photographs jointly.

How it works

  1. Image encoding (prism/encoder.py) — a ResNet-34 extracts shallow per-view feature maps and a global latent code summarizing the scene.
  2. SDF (prism/sdf_mlp.py) — for each 3D query point, projected per-view features and the global latent condition an MLP that outputs the signed distance. The lower MLP supplies geometry; higher levels supply local appearance context.
  3. Material + lighting heads — small MLPs decode Cook-Torrance GGX parameters (albedo, roughness, metalness) at the predicted surface, and the scene's point-light position, intensity, and ambient term.
  4. Differentiable rendering (prism/renderer.py) — NeuS importance-sampled ray marching produces expected hit points; brdf.py shades them under the predicted light using the GGX BRDF.
  5. Multi-objective loss (prism/losses.py) — photometric (L1/SSIM), depth, normal, silhouette / free-space, SDF sign / band, eikonal, and visual-hull terms. Loss weights and ablation switches live in config.py.

Results

On 19 held-out OmniObject3D objects across 6 categories:

  • Chamfer distance: 0.209 ± 0.043 (lower is better)
  • Foreground PSNR: 14.55 ± 2.52 dB (higher is better)

Loss ablations show the largest PSNR impact from removing photometric supervision (−4.48 dB), then depth (−1.14 dB), with smaller contributions from normal (−0.23 dB) and eikonal (−0.08 dB).

Five input views used for clock_029:

clock_029 input views

Extracted mesh rendering (clock_029):

clock_029 reconstructed mesh

Stack

  • Python — PyTorch, NumPy
  • Rendering — custom NeuS-style importance sampler + GGX shading
  • Data — OmniObject3D (Blender renders + raw scans)

Running it

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# train from scratch
python train.py --data_root /path/to/omniobject3d/extracted

# resume from default checkpoint (model.pt)
python train.py --resume

# overfit one object for debugging
python train.py --overfit --overfit_object bottle_001

# quantitative evaluation
python evaluate.py --checkpoint model.pt --out_dir eval_results/metrics

Aggregate and per-object metrics are written to eval_results/metrics/metrics.json.

Reports

  • Final_Project_Report.pdf — full write-up
  • PRISM_architecture.md — architecture sketch

Limitations

  • Spatially uniform BRDF per object (no SVBRDF maps yet).
  • Point-light approximation cannot capture HDRI environment lighting.
  • Thin structures are limited by ray-sampling budget; deeper hierarchies or longer training help.

License

MIT.