Add GeoLangSplat: training-free open-vocabulary 3D segmentation for Geospatial Gaussian Splats#64
Open
amrik05 wants to merge 1 commit into
Open
Add GeoLangSplat: training-free open-vocabulary 3D segmentation for Geospatial Gaussian Splats#64amrik05 wants to merge 1 commit into
amrik05 wants to merge 1 commit into
Conversation
…aussian splats
GeoLangSplat segments a Gaussian splat from a text prompt with no per-scene
training: it runs SAM 3 over views of the splat and lifts the 2D scores into 3D
with fVDB's alpha-weighted back-projection, returning one label per Gaussian in
input .ply order. A single engine handles aerial, satellite, and object/interior
captures; an `auto` recipe reads the scene geometry and synthesizes its own views.
* segment() / `gls segment` -- prompt in, segmented .ply / overlay / report out
* segment catalog -- run a vocabulary, cluster each prompt into objects, browse
and extract them as per-object .ply (notebook-friendly; `gls catalog`)
* one-shot low-VRAM streaming path plus an optional warm engine (`gls serve`)
* CPU-only test suite (SAM 3 and fVDB mocked), black-formatted, Apache-2.0
Lives under open_vocabulary_segmentation/ alongside langsplatv2.
Signed-off-by: akrishnakuma <akrishnakuma@nvidia.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
GeoLangSplat segments a Gaussian splat from a text prompt with no per-scene training. It runs
SAM 3 over synthesized (or real) views of the splat and lifts the 2D scores into 3D with fVDB's
alpha-weighted back-projection, returning one label per Gaussian in input
.plyorder. A singleengine handles aerial (oblique drone), satellite (near-nadir), and object/interior (360° inward)
captures; an
autorecipe reads the scene geometry and synthesizes its own views, so the commoncase is
gls segment scene.ply "thing"with no flags. Lives underopen_vocabulary_segmentation/geolangsplat/alongsidelangsplatv2.Key Components:
segment()/gls segment— prompt in, per-Gaussian selection out (segmented.ply, overlay, report).build_catalog()/gls catalog— run a vocabulary in one pass, split each prompt into 3D objects(connected components), dedup across prompts by 3D IoU, and expose a pandas table + per-object
.plyextraction; notebook-friendly viacat.browse().assess_scene()/gls check/gls doctor— geometry-only readiness check (no weights needed).gls exploreviewer.Scope:
Reference implementation: single-GPU, VRAM-bound (no multi-GPU / tiling), best on site/block-scale
scenes. Exercised on SafetyPark (aerial drone) and the JAX_* WorldView-3 satellite scenes.
Test Plan:
pytest tests/— ~120 tests, CPU-only, SAM 3 and fVDB mocked, no GPUblack --check --target-version=py311 --line-length=120cleangls segmentandgls catalogwith per-object.plyexport