Merge pull request #67 from mkkellogg/feature/import-external-splat

Support external '.splat' format

README.md

@@ -2,7 +2,7 @@
 
 This repository contains a Three.js-based implemetation of a renderer for [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/), a technique for generating 3D scenes from 2D images. Their project is CUDA-based and needs to run natively on your machine, but I wanted to build a viewer that was accessible via the web.
 
-The 3D scenes are stored in a format similar to point clouds and can be viewed, navigated, and interacted with in real-time. This renderer will work with both the `.ply` files generated by the INRIA project, or my own custom `.splat` files, which are a trimmed-down and compressed version of those files.
+The 3D scenes are stored in a format similar to point clouds and can be viewed, navigated, and interacted with in real-time. This renderer will work with the `.ply` files generated by the INRIA project, standard `.splat` files, or my own custom `.ksplat` files, which are a trimmed-down and compressed version of the original `.ply` files.
 
 When I started, web-based viewers were already available -- A WebGL-based viewer from [antimatter15](https://github.com/antimatter15/splat) and a WebGPU viewer from [cvlab-epfl](https://github.com/cvlab-epfl/gaussian-splatting-web) -- However no Three.js version existed. I used those versions as a starting point for my initial implementation, but as of now this project contains all my own code.
 <br>
@@ -12,7 +12,8 @@ When I started, web-based viewers were already available -- A WebGL-based viewer
 - Rendering is done entirely through Three.js
 - Code is organized into modern ES modules
 - Built-in viewer is self-contained so very little code is necessary to load and view a scene
-- Allows user to import `.ply` files for conversion to custom compressed `.splat` file format
+- Viewer can import `.ply` files, `.splat` files, or my custom compressed `.ksplat` files
+- Users can convert `.ply` files to the `.ksplat` file format
 - Allows a Three.js scene or object group to be rendered along with the splats
 - Focus on optimization:
     - Splats culled prior to sorting & rendering using a custom octree
@@ -24,7 +25,7 @@ When I started, web-based viewers were already available -- A WebGL-based viewer
 - Splat sort runs on the CPU – would be great to figure out a GPU-based approach
 - Artifacts are visible when you move or rotate too fast (due to CPU-based splat sort)
 - Sub-optimal performance on mobile devices
-- Custom `.splat` file format still needs work, especially around compression
+- Custom `.ksplat` file format still needs work, especially around compression
 
 ## Future work
 This is still very much a work in progress! There are several things that still need to be done:
@@ -102,7 +103,7 @@ const viewer = new GaussianSplats3D.Viewer({
     'initialCameraLookAt': [0, 4, 0],
     'halfPrecisionCovariancesOnGPU': true,
 });
-viewer.loadFile('<path to .ply or .splat file>', {
+viewer.loadFile('<path to .ply, .ksplat, or .splat file>', {
     'splatAlphaRemovalThreshold': 5,
     'showLoadingSpinner': true,
     'position': [0, 1, 0],
@@ -141,11 +142,11 @@ Parameters for `loadFile()`
 `Viewer` can also load multiple scenes simultaneously with the `loadFiles()` function:
 ```javascript
 viewer.loadFiles([{
-            'path': '<path to first .ply or .splat file>',
+            'path': '<path to first .ply, .ksplat, or .splat file>',
             'splatAlphaRemovalThreshold': 20,
         },
         {
-            'path': '<path to second .ply or .splat file>',
+            'path': '<path to second .ply, .ksplat, or .splat file>',
             'rotation': [-0.14724434, -0.0761755, 0.1410657, 0.976020],
             'scale': [1.5, 1.5, 1.5],
             'position': [-3, -2, -3.2],
@@ -157,7 +158,7 @@ viewer.loadFiles([{
     });
 ```
 
-The `loadFile()` and `loadFiles()` methods will accept the original `.ply` files as well as my custom `.splat` files.
+The `loadFile()` and `loadFiles()` methods will accept the original `.ply` files, standard `.splat` files, and my custom `.ksplat` files.
 
 <br>
 
@@ -174,7 +175,7 @@ scene.add(boxMesh);
 const viewer = new GaussianSplats3D.Viewer({
     'scene': scene,
 });
-viewer.loadFile('<path to .ply or .splat file>')
+viewer.loadFile('<path to .ply, .ksplat, or .splat file>')
 .then(() => {
     viewer.start();
 });
@@ -190,11 +191,11 @@ const renderableViewer = new GaussianSplats3D.RenderableViewer({
     'gpuAcceleratedSort': true
 });
 renderableViewer.addScenesFromFiles([{
-            'path': '<path to .ply or .splat file>',
+            'path': '<path to .ply, .ksplat, or .splat file>'
             'splatAlphaRemovalThreshold': 5,
         },
         {
-            'path': '<path to .ply or .splat file>',
+            'path': '<path to .ply, .ksplat, or .splat file>',
             'rotation': [0, -0.857, -0.514495, 6.123233995736766e-17],
             'scale': [1.5, 1.5, 1.5],
             'position': [0, -2, -1.2],
@@ -237,7 +238,7 @@ const viewer = new GaussianSplats3D.Viewer({
     'ignoreDevicePixelRatio': false,
     'gpuAcceleratedSort': true
 });
-viewer.loadFile('<path to .ply or .splat file>')
+viewer.loadFile('<path to .ply, .ksplat, or .splat file>')
 .then(() => {
     requestAnimationFrame(update);
 });
@@ -262,19 +263,19 @@ Advanced `Viewer` parameters
 | `gpuAcceleratedSort` | Tells the viewer to use a partially GPU-accelerated approach to sorting splats. Currently this means pre-computing splat distances is done on the GPU. Defaults to `true`.
 <br>
 
-### Creating SPLAT files
-To convert a `.ply` file into the stripped-down `.splat` format (currently only compatible with this viewer), there are several options. The easiest method is to use the UI in the main demo page at [http://127.0.0.1:8080/index.html](http://127.0.0.1:8080/index.html). If you want to run the conversion programatically, run the following in a browser:
+### Creating KSPLAT files
+To convert a `.ply` file into the stripped-down and compressed `.ksplat` format, there are several options. The easiest method is to use the UI in the main demo page at [http://127.0.0.1:8080/index.html](http://127.0.0.1:8080/index.html). If you want to run the conversion programatically, run the following in a browser:
 
 ```javascript
 const compressionLevel = 1;
 const splatAlphaRemovalThreshold = 5; // out of 255
 const plyLoader = new GaussianSplats3D.PlyLoader();
-plyLoader.loadFromURL('<URL for .ply file>', compressionLevel, splatAlphaRemovalThreshold)
+plyLoader.loadFromURL('<path to .ply or .splat file>', compressionLevel, splatAlphaRemovalThreshold)
 .then((splatBuffer) => {
-    new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.splat');
+    new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.ksplat');
 });
 ```
-Both of the above methods will prompt your browser to automatically start downloading the converted `.splat` file.
+Both of the above methods will prompt your browser to automatically start downloading the converted `.ksplat` file.
 
 The third option is to use the included nodejs script:
 

demo/dropin.html

@@ -94,11 +94,11 @@
     const renderableViewer = new GaussianSplats3D.RenderableViewer();
     renderableViewer.addScenesFromFiles([
                                           {
-                                            'path': 'assets/data/garden/garden.splat',
+                                            'path': 'assets/data/garden/garden.ksplat',
                                             'splatAlphaRemovalThreshold': 20,
                                           },
                                           {
-                                            'path': 'assets/data/bonsai/bonsai_trimmed.splat',
+                                            'path': 'assets/data/bonsai/bonsai_trimmed.ksplat',
                                             'rotation': [-0.14724434, -0.0761755, 0.1410657, 0.976020],
                                             'scale': [1.5, 1.5, 1.5],
                                             'position': [-3, -2, -3.2],

demo/garden.html

@@ -38,7 +38,7 @@
       'halfPrecisionCovariancesOnGPU': true
     });
     let path = 'assets/data/garden/garden';
-    path += isMobile() ? '.splat' : '_high.splat';
+    path += isMobile() ? '.ksplat' : '_high.ksplat';
     viewer.loadFile(path)
     .then(() => {
         viewer.start();

demo/index.html

@@ -246,6 +246,10 @@
       return plyParser.parseToSplatBuffer(compressionLevel, alphaRemovalThreshold);
     }
 
+    function convertStandardSplatToSplatBuffer(bufferData){
+      return GaussianSplats3D.SplatLoader.parseStandardSplatToSplatBuffer(bufferData);
+    }
+
     window.onFileChange = function(arg, fileNameLabelID) {
       const fileNameLabel = document.getElementById(fileNameLabelID);
       const url = arg.value;
@@ -299,7 +303,7 @@
           window.setTimeout(() => {
             try {
               const splatBuffer = convertPLYToSplatBuffer(fileReader.result, compressionLevel, alphaRemovalThreshold);
-              new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.splat');
+              new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.ksplat');
               conversionDone();
             } catch (e) {
               conversionDone(e);
@@ -394,6 +398,10 @@
 
       const viewFileName = viewFile.files[0].name;
       const isPly = viewFileName.toLowerCase().trim().endsWith('.ply');
+      let isStandardSplat = false;
+      if (!isPly) {
+        isStandardSplat = GaussianSplats3D.SplatLoader.isStandardSplatFormat(viewFileName);
+      }
 
       currentAlphaRemovalThreshold = alphaRemovalThreshold;
       currentCameraUpArray = cameraUpArray;
@@ -403,7 +411,7 @@
         const fileReader = new FileReader();
         fileReader.onload = function(){
           try {
-           runViewer(fileReader.result, isPly, alphaRemovalThreshold, cameraUpArray, cameraPositionArray, cameraLookAtArray);
+           runViewer(fileReader.result, isPly, isStandardSplat, alphaRemovalThreshold, cameraUpArray, cameraPositionArray, cameraLookAtArray);
           } catch (e) {
             setViewError("Could not view scene.");
           }
@@ -440,7 +448,7 @@
       }
     });
 
-    function runViewer(splatBufferData, isPly, alphaRemovalThreshold, cameraUpArray, cameraPositionArray, cameraLookAtArray) {
+    function runViewer(splatBufferData, isPly, isStandardSplat, alphaRemovalThreshold, cameraUpArray, cameraPositionArray, cameraLookAtArray) {
       const viewerOptions = {
         'cameraUp': cameraUpArray,
         'initialCameraPosition': cameraPositionArray,
@@ -455,7 +463,11 @@
       if (isPly) {
         splatBuffer = convertPLYToSplatBuffer(splatBufferData, 0, alphaRemovalThreshold);
       } else {
-        splatBuffer = new GaussianSplats3D.SplatBuffer(splatBufferData);
+        if (isStandardSplat) {
+          splatBuffer = convertStandardSplatToSplatBuffer(splatBufferData);
+        } else {
+          splatBuffer = new GaussianSplats3D.SplatBuffer(splatBufferData);
+        }
       }
 
       document.getElementById("demo-content").style.display = 'none';
@@ -491,7 +503,7 @@
       <div style="padding-left: 20px; padding-right: 20px;">
         This is a Three.js-based implemetation of a renderer for <a href="https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/">3D Gaussian Splatting for Real-Time Radiance Field Rendering</a>, a technique for generating 3D scenes from 2D images. Their project is CUDA-based and needs to run natively on your machine, but I wanted to build a viewer that was accessible via the web.
         <br><br>
-        The 3D scenes are stored in a format similar to point clouds and can be viewed, navigated, and interacted with in real-time. This renderer will work with both the <span class="file-ext-small">.ply</span> files generated by the INRIA project, or my own custom <span class="file-ext-small">.splat</span> files, which are a trimmed-down and compressed version of those files.
+        The 3D scenes are stored in a format similar to point clouds and can be viewed, navigated, and interacted with in real-time. This renderer will work with the original <span class="file-ext-small">.ply</span> files generated by the INRIA project. It also accepts both my own custom <span class="file-ext-small">.ksplat</span> files or the standard <span class="file-ext-small">.splat</span> files, which are both trimmed-down versions of the original <span class="file-ext-small">.ply</span>.
       </div>
       <br>
     </div>
@@ -522,7 +534,7 @@
         <div class="content-row">
             <div id ="view-panel" class="splat-panel" style="height:300px;">
                 <br>
-                <div class="small-title">View a <span class="file-ext">.ply</span> or <span class="file-ext">.splat</span> file</div>
+                <div class="small-title">View a <span class="file-ext">.ply</span>, <span class="file-ext">.ksplat</span>, or <span class="file-ext-small">.splat</span> file</div>
                 <br>
                 <table style="text-align: left;">
                     <tr>
@@ -589,7 +601,7 @@
 
             <div id ="conversion-panel" class="splat-panel" style="height:300px;">
                 <br>
-                <span class="small-title">Convert your own <span class="file-ext">.ply</span> to <span class="file-ext">.splat</span></span>
+                <span class="small-title">Convert a <span class="file-ext">.ply</span> file to <span class="file-ext">.ksplat</span> format</span>
                 <br>
                 <br>
                 <table style="text-align: left;">

demo/stump.html

@@ -36,7 +36,7 @@
         'initialCameraLookAt': [0.60910, 1.42099, 2.02511]
     });
     let path = 'assets/data/stump/stump';
-    path += isMobile() ? '.splat' : '_high.splat';
+    path += isMobile() ? '.ksplat' : '_high.ksplat';
     viewer.loadFile(path)
     .then(() => {
         viewer.start();

demo/truck.html

@@ -36,7 +36,7 @@
         'initialCameraLookAt': [1, 1, 0]
     });
     let path = 'assets/data/truck/truck';
-    path += isMobile() ? '.splat' : '_high.splat';
+    path += isMobile() ? '.ksplat' : '_high.ksplat';
     viewer.loadFile(path)
     .then(() => {
         viewer.start();

src/SplatLoader.js

@@ -1,3 +1,4 @@
+import * as THREE from 'three';
 import { SplatBuffer } from './SplatBuffer.js';
 import { fetchWithProgress } from './Util.js';
 
@@ -8,11 +9,28 @@ export class SplatLoader {
         this.downLoadLink = null;
     }
 
+    static isFileSplatFormat(fileName) {
+        return SplatLoader.isCustomSplatFormat(fileName) || SplatLoader.isStandardSplatFormat(fileName);
+    }
+
+    static isCustomSplatFormat(fileName) {
+        return fileName.endsWith('.ksplat');
+    }
+
+    static isStandardSplatFormat(fileName) {
+        return fileName.endsWith('.splat');
+    }
+
     loadFromURL(fileName, onProgress) {
         return new Promise((resolve, reject) => {
             fetchWithProgress(fileName, onProgress)
             .then((bufferData) => {
-                const splatBuffer = new SplatBuffer(bufferData);
+                let splatBuffer;
+                if (SplatLoader.isCustomSplatFormat(fileName)) {
+                    splatBuffer = new SplatBuffer(bufferData);
+                } else {
+                    splatBuffer = SplatLoader.parseStandardSplatToSplatBuffer(bufferData);
+                }
                 resolve(splatBuffer);
             })
             .catch((err) => {
@@ -21,6 +39,77 @@ export class SplatLoader {
         });
     }
 
+    static parseStandardSplatToSplatBuffer(inBuffer) {
+        // Standard .splat row layout:
+        // XYZ - Position (Float32)
+        // XYZ - Scale (Float32)
+        // RGBA - colors (uint8)
+        // IJKL - quaternion/rot (uint8)
+
+        const InBufferRowSizeBytes = 32;
+        const splatCount = inBuffer.byteLength / InBufferRowSizeBytes;
+
+        const headerSize = SplatBuffer.HeaderSizeBytes;
+        const headerUint8 = new Uint8Array(new ArrayBuffer(headerSize));
+        const headerUint32 = new Uint32Array(headerUint8.buffer);
+
+        headerUint8[0] = 0; // version major
+        headerUint8[1] = 0; // version minor
+        headerUint8[2] = 0; // header extra K
+        headerUint8[3] = 0; // compression level
+        headerUint32[1] = splatCount;
+        headerUint32[6] = 0; // compression scale rnage
+
+        let bytesPerCenter = SplatBuffer.CompressionLevels[0].BytesPerCenter;
+        let bytesPerScale = SplatBuffer.CompressionLevels[0].BytesPerScale;
+        let bytesPerColor = SplatBuffer.CompressionLevels[0].BytesPerColor;
+        let bytesPerRotation = SplatBuffer.CompressionLevels[0].BytesPerRotation;
+        const centerBuffer = new ArrayBuffer(bytesPerCenter * splatCount);
+        const scaleBuffer = new ArrayBuffer(bytesPerScale * splatCount);
+        const colorBuffer = new ArrayBuffer(bytesPerColor * splatCount);
+        const rotationBuffer = new ArrayBuffer(bytesPerRotation * splatCount);
+
+        for (let i = 0; i < splatCount; i++) {
+            const inCenterSizeBytes = 3 * 4;
+            const inScaleSizeBytes = 3 * 4;
+            const inColorSizeBytes = 4;
+            const inBase = i * InBufferRowSizeBytes;
+            const inCenter = new Float32Array(inBuffer, inBase, 3);
+            const inScale = new Float32Array(inBuffer, inBase + inCenterSizeBytes, 3);
+            const inColor = new Uint8Array(inBuffer, inBase + inCenterSizeBytes + inScaleSizeBytes, 4);
+            const inRotation = new Uint8Array(inBuffer, inBase + inCenterSizeBytes + inScaleSizeBytes + inColorSizeBytes, 4);
+
+            const outCenter = new Float32Array(centerBuffer, i * bytesPerCenter, 3);
+            const outScale = new Float32Array(scaleBuffer, i * bytesPerScale, 3);
+            const outRotation = new Float32Array(rotationBuffer, i * bytesPerRotation, 4);
+
+            const quat = new THREE.Quaternion((inRotation[1] - 128) / 128, (inRotation[2] - 128) / 128,
+                                              (inRotation[3] - 128) / 128, (inRotation[0] - 128) / 128);
+            quat.normalize();
+            outRotation.set([quat.w, quat.x, quat.y, quat.z]);
+            outScale.set(inScale);
+            outCenter.set(inCenter);
+
+            const outColor = new Uint8ClampedArray(colorBuffer, i * bytesPerColor, 4);
+            outColor.set(inColor);
+        }
+
+        const splatDataBufferSize = centerBuffer.byteLength + scaleBuffer.byteLength + colorBuffer.byteLength + rotationBuffer.byteLength;
+        let unifiedBufferSize = headerSize + splatDataBufferSize;
+
+        const unifiedBuffer = new ArrayBuffer(unifiedBufferSize);
+        new Uint8Array(unifiedBuffer, 0, headerSize).set(headerUint8);
+        new Uint8Array(unifiedBuffer, headerSize, centerBuffer.byteLength).set(new Uint8Array(centerBuffer));
+        new Uint8Array(unifiedBuffer, headerSize + centerBuffer.byteLength, scaleBuffer.byteLength).set(new Uint8Array(scaleBuffer));
+        new Uint8Array(unifiedBuffer, headerSize + centerBuffer.byteLength + scaleBuffer.byteLength,
+                       colorBuffer.byteLength).set(new Uint8Array(colorBuffer));
+        new Uint8Array(unifiedBuffer, headerSize + centerBuffer.byteLength + scaleBuffer.byteLength + colorBuffer.byteLength,
+                       rotationBuffer.byteLength).set(new Uint8Array(rotationBuffer));
+
+        const splatBuffer = new SplatBuffer(unifiedBuffer);
+        return splatBuffer;
+    }
+
     setFromBuffer(splatBuffer) {
         this.splatBuffer = splatBuffer;
     }

src/Viewer.js

@@ -424,7 +424,7 @@ export class Viewer {
         };
         return new Promise((resolve, reject) => {
             let fileLoadPromise;
-            if (fileURL.endsWith('.splat')) {
+            if (SplatLoader.isFileSplatFormat(fileURL)) {
                 fileLoadPromise = new SplatLoader().loadFromURL(fileURL, downloadProgress);
             } else if (fileURL.endsWith('.ply')) {
                 fileLoadPromise = new PlyLoader().loadFromURL(fileURL, downloadProgress, 0, plySplatAlphaRemovalThreshold);