[WIP] Add CubeTexture blur #30165

elalish · 2024-12-19T00:16:08Z

The idea here is to make the PMREM cubemap blur function available as a generic public blur method for a CubeTexture. So far I only have a sketch, but this being my first attempt at using TSL, I've run into some early problems that I'd like to ask @sunag about.

github-actions · 2024-12-19T00:19:11Z

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	339.44 79.06	339.44 79.06	+0 B +0 B
WebGPU	488.86 135.68	489.66 135.89	+798 B +212 B
WebGPU Nodes	488.33 135.55	489.1 135.77	+775 B +220 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	465.3 112.11	465.3 112.11	+0 B +0 B
WebGPU	558.8 151.3	559.79 151.53	+996 B +232 B
WebGPU Nodes	514.87 141.07	515.87 141.29	+996 B +224 B

elalish · 2024-12-19T00:18:23Z

src/renderers/common/extras/PMREMGenerator.js

+	const cubeUVsampler=Fn((envMap, sampleDirection)=>{
+		return bilinearCubeUV( envMap, sampleDirection, mipInt, CUBEUV_TEXEL_WIDTH, CUBEUV_TEXEL_HEIGHT, CUBEUV_MAX_MIP );
+	});
+	material.fragmentNode = blur( { ...materialUniforms, latitudinal: latitudinal.equal( 1 ), sampler: cubeUVsampler } );


This is the heart of the refactor I'm attempting - make the sampling function something that can be passed in so I can use different ones for a CubeUV texture vs an actual CubeTexture.

src/nodes/pmrem/PMREMUtils.js

sunag · 2024-12-19T00:21:53Z

I'd like to ask @sunag about.

I can help with whatever you need :)

src/renderers/common/extras/PMREMGenerator.js

elalish · 2024-12-19T18:19:40Z

Thanks @sunag - that got me unstuck for a bit. However, in my last commit I'm trying to pull in some of the #29900 changes but I hit a weird error, this.value is null in the function below:

getTransformedUV( uvNode ) {

		if ( this._matrixUniform === null ) this._matrixUniform = uniform( this.value.matrix );

I would guess this has something to do with my outputDirection node, but I don't think I've changed anything about it? In any case, fodder for a clearer error message. Any thoughts?

src/renderers/common/extras/PMREMGenerator.js

sunag · 2024-12-19T20:15:13Z

that got me unstuck for a bit. However, in my last commit I'm trying to pull in some of the #29900 changes but I hit a weird error, this.value is null in the function below:

I pushed a commit to fix set uniform value, this happened because texture( null ) is uniform, you can change the value by changing the .value property. But I think we can improve this error message.

elalish · 2024-12-19T22:57:19Z

Ah, thank you again! I don't know how you manage to develop all this without TS. I have the actual blur function written now - I just need to test it.

elalish · 2024-12-23T20:30:51Z

Okay, I might have this close to working, and I updated the new API with feedback from @mrdoob. However, upon trying to test it, I realized that CubeRenderTarget is not exported to the public API - it inherits from WebGLCubeRenderTarget which is, but that doesn't have any of the new TSL stuff. Where should this function be exposed?

elalish · 2024-12-23T22:50:11Z

src/Three.WebGPU.js

@@ -6,6 +6,7 @@ export { default as Lighting } from './renderers/common/Lighting.js';
 export { default as BundleGroup } from './renderers/common/BundleGroup.js';
 export { default as QuadMesh } from './renderers/common/QuadMesh.js';
 export { default as PMREMGenerator } from './renderers/common/extras/PMREMGenerator.js';
+export { default as CubeRenderTarget } from './renderers/common/CubeRenderTarget.js';


Not sure why this wasn't exported before for WebGPU - hopefully this is okay?

elalish · 2024-12-23T23:03:21Z

src/renderers/common/CubeRenderTarget.js

+			return envMap.sample( sampleDirection );
+
+		} );
+		blurMaterial.fragmentNode = blur( { n, latitudinal: latitudinal.equal( 1 ), poleAxis: vec3( poleAxis ), outputDirection: positionWorldDirection, weights, samples, dTheta, sampler: cubeSampler } );


@sunag thanks for getting the PMREM version of this blur working - now this one is giving me trouble. I'm getting a lot of exceptions like getOutputNode( builder ) where builder is undefined. Also this line device.queue.writeBuffer( bufferGPU, 0, buffer, 0 ); is giving me TypeError: Overload resolution failed. which sounds pretty bad. I'm going to try to make a simpler repro in a three.js example instead of MV.

Okay, I think this is just async issues with WebGPURenderer in MV.

elalish

@mrdoob @sunag I got a bit further by hacking this function into an existing example, but I'm stuck again. I've gone ahead an put all my API feedback since this is my first venture into using WebGPURenderer and TSL.

elalish · 2024-12-27T22:23:48Z

examples/webgpu_cubemap_adjustments.html

@@ -72,6 +72,12 @@
 				cube2Texture.generateMipmaps = true;
 				cube2Texture.minFilter = THREE.LinearMipmapLinearFilter;

+				renderer = new THREE.WebGPURenderer( { antialias: true } );
+				await renderer.init();


This was a quick fix to deal with an "uninitialized renderer" warning, which should maybe be an error since it just skips my shader entirely and makes a black texture? It also says I should use renderAsync, but i'm using camera.update, which doesn't seem to have an async version? Not sure why this hasn't been a problem for fromEquirectangularTexture...

elalish · 2024-12-27T22:26:27Z

src/renderers/common/CubeRenderTarget.js


 import { WebGLCubeRenderTarget } from '../../renderers/WebGLCubeRenderTarget.js';
 import { Scene } from '../../scenes/Scene.js';
 import { CubeCamera } from '../../cameras/CubeCamera.js';
 import { BoxGeometry } from '../../geometries/BoxGeometry.js';
 import { Mesh } from '../../objects/Mesh.js';
 import { BackSide, NoBlending, LinearFilter, LinearMipmapLinearFilter } from '../../constants.js';
+import { cubeTexture as TSL_CubeTexture } from '../../nodes/accessors/CubeTextureNode.js';


This is weird - when I tried to have a local variable named cubeTexture it didn't work - said it was a function instead (this one). Looks like it's getting properly renamed here, but apparently not? Is it somehow globally included somewhere?

elalish · 2024-12-27T22:27:35Z

src/renderers/common/CubeRenderTarget.js

+			return envMap.sample( sampleDirection );
+
+		} );
+		blurMaterial.fragmentNode = blur( { n, latitudinal: latitudinal.equal( 1 ), poleAxis: vec3( poleAxis ), outputDirection: positionWorldDirection, weights, samples, dTheta, sampler: cubeSampler } );


Okay, I think this is just async issues with WebGPURenderer in MV.

elalish · 2024-12-27T22:31:08Z

src/renderers/common/CubeRenderTarget.js

+
+		const camera = new CubeCamera( 1, 10, this );
+
+		const width = cubeTex.source.data[0].width;


I've run into this issue before in Three (and complained about it) - this feels super fragile, but how am I supposed to find the size of a cubeTexture? Some have image, some have source, and I'm pretty sure if you generate one with a render target it has none? Seems like we're missing an API for this. Can we please just have a .size?

elalish · 2024-12-27T22:33:25Z

src/renderers/common/CubeRenderTarget.js

+			samples.value = blurParams2.samples;
+			dTheta.value = blurParams2.radiansPerPixel;
+
+			camera.update( renderer, scene );


Now that this function is finally running I'm getting an exception that comes from here (oddly not the first camera.update call?): TypeError: Failed to execute 'writeBuffer' on 'GPUQueue': Overload resolution failed. No idea what to do with that.

elalish · 2024-12-27T22:37:50Z

examples/webgpu_cubemap_adjustments.html

+				await renderer.init();
+
+				const cube2Blur = new THREE.CubeRenderTarget(cube2Texture.source.data[0].width);
+				cube2Blur.fromCubeTexture(renderer, cube2Texture, 0.1);


I'm throwing in a fixed-size blur for now to test - to have it hooked up to the backgroundBlur UI I believe we'll need some kind of cubeBlurNode that works a bit like pmremNode, automatically regenerating the texture as needed.

elalish · 2024-12-30T22:26:15Z

Since this is my last day at Google (and as a maintainer of MV, see google/model-viewer#4975) I don't think I'll get a chance to finish this PR. Hopefully it looks useful enough for someone else to pick up and push over the finish line. Thanks for all the help, @sunag! The TSL you've built is quite impressive, though hopefully by the time I play with it again it'll have a few more docs 😄.

sunag · 2024-12-30T23:45:20Z

@elalish This news took me by surprise. You have done an incredible job over the years with MV and with all the improvements in physical-based materials, tone mapping, etc. I have always found your high level work and rich in detail, your work will certainly be missed by the community. I hope we can meet again on life's journey and I wish you all the best in your achievements.

elalish · 2025-01-05T06:42:45Z

Thanks a lot @sunag! If you ever find yourself in New Zealand, come visit.

elalish added 2 commits December 18, 2024 16:10

factor out sampler

deca60e

Merge branch 'dev' of github.com:mrdoob/three.js into cubeBlur

de3b18e

elalish commented Dec 19, 2024

View reviewed changes

fixing parameters

e87294d

elalish mentioned this pull request Dec 19, 2024

WebGPURenderer: PMREMGenerator - calculate blur weights and parameters once #29900

Open

elalish added 2 commits December 19, 2024 09:18

further simplification

1773a8f

use getParams function

05da83f

github-advanced-security bot found potential problems Dec 19, 2024

View reviewed changes

src/renderers/common/extras/PMREMGenerator.js Fixed Show fixed Hide fixed

elalish added 2 commits December 19, 2024 11:53

completed fromBlur

4c777e8

Merge branch 'dev' of github.com:mrdoob/three.js into cubeBlur

ad5cb2d

github-advanced-security bot found potential problems Dec 19, 2024

View reviewed changes

src/renderers/common/extras/PMREMGenerator.js Fixed Show fixed Hide fixed

switched to userData

fe882ea

elalish force-pushed the cubeBlur branch from ee1eb30 to fe882ea Compare December 19, 2024 22:55

rename to fromCubeTexture

1981a27

elalish added 2 commits December 23, 2024 14:43

export CubeRenderTarget

5afa0c9

mrdoob approves

4346838

elalish commented Dec 23, 2024

View reviewed changes

elalish mentioned this pull request Dec 23, 2024

[WIP] switch to WebGPU google/model-viewer#4939

Open

test in example

a5d3e89

elalish commented Dec 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add CubeTexture blur #30165

[WIP] Add CubeTexture blur #30165

elalish commented Dec 19, 2024

github-actions bot commented Dec 19, 2024 •

edited

Loading

elalish Dec 19, 2024

sunag commented Dec 19, 2024

elalish commented Dec 19, 2024

sunag commented Dec 19, 2024 •

edited

Loading

elalish commented Dec 19, 2024

elalish commented Dec 23, 2024

elalish Dec 23, 2024

elalish Dec 23, 2024

elalish Dec 27, 2024

elalish left a comment •

edited

Loading

elalish Dec 27, 2024

elalish Dec 27, 2024

elalish Dec 27, 2024

elalish Dec 27, 2024

elalish Dec 27, 2024

elalish Dec 27, 2024

elalish commented Dec 30, 2024

sunag commented Dec 30, 2024

elalish commented Jan 5, 2025


		const camera = new CubeCamera( 1, 10, this );

		const width = cubeTex.source.data[0].width;

[WIP] Add CubeTexture blur #30165

Are you sure you want to change the base?

[WIP] Add CubeTexture blur #30165

Conversation

elalish commented Dec 19, 2024

github-actions bot commented Dec 19, 2024 • edited Loading

📦 Bundle size

🌳 Bundle size after tree-shaking

Choose a reason for hiding this comment

sunag commented Dec 19, 2024

elalish commented Dec 19, 2024

sunag commented Dec 19, 2024 • edited Loading

elalish commented Dec 19, 2024

elalish commented Dec 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elalish left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elalish commented Dec 30, 2024

sunag commented Dec 30, 2024

elalish commented Jan 5, 2025

github-actions bot commented Dec 19, 2024 •

edited

Loading

sunag commented Dec 19, 2024 •

edited

Loading

elalish left a comment •

edited

Loading