WebGLRenderer: Add support for Node Materials

[gkjohnson]

Description

I wanted to open a discussion to talk about the possibility of adding Node Materials to WebGLRenderer - as far as I understand Node Materials are only supported with WebGPURenderer which poses a lot of issues for community adoption, practical testing, and library support:

WebGPURenderer looks like it's coming along great but until it's more fully featured and mature I cannot use it as a platform for any of my professional work. Likewise many tools I've developed and work on are already built on WebGLRenderer and cannot ported to WebGPURenderer. I assume many developers are in the same boat. This means I haven't been able to invest any time in learning and trying the new node material system despite being very interested in the benefits after years of trying to solve shader and material problems.

Adding support to WebGLRenderer would help afford the well-experienced three.js community the ability to test the system in large scale, complex projects and give practical feedback sooner rather than trying it to WebGPURenderer. This will also help ease the transition projects and developers that currently need to use WebGLRenderer. Currently all material development currently being done for WebGL will be rendered useless in WebGPU which this can help avoid.

And from the library perspective - I need to be able to develop something that will work in both WebGLRenderer and WebGPURenderer. Asking developers to maintain two versions of their materials or libraries isn't great and I think will otherwise just make this renderer transition significantly more difficult than it needs to be.

I appreciate all the work done on the new material systems - it looks like a large improvement over what we have now so it would be great to make this transition as smooth as possible.

cc @sunag @RenaudRohlinger

Solution

Add support for NodeMaterials to WebGLRenderer.

Alternatives

Community feedback is significantly delayed, developers and libraries are fragmented.

Additional context

No response

[DennisSmolek]

Correct me if I'm wrong, but doesn't switching to the WebGPURenderer (which supports nodes) work on WebGL only instances as the fallback?

Or are you wanting a way to mix the existing GLSL code with the outputs of the node system?

Are you just wanting the output of the visual side of things or are you wanting things like the compute and functional nodes as well?

[gkjohnson]

I understand that WebGPURenderer has a webgl fallback but this is about using Node Materials with WebGLRenderer. In this case I don't care about the underlying browser graphics API being used - I care about the compatibility with the WebGLRenderer class that many projects are heavily invested in already and cannot or are not ready to switch for a variety of reasons.

Or are you wanting a way to mix the existing GLSL code with the outputs of the node system?

I just want to use node materials as they are with WebGLRenderer and leverage all the benefits that the node system already promises and provide feedback. I'm not proposing new features otherwise.

[sunag]

Thanks for the analysis @gkjohnson, it's a very interesting subject. In these last months where WebGPURenderer is gaining more users, it became clear that we need a migration guide to TSL, not just the a specification. I believe that just as it was done in previous years to design the new Nodes System and TSL, now it will be to smooth the transition.

I fear that it will not be possible to migrate all the TSL features, except for the basic functions already known in the old node system, but I believe that this is not enough. We had to discontinue the new node system for WebGLRenderer because it was falling far behind in features compared to WebGPURenderer (#28167), the renderer ended up being re-architected not only because of WebGPU but also to align with the new node system, and bring new features like backdrop for example, things that would not be possible using GLSL with simple functions in WebGLRenderer without complex work involved, only in this specific case WebGPURenderer does not render the opaque materials twice when using transmission materials like WebGLRenderer, instead it copies the g-buffer to a texture, for this it is also necessary that the color transform is done in post-production internally, this backdrop effect can be accessed on any material using viewportTexture(). The event system like '.updateBefore()' that allows nodes to create pre-pass, at the time of rendering the object, which can also be shared with other related ones, in WebGPURenderer the light and shadow system can be extended using nodes, and this allows users create new shadows and shadow filters as well as the light, light model and the light system without needing to modify the core for this. Including other things like compute()... surely this is not all just what I remembered now.

Alternatively, I would like to know if there is anything else we can improve like the TSL transpiler, and a guide for this migration.

[Mugen87]

I'd like to understand in more detail why migrating to the new renderer is any issue. To me, the real migration task is to move from GSLS to TSL. We want to support this process with more documentation and guides and also technically with a TSL transpiler. But WebGPURenderer itself is not that different compared to WebGLRenderer, at least API-wise. Most public methods are identical, it is required in some cases to switch to an async/await pattern though. As long as you didn't use an extensively forked version of the renderer, exchanging the renderers should not require a lot of migration effort.

Generally speaking, TSL and WebGPURenderer are superior compared to WebGLRenderer with its hardwired materials and lights. As outlined by @sunag supporting node materials in WebGLRenderer is essentially developing a new WebGLRenderer. I've said it earlier but the project should not make the mistake to invest too many resources in maintaining things which are in fact legacy. From a strategic point of view, it's better to focus on the future by adding missing features to the new renderer, improving performance and fixing bugs. I understand when companies (or individuals) have built solutions around WebGLRenderer but it should be clear that a migration to TSL and WebGPURenderer is inevitable. I also understand there are some reservations about the maturity level of WebGPU but it's important to highlight that a WebGPU usage isn't mandatory when using the new renderer. WebGL 2 can always be forced with a renderer parameter.

TBH, I have expected this discussion sooner or later since we always had it when moving to new approaches. E.g. Geometry to BufferGeometry, WebGL 1 to WebGL 2 or legacy to physically correct lighting. There were always developers who wanted to retain the old structures for different reasons but the project consistently deprecated (and eventually removed) legacy code paths. Which was in retrospective the right decision. It should be clear that at some point in the future WebGLRenderer will be deprecated as well. To me, the renderer is now in some sort of pre-stage. We still provide bug fixes and smaller additions, but implement no big features or larger refactorings anymore. And I think that's the right direction.

[gkjohnson]

Thanks Sunag - I understand the new nodes system includes a lot of "magic" that can't necessarily be easily back ported to WebGLRenderer without significant work and I think that's okay. A more standard subset of features akin to what would have been supported previously or with a more traditional nodes system would be sufficient imo and serve as an on-ramp for users - then more complex nodes and TSL systems can be "better with WebGPURenderer".

I'm mostly looking for a way to make sure (some very complex) materials and shaders being written right now can be used later down the line or a porting process can be done progressively rather than all at once, possibly fragmenting the community in the process (libraries will have to decide to support their huge userbase of WebGLRenderer users, abandon them and only support WebGPURenderer, or support two completely different codebases).

---

Mugen -

E.g. Geometry to BufferGeometry, WebGL 1 to WebGL 2 or legacy to physically correct lighting.

I'm having a hard time seeing how these changes can be compared to upgrading to WebGPURenderer at all. WebGL1 > 2 was transparent for almost everyone since it was enabled by default and physically correct lighting was a flag and multiplying light intensities by some constants. Deprecating Geometry is maybe the only significant one but even when I started using the project almost 10 years ago, using "Geometry" already did not feel recommended. There have been a lot of deprecations and whatnot over the lifespan of three.js and for what it's worth none of them have really been concerning. I think it needs to be recognized how much more significant this change is.

Most public methods are identical, it is required in some cases to switch to an async/await pattern though

This alone can mean significant architectural changes for user projects, guarding against race conditions, etc.

I also understand there are some reservations about the maturity level of WebGPU but it's important to highlight that a WebGPU usage isn't mandatory

To be clear I wasn't expressing concern about the underlying webgpu browser API - I'm expressing concern about the three.js WebGPURenderer implementation. WebGLRenderer has over 10 years of bug fixes, API refinement, memory leak fixes, etc. Of course a lot's been learned in that time and is applied to WebGPURenderer but I've written and used enough code to know that a complete rewrite of anything needs some time to be proven before it should be relied on for any large scale projects.

Dependencies on ecosystem libraries is also something that needs to be considered. Tens of thousands of projects using the "postprocessing" package every week, for example, and it will not work with WebGPURenderer which means any project that depends on it will not be able to upgrade. I know this project has post processing effects, as well, but historically the three.js screen effects have not been comparable in terms of performance or quality - which is fine. It should be expected that dedicated package is higher quality.

That project has been in the process of upgrading its architecture for almost 4 years. How long until WebGPURenderer will be supported? Again no knock on the project; it's great work and we all know this stuff takes time especially when it's free contributions. But the reality is that many people rely on projects like this and it's going to impact the sense of "readiness" and ability to migrate to a newer three.js renderer.

I in-part created this issue because another package, three-geospatial, is creating some amazing, complex effects and because of the above limitations the codebase is stuck in WebGLRenderer and I foresee the maintainer not having the time, availability, or interest in porting it forward. Any project depending on this one now will not be able to migrate forward now, either. This may be a small aside but the notation of TSL also makes writing the scale of shaders such as the ones in that project feel completely unapproachable, which is one of the areas I'm hoping more feedback can be surfaced.

Three.js should be promoting building modern, future-proof shaders which should ultimately help accelerate us to a WebGPURenderer-only world rather than continuing to have to build using the old system simply because dependencies and renderer maturity are preventing it. I feel like the inability to write any cross-renderer shaders is already having a larger impact than people seem to be aware of. I'm curious to hear the thoughts on the above issues.

To be clear I'm very excited for WebGPURenderer and the future of node materials. To be honest I've been excited for node materials for the last several years and truly I'd be hoping that I'd have been able to take more practical advantage of them by now 😭

Also happy new years everyone! 🎊

[vanruesc]

That project has been in the process of upgrading its architecture for almost 4 years.

Has it really been that long already 💀

How long until WebGPURenderer will be supported?

I'm excited for node materials but haven't used them for the same reasons you've described. If WebGPURenderer supports shader-based materials alongside node materials, it should be a drop-in replacement for postprocessing.

The part that would need to be changed in postprocessing is the manipulation of three's built-in materials to render additional scene data during the main geometry pass via MRT. From my point of view, node materials just provide a better way of manipulating those built-in materials and while I've been curious about that aspect in the past (#16440 (comment), #16440 (comment)) I've not had the chance yet to actually try it out.

WebGLRenderer's materials can only be tweaked via onBeforeCompile string manipulation which is fragile. A while ago, I figured out that those manipulations can largely be avoided for the MRT use-case if three would define explicit and predictable shader outputs (#27808). And I think node materials already provide something similar. However, WebGLRenderer is still widely used, so I'll try to get back to finishing that PR because it would make the switch to node materials easier later on.

[CodyJasonBennett]

If it helps, another perspective is that the use of WebGPURenderer requires a complete rewrite for the surrounding ecosystem, and there's no clear path to support both WebGLRenderer and WebGPURenderer, as the ecosystem will inevitably be fractured. The only present solution requires multiple builds and a bespoke shader chunk system using TSL transpilers or forked implementations, which is simply infeasible. Further, as WebGPU and WGSL are not fully specified, we can't productionize on them at all in the browser, which personally limits me to contributing as an enthusiast. Backporting node materials would serve as this path and a platform to write future-proof shaders today that leverage new backends later.

This is incidentally nice for postprocessing since it doesn't have to fork from the uber material system to create a G-buffer necessary for material properties like roughness and normals (maps) for SSR/GI. My SSILVB based on Activision GTAO did for normal map detail, but I had since abandoned it due to a plethora of ANGLE/WebGL bugs. Original plans and linked triage were from when nodes were WebGL only (no TSL) and WebGLRenderer did support it. I maintained this original system and addons in the three-stdlib fork and here, but I since removed it in v2.

I would be happy to champion this work if the project allows, as my work was prepared to drop use of three upstream for our own production renderers (OSS, on top of three) since we felt there was an ivory tower status around future development in three.js and had a bigger stake in projects like Slang over TSL. If we could open up discussion on nodes and agree on some goals, we have much to contribute that would otherwise be potentially duplicate. This includes graphics work, which I have been careful to backport or mirror when possible.

In any case, I plan to upstream fixes or features if able, but my general ambition is greatly limited without clear knowledge as to what is in flight and what use cases three.js looks to with these systems. This is a discussion I've been meaning to have since 2021 as nodes materialized in the WebGPURenderer, but opted to play and contribute to specs directly instead. My focus was instead to WebGL2, which Renaud has continued in open source.

[mrdoob]

@sunag How hard would it be to implement something along the lines of this?

import * as THREE from 'three';
import { GLSLNodeBuilder, NodeMaterial } from 'three/webgpu';
import { color, float, time, sin, mul, add } from 'three/tsl';

const renderer = new THREE.WebGLRenderer();
document.body.appendChild(renderer.domElement);

const builder = new GLSLNodeBuilder(); // <--

const timeFactor = sin(mul(time, float(2.0)));
const dynamicColor = mul(color(0.5, 0.2, 1.0), timeFactor);

const nodeMaterial = new NodeMaterial();
nodeMaterial.colorNode = dynamicColor;

const material = builder.build(nodeMaterial); // <--

const geometry = new THREE.SphereGeometry(1, 32, 32);
const mesh = new THREE.Mesh(geometry, material);
scene.add(mesh);

function animate() {
  requestAnimationFrame(animate);
  renderer.render(scene, camera);
}
animate();

[CodyJasonBennett]

It seems that nodes have a deep dependency on the current renderer/backend (setMRT/nodes/library are entrenched), but what MrDoob suggested I think is a good goal. At least, I don't see a way to completely backport support without backporting the whole rendering API with unconditional use of nodes. Alternatively, #29781, but this could be done in a future-proof way with TSL, which reduces that use-case to portability.

[sunag]

For WebGLRenderer I think the only viable way is to bring WebGLNodeBuilder back. GLSLNodeBuilder will produce very different code that is incompatible with ShaderChunk and is based on UniformGroups for all uniforms but I still wonder if we should invest in a legacy renderer as @Mugen87 said #30185 (comment)... @brunosimon ( https://www.youtube.com/watch?v=C2gDL9Qk_vo ), @RenaudRohlinger ( https://www.utsubo.com/ ) and others have shown that it is already possible to make amazing projects in WebGPURenderer, it is fair to be afraid because it is new and we may encounter bugs along the way, but the more projects are made in WebGPURenderer the sooner we can polish the renderer and make it more stable and performant, and your help would be amazing. I also have doubts about the instability that the node system can have in WebGLRenderer.

With WebGPURenderer we already have MRT support built into materials and post-processing, so it could be great to spend our time trying to make these features compatible with the new renderer. The result of this test is more related to the use of the approach than to the backend: #29597 (comment). I know we don't have complete documentation yet, but we are open to help in any way we can, creating features, answering questions, creating examples and improving the system to speed up this development.

TSL is also not a "closed language", before being a shader syntax it is a Three.js graphics API, so it is also possible to port WGSL and GLSL functions, if the focus is programming for a specific backend, wgslFn and glslFn nodes can be used and we can probably improve this support as well. A lot of the functionality we have exclusive to WebGPURenderer comes from users who were using the renderer as a base for their projects and were looking for WebGPU functionality, and we did that, with most of them falling back to WebGL2. So to me the alternative mentioned #29781 seems to make more sense.

[Mugen87]

Adding support of TSL will also prolong the lifetime of WebGLRenderer and constantly produce new feature requests so a big amount of our time won't be invested in WebGPURenderer anymore. We already support occlusion queries or transform feedback in WebGPURenderer's WebGL backend and I can guarantee the community wants these for WebGLRenderer as well if we now include TSL support. This is just the wrong signal and I do not support going down that path.

It's like when a car manufacture would now invest budget in enhancing a fuel-based combustion engine instead of investing in electric engines, battery tech and autonomous driving. There are better alternatives so let's focus on them.

And to clarify: It is not sufficient to just generate the shader source from TSL. As @sunag mentioned the whole uniforms/bindings topic is implemented differently (more efficiently) in WebGPURenderer so we would need a complete separate code path (WebGLNodeBuilder) for WebGLRenderer.

[CodyJasonBennett]

Is it technically possible to implement the backend interface in WebGLRenderer so it would align with WebGLBackend? I understand there's still a strict dependency on nodes for library/mrt fields, and node generators have a deep dependency on the current backend; otherwise, it can be used in WebGLRenderer all the same. Ideally, generators would not have a dependency on backends at all and reference a capabilities or pipeline layout object. I ask since I'm willing to do this work myself and would pay just for the ability to do it, even if limited to a subset of features (e.g., no transform feedback). The alternative #29781 still puts the ecosystem in stasis like pmndrs/postprocessing, and it's uncomfortable to do this in user-land since TSL is not a real standard, so we're beholden on a patch basis to three.js (the only implementation) all the same. In any case, I'm concerned about shutting down any stepping stone equal to any migration path.