Switch from automatic to manual depth testing for billboards

[mzschwartz5]

Description

Previously, when when using terrain, clamping entities to ground, and enabling depthTestingAgainstTerrain, billboards, labels, and points would noisily clip against the terrain at global scales. (See images https://github.com/iTwin/platform-bentley-community/issues/234).

This issue isn't unexpected: billboards and others will face the camera as it moves, causing them to dip into the globe and become occluded. And when clamped to ground, the problem is exacerbated, with noisy terrain sporadically poking through the entities.

Ideally, billboards would not get clipped at this global scale, but would continue to respect terrain-depth-testing at local scales (e.g. a billboard should become invisible when behind a mountain). The solution this PR targets is described in this comment.

NOTE:

Please review commit-by-commit (starting at Moves to manual billboard depth testing in shaders). Specifically, I did some refactoring in Refactors Billboard fragment shader depth testing into functions, which will make it difficult to review all the changes together. Also, I'm going to leave comments on individual commits to help explain changes, and those will only appear in the right places if looking at those commits.

Issue number and link

#12410

Testing plan

Basic billboard clamped to terrain:

Current release vs. localhost

These should behave the same at local scales. At global scales, the PR-version behaves much better - as if terrain is disabled.

Basic billboard clamped to terrain, with disableDepthTestDistance:

Current release vs. localhost

Try this with various depth test distances: 0, really large (non-infinite) numbers, positive infinity, etc. Note: in the current release, there's a bug. The special 3-point depth testing of billboards ONLY applies within the disabled depth test distance region. For instance, if you set the disable distance to 1e20, you would expect pretty much no depth testing - but the 3-point testing occurs all the time now. This bug is fixed in this PR.

Another difference: with this PR, the 3-point depth testing only applies within 5000m, whereas before it would apply within whatever the depth testing distance was (or 5000m if none was set).

And, as before, on global scales, the PR-version behaves much better (as if there's no terrain to occlude).

Basic billboard, on terrain but not clamped:

Current release vs. localhost

These should behave the same at a local scale. Try playing around with different disableDepthTestingDistance settings as well; both examples should continue to behave the same, and no 3-point testing will be done.

Again, on global scales, the PR-version behaves much better.

Billboards among 3D models (regression test)

Current release vs. localhost

Should have identical behavior. However, if you turn on CLAMP_TO_3D_TILE, the current build's billboard clips through the models (because, I believe, with clamping disableDepthTestDistance defaults to 5000m, but with no terrain, it doesn't do the 3-point testing). In the PR-version, this is fixed.

Basic billboard on GP3D tiles

Current release vs. localhost

At global scales, PR-version behaves much better. At local scales, current release has the same bug described before: it clips through the globe because of the default 5km depth test distance and no globe). The PR-version behaves better, but doesn't do the 3-point testing because, again, no globe or terrain.

Billboards with translucent background (no terrain)

Current release vs. localhost

The behavior here differs slightly, but I think it's improved. In the current version, if the blue billboard is on top of the red one, you just see blue. If the red one is on top, they blend to make purple (as expected). In the PR-version, you get purple no matter which is on top.

Current billboard translucency:

CurrentTranslucency.mov

PR billboard translucency:

DevTranslucency.mov

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
• I have updated the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[github-actions]

Thank you for the pull request, @mzschwartz5!

✅ We can confirm we have a CLA on file for you.

[mzschwartz5]

I removed the labelTranslate / pass in 0 now. I honestly don't know the original reason for it - the visual results seem much cleaner to me without it. With it, there's like padding around the billboards that causes it to render way after it's occluded by terrain.

[jjspace]

This definitely seems improved at the global scale. However I noticed some weird "flickering" when zooming more locally. I assume this is just due to sampling or terrain loading or I just got lucky with the billboard "point" between some rocks that only obstruct it at certain angles. Probably still better than it was but thought it was worth noting.

simplescreenrecorder-2025-08-19_11.45.00.mp4

I also wanted to confirm, is this expected to impact 3dTiles or only Terrain? I'm still seeing it clip with the google 3d tiles

[mzschwartz5]

I'm inclined to think the flickering predates this PR, as my changes should only make things more permissive on a global scale. Unless the labelTranslate field I messed with is responsible. I can look into it. Could also just be terrain loading as you said, it's hard to tell.

Interesting that it's still clipping on 3D tiles... I expected this to work on both terrain and 3D tiles, and I thought it did, in my testing.

edit- nope, this fix definitely only works for terrain. Will continue investigation into whether it can be extended to 3D tiles.

[mzschwartz5]

Previously, the distance at which the three-point depth testing technique occurred was set to the disableDepthTestDistance, or 5km by default if not set. Setting it to disableDepthTestDistance was a strange and buggy choice, though, because the original author misinterpreted the setting. They reversed the meaning to only 3-point depth test within that distance.

Since the original intent was to make close up viewing of billboards clamped to terrain better, I think always having it be 5km (while respecting the disableDepthTestDistance value) is okay.

[lukemckinstry]

Can you explain this part (3 point depth test and the bug around misuse of disableDepthTestDistance) again more thoroughly, or point to where it is already written up? I remember talking about this in-person, but I want to make sure I (and others) can understand it clearly.

[mzschwartz5]

Yeah, I'll try my best - it's a little hard to wrap one's head around. First of all, this blog describes the qualitative behavior of billboards clamped to terrain, before this PR. tl;dr - in order to improve billboard visibility, a special 3-point test is performed. The idea is: if any of 3 key points on a billboard is visible, the whole billboard should be visible. If all are occluded, the whole billboard gets occluded.

This special 3-point test was (before this PR) applied only if the camera was within a certain distance of the billboard. The way this distance was determined is what was buggy. The logic was, "if the camera is closer than disableDepthTestDistance, do this 3-point test." (And if disableDepthTestDistance is undefined, default to 5km). But this actually defeats the point ofdisableDepthTestDistance. disableDepthTestDistance is supposed to disable occlusion within a given distance, but this 3-point test was actually fully occluding the billboard within that distance, if all 3 key points were occluded.

With this PR, I just hardcoded the distance to 5km - but, importantly, it should also respect disableDepthTestDistance now. Also importantly, the 3-point test still does what it was intended to do: preserve visibility of terrain-clamped billboards when viewed close up.

[mzschwartz5]

This is what I mentioned above. Removing in favor of always using a 5km distance.

[mzschwartz5]

The crux of this PR's change. Disable automatic depth testing and do it manually in the shaders so we have more control of how it happens.

Note: Given the comment below about translucency, I'm a little wary I may have introduced regressions, as I don't do a LEQUAL check in the shaders, but it seems to work better this way (see test case in PR description)

[mzschwartz5]

Also note- we still leave automatic write to the depth buffer enabled, we just don't test against it automatically.

[mzschwartz5]

These changes, among several others in this PR, are just moving variables outside of macro checks. These used to be only used if #ifdef FRAGMENT_DEPTH_CHECK was defined, but now they're used all the time

[mzschwartz5]

This isn't a new function, really. It just breaks the old getGlobeDepth function into two pieces, so we can call this piece separately.

[mzschwartz5]

More moving of variables outside compiler macros... And now we check if enableDepthCheck is turned on outside of the macro. This used to just mean "is the 3-point depth check enabled," but I repurposed it to mean "is depth testing enabled at all".

If it is, we do all forms of depth testing below (3-point, and regular)

[mzschwartz5]

I would recommend reviewing the VS before the FS.

[mzschwartz5]

Again, the crux of this PR. This the manual depth testing that replaces automatic depth testing. Except, now, with manual control, we can depth test against a flat plane at global scales.

[mzschwartz5]

Was never really necessary - eyeDepth is used in one place exactly. The point was to store off the value of positionEC.z before it gets mutated, but that's not necessary either. Now (down below) we just use this value where it's needed, but before the important mutations occur. (...but after we potentially set eye depth to 0 if the billboard fails the 3-point test).

[mzschwartz5]

I repurposed enableDepthCheck - it used to mean "is the three-point depth check applied?" But now it just means what it sounds like "do we depth test at all?"

To accommodate this change, I had to move some chunks upwards in the vert shader. But it's actually more appropriate in this location anyway; i.e. all the disableDepthTestDistance logic is together now.

[mzschwartz5]

I honestly don't know what the purpose of labelTranslate here was, but it had the (bad) effect of making 3-point testing not work until a label was wayyyy occluded by terrain. Separate from all other changes in this PR, removing labelTranslate and using vec2(0.0) looks way better.

[mzschwartz5]

@ggetz @jjspace, ready for re-review. See PR description: I left detailed test cases, and specifically read the note up top about reviewing commit-by-commit. I left a bunch of comments to guide review on one commit in particular.