Thread by @SebAaltonen

1 million SDF cubes (950 MB SDF volume). Unoptimized baseline benchmark. Before the sparse data structure.

Running time around 150% in NSight (9.8 ms without).

Some analysis below...

PCI-E 4.0 bandwidth usage is now 6.9%. This was a bottleneck when my index buffer was in system memory. Measured 28 GB/s PCI-E 4.0 bandwidth with that setup. Albeit awesome numbers, putting this buffer to GPU memory made the simple cube test (no sphere trace) 5x faster...

L1$ hit rate hovers at around 90%. L2$ hit is around 60%. The first version didn't have any volume texture mip levels, and showed L1$ hit rate of 30% and L2$ hit rate of 20%.

The performance got 3x faster on RTX 3090 after that change. I can't see any visual difference.

"L1TEX Texture Data Throughput" is the bottleneck. That unit throughput has lots of 50% peaks. But never above it. I guess NSight has trouble showing volume texture bottlenecks (volume texture needs two samples). Even if I artificially add sample instructions it stays in 50%.

The ALU pipelines are obviously almost idling for sphere tracing. This kind of workload uses less than 50% ALU on AMD GCN2 in Claybook, and Ampere has double ALU pipes, so I never expected to see ALU bottleneck here. Even the optimized version should not show much ALU usage.

I am guessing that we see a lot of this ROP stall problem here. We have large bounding volumes and only small amount of rays really do lot of work.

Unfortunately we don't have super fine grained warp timeline visualization on any PC tools to show this.

https://twitter.com/SebAaltonen/status/1011211972904472576

The new sparse algorithm renders small cubes (8x8x8 SDF each), making the pixel shader cost similar to each lane. Thus the above problem should be much smaller. This needs to be combined with a GPU-driven occlusion culling of course to ensure no overdraw.