Thread by @tomaka17

Small thread about why command buffers in vulkano (a Rust library that wraps around Vulkan) are so complicated.

Let's take an example situation to illustrate:

You allocate two buffers A and B, then you write the content of buffer A (in your program), then you want to ask the GPU to copy the content of buffer A to buffer B.

Contrary to OpenGL, everything is asynchronous in Vulkan. The way you do this is you send an instruction to the GPU, and your program continues to run while the GPU performs the copy.

Because of this, you need some synchronization. It is forbidden for the user to read/write the content of the two buffers while the GPU is performing the copy, otherwise you have a race condition.

But there's another problem: in Vulkan, you can prepare commands in advance.

What that means is that you can prepare in advance a command that copies from buffer A to buffer B, then execute it later. You are strongly encouraged to do so, because preparing commands might perform some CPU-intensive compilation step.

A very common pattern in video games consists in writing to buffer A, then executing the copy command (prepared ahead of time), then perform other commands that use the buffer B, then wait for the GPU to finish the copy, then repeat.

While this pattern is safe (there's no simultaneous read/write), I don't think it is possible to use the Rust borrow checker in order to enforce that, because of the fact that you can prepare commands ahead of time.

One could imagine a command object that holds references or ownership of the buffers. But again, since you're preparing commands ahead of time in order to execute them multiple times, there's no other option but to hold shared references and perform the safety checks at runtime.

Another difficulty is that you are strongly encouraged in the Vulkan world to allocate a low number of buffers and sub-divide them into multiple sub-buffers that you manipulate concurrently. In other words, an area.

This again complicates the safety checks.

To accommodate for that, the Buffer item in vulkano is not a struct but a trait. Vulkano provides a type named UnsafeBuffer, similar to the UnsafeCell of the standard library, and the user is supposed to implement their own synchronization strategy on top of it.

Of course, vulkano provides some safe implementations of Buffer which you can use. You don't have to manipulate UnsafeBuffer directly unless none of the existing types suit you.

Another difficulty is that you want the GPU to be able to execute multiple commands in a row. For example, you want to ask the GPU to copy from buffer A to B, then from B to C.

What you clearly don't want to do is have the CPU wait for the copy from A to B to finish before asking the GPU to copy from B to C.

That would be a performance killer.

That's why vulkano has a GpuFuture trait. When you submit a command, a GpuFuture is returned.

A GpuFuture is similar to a Future, except that it represents the point in time when the GPU has finished doing something.

A GpuFuture doesn't return any value, but you can use them to enforce an order in the commands that the GPU performs.

For example, copying from A to B returns a GpuFuture, and you can then tell vulkano that the copy from B to C must start afterwards by passing this GpuFuture to the execute function.

GpuFutures are also used to enforce synchronization.

A GpuFuture can "hold" shared or exclusive ownership of buffers (held at runtime; again, we can't use the borrow checker).

It is legal to submit a command that requires an exclusive borrow of a Buffer after a GpuFuture that holds an exclusive borrow to that same buffer.

To summarize: if you submit a command for immediate execution, vulkano will try to lock the buffer by calling methods on the Buffer trait.

If you submit a command for execution after a GpuFuture, vulkano will first try to prolong an existing lock inside of that GpuFuture.

I'll stop here, but maybe you start to get why it's so difficult to write a safe low-level wrapper for Vulkan, and why it took me so much time to come out with vulkano's design.

And I didn't even mention pipeline barriers! That's maybe for another day.

An arena*