Who Is Writing What I’m Reading?
We are Mario Palmero and Norman Schaar, from Tequila Works, though for the time this article is out there, Norman will probably have left the company to face a new adventure.
Mario is a game programmer with interests in art, graphics and game design that tries to get involved in everything and bothers everybody with his curiosity at the end of the day. And Norman is a former artist converted into technical artist. We both share an eagerness for creating new things and pushing the limits of what we can do.
Mario knew about the Norman’s work in the field of Vertex Shader and used some ideas from his work to apply them in Rime months before Norman joined Tequila Works. So Norman was a respected professional for him, though that changed once they met. And Mario was an unknown person for Norman before he arrived at Tequila Works, but unluckily for him that isn’t the case anymore either.
The Dark Ages before Textures in The Vertex Shader
Previously, before DirectX 11, there was no fast and easy way to sample textures in the vertex shader. You could still do a lot of fancy things in the vertex shader, you could use vert attributes to store additional data in your 3d package and use that in the vertex shader. So you could store essentially morph targets in the vertex color to do fancy mesh deformations, pivot points in the UV channels to move and rotate subobjects independently and I’ve used these techniques in the past to generate thousands of GPU particles on mobile two years ago. It is still very powerful! But you are limited by how many attributes you can store in a geometry.
Things however are different when textures are involved...
A Brave New World with Textures in the Vertex Shader
What textures allow you to do is to store an incredible amount of data that is easily retrievable by the GPU. Positions XYZ can be represented as RGB colors. A rotation can be saved as a quaternion, which can be saved as an RGBA texture.
With that in mind, you can create crazy particle animations in Pflow for example and bake those out. You can store particle attributes as colors: scale can be the red channel in a texture, while “temperature” can be the green channel of the texture, etc. You just need to make sure that your shaders samples those colors and knows what to do with them.
In a 4096x4096 texture that uses RGB for particle position and A for scale you could have a 120 frame long animation of 139810 particles. For a smaller particle system with just 2048 particles you could have 8192 frames of animation. At 15 frames per second that would be over 9 minutes of particle animation. We take advantage of the bilinear filtering of the texture to interpolate between our values
Similarly, you can bake out rigid body physics animations to texture. Instead of baking each verts position as a pixel, you can store the elements position and rotation to achieve the same result with a much smaller texture. You can use two 1024x1024 textures (one has the position, the other has the rotation) to store a brick wall made out of 1024 bricks collapsing over the span of 1024 frames. Additionally you have to store the element’s pivot position in the UVs or vertex color. We can still make use out of the vert attributes!
The Sexy Vertex-Count-Agnostic Morph Targets
Another technical challenge that is solved with this approach is bringing baked fluid simulations to a game. Of course the problem here is that topologies are different, vert counts are different, etc. So the mindset here is to think of a mesh sequence as a cloud of triangles that are arranged differently in every frame. Sometimes there will be triangles left unused and those we will simply collapse to [0,0,0]. Here is the gist of the workflow:
Mesh Optimization. Each mesh in the sequence is optimized to something reasonable. Let’s say something around 3000 to 10.000 triangles
Triangle pairing is used to attempts to group triangles together. The more triangles are paired, the better. As two paired triangles only need 4 verts to be defined, whereas two unpaired triangles would need 6 verts. So we are using fewer pixels on the texture.
Textures. Two textures are generated, one that contains the vert’s position and another that contains the vert’s normals.
Mesh. A mesh is generated with the paired and unpaired triangles. The mesh UVs are used to sample in the proper position of the texture.
Shader. The shader only has to retrieve the vert position and the normals from the texture and make sure that the UVs jump from mesh sequence to mesh sequence in the texture, as interpolation doesn’t make sense. We can also use the texture’s nearest filtering method for this.
As a bonus you can use the alpha channel of both textures to store additional data such as temperature, thickness, or even the percentage of two colors to fake the mixing of two different colored liquids.
It Glitters but, Is It Gold?
Textures as data containers has some cool advantages but some drawbacks also. Let’s take a look at the advantages first:
Access. The feeding of the information is done right into the GPU without the intervention of the CPU. Usually the communication between those two systems can become a bottleneck. With textures we can avoid that problem.
Share. Textures have very standardized formats and the engines usually have some tools to configure them easily. It’s a natively portable format that can be edited with several software packages.
Order. The textures provide an intrinsic order based on spatial coherence and we can take advantage of that.
Now let’s check the drawbacks:
Debugging. If anything fails is more difficult to follow the track of the problem. We can assure you that you are going to sharpen your wits to find any bug in the pipeline.
Precision. There is a limitation of precision on the format that has to be dealt with. Try to keep things local as much as you can. You can use newer texture formats that allow for higher bits in the worse case scenario.
Creation. To convert the data into colors we need to create some custom tools. The most popular suites don’t have proper ways to do this conversion.
It Tastes like Fireworks
The time has come to talk about the idea that inspired us to write this article. We were working in setting up the parameters for a VR project here at Tequila Works and, taking into account the previous work of Norman in the Vertex Shader field, we were discussing about baking animation and cloth simulation into textures using the Vertex-Count-Agnostic Morph Targets.
"Could we bake cloth simulation into textures for cinematic scenes? Could we bake amazing facial animation?”
"Yes! Of course! But the amount of data is going to be huge.”
"So, can we compress the data depending on the amount of movement? Maybe we can store just the data when we can’t interpolate.”
"Sounds feasible!”
More or less those were our thoughts. We already had the script to bake the vertices into a texture (due to Norman’s previous work), we were working some numbers (several Gigabytes of data textures) and trying to solve some compression improvements, and also flirting with an idea taken from Hans Godard skinning solver when…
When Mario came out of a pee break with an idea:
Mario: “What if, instead of storing vertex positions, we store the data of the bones in the textures?”
Norman: “...”
Mario: “The vertex shader already does the skinning, we just need the same info stored in different places.”
Norman: “It’s so insane it actually could work.”
And after some hours full of excitement and adrenaline playing with the puzzle we had a good picture of what we could do. And everybody around the office thought that those two guys went crazy talking about putting animation into textures.
The Guts of Our Beast
How We Actually Do It? It’s simple, we just need to feed the information that the skinning algorithm gets through texture and mesh information, so the CPU doesn’t have to pass all the information. Let’s take a look:
