This post is for answering that question, and it’s the second in a series of three posts about my rebuild of ESP32-fluid-simulation. The previous one was a task-level overview of the whole project, and the next one will be about the fluid simulation’s physics. So, how did I handle the input and output of the fluid sim this time?

The answer starts with the hardware, naturally. As I established, I went with the ESP32-2432S032 development board that I heard about on Brian Lough’s Discord channel, where it was dubbed the “Cheap Yellow Display” (CYD). That choice guided the libraries that I was going to build on, and in turn the libraries defined the coding problems I had to solve.

image of the 'Cheap Yellow Display'
The ESP32-2432S032, a.k.a. the Cheap Yellow Display

Materially, the only component I used was the touchscreen module, and it used an ILI9341 LCD driver and an XPT2046 resistive touch screen controller. In some demonstrative examples, Lough used the TFT_eSPI library to interact with the former chip and the XPT2046_Touchscreen library for the latter chip, and these examples included what pins and configuration to associate with each. None of this setup I messed with.

Review: Adafruit_GFX coordinate system

This fairly niche convention is obvious to some but new to others. It even tripped me up. So, I’ll cover it here.

The Adafruit_GFX library has set conventions that are now widespread across the Arduino ecosystem. Even up to function signatures (the function name, input types, output types, etc), the way to interact with display libraries that adhere to these conventions doesn’t change from library to library—save a couple of lines or so. A real example of this: my transition from an RGB LED matrix to an LCD was trivial, yet there couldn’t be more of a gap between these two technologies. This is because the RGB LED matrix library, Adafruit_Protomatter, and the LCD library, TFT_eSPI both followed the conventions.

One of these conventions is their coordinate system. When I say “coordinate system”, you might think “Cartesian”. However, the Adafruit_GFX library does not use the ordinary Cartesian coordinate system, even though they use “$(x, y)$” coordinates to refer to pixels on a display. The difference just comes down to what moving in the positive-$x$ direction and moving in the positive-$y$ direction mean. In the ordinary Cartesian coordinate system, that’s rightwards and upwards respectively, but in the Adafruit_GFX library that’s rightwards and downwards respectively.

diagram showing Adafruit_GFX coordinates

This should be compared to the way a 2D array is indexed. Given the 2D array float A[N][M], A[i][j] for some i and j refers to a specific element in that array. In a way, we can also think of i and j as coordinates.

diagram showing i-j indexing
Note: i and j are represented in this diagram as "i, j"

In fact, if we equate i to “$y$” (the downward-pointing one used by Adafruit_GFX) and j to $x$, I’d argue that 2D array indexing and the Adafruit_GFX coordinate system are wholly equivalent—separated only by the rename we just described. As long as we remember this rename, I’ve had success using one for the other with no transformations whatsoever. This becomes useful later in this post.

We can cover the touch task first. I already had an idea for what it should do before I started writing it: a user had to be able to drag the stylus around the screen and then see the water stirring as if they stuck their arm into it and whirled it around in reality. With that in mind, what should we want to capture?

First, we obviously should check if the user is touching the screen in the first place! Second, assuming that the user is touching the screen, we obviously should also get where the user is touching the screen. The final piece of data we should capture isn’t obvious. I’ll explain why in the next post, but suffice it to say that we’ll also want the direction and speed in which the user is dragging the stylus—the velocity of the dragging.

To deal with the first two matters, reading the documentation for the XPT2046_Touchscreen library takes us most of the way. To check whether the user is touching the screen is a single call of touched() (or tirqTouched() && touched(), the && meaning that the fast path function tirqTouched() returning false would save us from calling touched()). Assuming this returns true, getting where the user is touching the screen is a call of getPoint()! The only bit of care that we need to take here is this: the XPT2046_Touchscreen library outputs $(x, y)$ coordinates that follow the Adafruit_GFX coordinate system, but it assumes that the screen is 4096x4096. That can be fixed with a bit of linear rescaling. To be exact, we can multiply the $x$-coordinate (a.k.a. the j-coordinate) by $320/4096$ and the $y$-coordinate (a.k.a. the i-coordinate) by $240/4096$. Here’s another next-post matter though, I couldn’t get the fluid simulation to run at full-resolution, so I ran it at sixteenth-resolution. That means that the scaling factors actually were $80/4096$ and $60/4096$ respectively.

There’s nothing built-in for the third matter. I had to tease out the velocity, and an idea that I exploited to get it is that, as the stylus is dragged across the screen, it had to have traveled from where we last observed a touch to where we see a touch now. This is a displacement that we divide by the time elapsed to get an approximation of the velocity. To use vector notation, we can write this as the expression

\[\tilde v = \frac{\Delta x}{\Delta t}\]

where $\Delta t$ is the time elapsed and $\Delta x$ is a vector composed of the change in the $x$-coordinate and the change in the $y$-coordinate. This approximation gets less accurate as $\Delta t$ increases, but I settled for 20 ms without too much thought. I enforced this period with a delay.

diagram showing how we approximate velocity using the previous displacement
Appoximating the current velocity with the previous displacement

That said, the first caveat is that this is, again, in the Adafruit_GFX coordinate system, so a positive $y$-component of the velocity signals downward motion—not upwards motion. The second caveat is that this idea doesn’t define what to do when the user starts to drag the stylus, where there is no previous touch. Strictly speaking, we can save ourselves from going into undefined territory if we code in this logic: if the user was touching the screen before and is still touching the screen now, then we can calculate a velocity, and in all other cases (not now but yes before, not now and not before, and yes now but not before) we cannot.

Finally, if we had detected a touch and calculated a velocity, then the touch task succeeded in generating a valid input, and this can be put in the queue!

That leaves the render task, using the TFT_eSPI library. The fact was that the fluid simulation put out a set of 2D arrays that represent (by using RGB, that is) the fluid colors. Let’s say that I had full-resolution arrays instead of sixteenth-resolution ones, then the render task would’ve been extremely simple because a pair of some i and j to index them would’ve corresponded to a pixel at a specific $(x, y)$ and vice versa. (To be exact, recall the Adafruit_GFX coordinate system, but anyway!) Therefore, an approach that would technically work is to go pixel by pixel, the loop being to retrieve the corresponding color, encode it, and then send it out. In fact, we need to shape up this simple solution so that it can address a couple of problems.

First, let’s reintroduce the fact that we only have sixteenth-resolution arrays. Because this now means that the arrays correspond to a screen smaller than the one we have, we have a classic upscaling problem. There are sophisticated ways to go about it, but I went with the cheapest one: each element in the array is a 4x4 square on the screen. From what I could tell, it was all I could afford. Because it meant that the 4x4 square was of a single color, I could reuse the encoding work sixteen times! Really though, if I had more computing power, I suspect that this would’ve been an excellent situation for those sophisticated methods to tackle.

This choice of upscaling alone might offer a fast enough render of the fluid colors, especially if we batch the 16 pixels that make up the square into a single call of fillRect(). That’s one of the functions that was established by Adafruit_GFX. However, I found that I needed an even faster render, so I turned to some features that were unique to TFT_eSPI: “Sprites” and the “direct memory access” (DMA) mode.

Now, googling for “direct memory access” is bound to yield what it is and exactly how to use it, but to use the DMA mode offered by TFT_eSPI, we only need to know the general idea. That is, a peripheral like the display bus can be configured to read a range of memory without the CPU handling it. For us, this means we would be able to construct the next batch of pixels while the last one is being transferred out. However, to do this effectively, we’ll need to batch together more than just 16 pixels.

That’s where “Sprites” come in. Yes, you might think of pixel-art animation when I say “sprites”, but in this context, it’s a convenient wrapper around some memory. Presenting itself as a tiny virtual screen, called the “canvas”, it offers the same functions that we can expect from a library following Adafruit_GFX. As long as we remember to use coordinates that place the square in this canvas (and not the whole screen!), we can load it up with many squares using the same fillRect() call, but under the hood no transferring takes place yet. Once this sprite is packed with squares, only then do we initiate a transaction with a single call to pushImageDMA(), this function invoking the DMA mode. And from there, we can start packing a new batch of squares at the same time.

fillRect() is called with (x_local, y_local) as where the square starts, pushImageDMA WILL be called later with (x_canvas, y_canvas) as where the sprite starts, and the previous sprite is still transferring

The caveat: if we pack squares into the same memory that we’re transferring out with DMA, then we’d end up overwriting squares before they ever reach the display. Therefore, we’d want two sprites—one for reading and one for writing—and then we’d flip which gets read from and which gets written to. This flip would happen after initiating the transaction but before we start packing new squares. Finally, the terminology for this is “double buffering”, more specifically that’s the “page flipping” approach to it, but the purpose of it here is more than just preventing screen tearing.

Classical page flipping is two buffers and two pointers: no data is copied between the buffers but the pointers get swapped. My code does the same but had to use the array indices 0 or 1

That about covers the touch and render tasks, altogether describing how I used the touchscreen module. But between the hardware and my own code is the TFT_eSPI and XPT2046_Touchscreen libraries, and that set in stone the features and conventions I got to work with. In particular, I had to lay out the exact relationship between the i and j indices with which the fluid simulation worked and the $(x, y)$ coordinates that have become ubiquitous across Arduino libraries, in large part because of the Adafruit_GFX library. With that in mind, using XPT2046_Touchscreen was just a matter of scaling and maintaining a bit of memory. On the other hand, I turned to the DMA mode and “Sprites” that were uniquely offered by the TFT_eSPI library just to keep pace, but these features largely kept within the Adafruit_GFX box, and just a bit of care beyond that (double buffering, that is) was needed.

With this post and the last post done, all of the context has been set: the next post is about implementing the physics. Stay tuned! But if you’re here before that post comes out, there’s always the code itself on Github.