#7: Controlling VGA from HPS

How are things going? Any real head scratchers from anyone that they want to bring up? Have people started experimenting with the PIO ports a little bit? Maybe if you haven't had lab section just yet, you haven't started, but that would be the next thing to start experimenting with. I'm going to today go through VGA control from the HPS. There's an example on the course website that shows how to do that. And I'll talk through this in the process of doing it, but this would be the next lab step that I would recommend after this week's checkout, is to start experimenting with the VGA control from the HPS, and then adding PIO ports to that existing example. Because once you have the integrator that you've built and tested in ModelSim, the VGA control primitives from this example I'm going to show today, and the ability to add PIO ports, those are all the pieces that are going to be required for lab one. And then all that remains, it's a non-trivial task, but all that remains then is to put those pieces together, but that all of the scaffolding is then there. Before I start, I thought I would, I wanted to answer for everyone a good question that I got from, I think it was from Rohan in the Monday section, which was, suppose that I wanted to start a Cordis project from scratch for this board. How would I do that? And sort of more generally, is that a good idea? And I thought that was a good question, and it's one that maybe pertains to everyone. So I will just say that it is possible to do that. It is really, really hard. If you open up the Cordis project, which all of you, I believe, have by now, and look at that top level module with all of that IO, and everything that connects to the HPS, if you wanted to build a project from scratch, you'd have to hook all of that up yourself, which again is doable, but really hard and really error-prone. And Altera knows this. So what they did is that they provided a project, which was, I think, originally called, Bruce can correct me if I get this, it was something like university project or something like this. And in any case, they provided a project to users that had essentially everything that is on that FPGA in a Q-SYS layout. And that was the recommended place to start from. So if you want to create a bare-bones project, which is a good idea, the recommended way to do that is start with an existing project and take things away until you've stripped it down to something that's bare-bones, that still has all of the sort of connection scaffolding that you require, as opposed to starting from scratch and building up to that point. I recommend taking it away instead of adding in order to get to a bare-bones project, if that makes sense. And this speaks to sort of a philosophical point about this board, which is, it is, as you've all discovered, really complicated. And the people in this class, it's sort of a self-selecting crew of folks that I know a lot of you love rabbit holes and love going down rabbit holes and understanding things deeply. And that's wonderful. And this is not me discouraging you from doing that. If you want to do that, you absolutely should. This is just a word of caution to everyone that the rabbit holes for this board are deep. And there are a lot of them. And there isn't always a pot of gold at the bottom of them. Oftentimes, you dive down these places, and you gather some insight, and then you come back to the real world with some insight about your system. Sometimes, that's the case here. And sometimes, you go down, and you find just weird sort of arbitrary, Altera-specific stuff, and you don't always come back with something meaningful. So if you want to dive down some rabbit holes, just be careful and be a little bit selective about the ones that you dive down. There are some that could be really worth diving down. And the VGA stuff that I talk about today is maybe an example of this. If you dove deep into the VGA system, you might be able to do cool things like add your own fonts for the VGA screen or the ability to change font color. As the system's set up right now, we don't have the ability to do that. So if you want to go down the VGA rabbit hole and figure all that stuff out, that might be a good one to go down. I might just encourage you to do that as something like a final project, because I don't know how hard that is. I've never tried it. And I suspect that it's hard, sort of scoped for final project levels of difficulty. But in any case, that's just sort of my word of caution to a bunch of rabbit hole lovers to be a little bit careful with this system because your time is valuable and be a little bit judicious in how you spend it here. I may interject a couple of things. One is VGA is a good example. But if you're going to do that, start very, very cautiously. Open up the VGA subsystem, change one parameter, watch what happens, change it back. And do that for a while before you start writing your own font module. I've done that, by the way. And it's tedious. The other thing is there's one hidden piece in this we haven't talked about at all. And that is if you start out with a fresh project, a Quartus project, put all your code in there, compile it, get it to compile, even without, let's just say we don't have anything from Qsys in it even, just raw Verilog, and try and blink an LED, it won't blink. And the reason is that it's deaf, dumb, and blind. There are no connections to the outside world unless you give it a connection list that connects your signal names like clock 50 to pin A34 on the FPGA. That's done for you. It's hidden. If you want to see that, it would be a good idea to. Don't change anything. There's a menu, I think, called Assignments on Quartus 2. You can open that up, look at the assignment editor, and go crazy. Any questions for anyone before we do some PGA stuff? What I'm going to show you today is, as I think I mentioned previously, one of probably three ways of controlling the VGA screen. You control it from the HPS. You can write a bus master to control it in Verilog. That's going to be the topic of a future lecture, because you're going to have to do that for Lab 3. Or you can write directly to VGA memory. And on the course website, there's an example of a student-written module from a few years ago that does this. And you have to modify it a little bit to get it to work. But that allows you to write to the VGA screen without the Avalon bus bottlenecking you. So that's the fastest way to do it. And maybe we'll do a lecture on that as well, if folks are interested. But in any case, let me share my screen, and we can start with this. And interrupt me at any time, as usual. Can I ask sort of a general question about the chips that we work with? Yeah, for sure. So I understand that the system that we use has a system on the chip, which is referring to Linux or the HPS, right? I was wondering, in the other boards that you can buy from Altera that doesn't have system on chip, how exactly does the interaction between the computer and the FPGA work? Would you just attach the FPGA to some sort of network interface card on a computer? Or how does that work? Bruce, do you want to ask about that? I've used two different boards from Altera that do not have HPS systems on it. They don't have the SOC. And communicating with them is extraordinarily difficult. There are no good network drivers that are very fast that you can write in Verilog. Altera ships with a CPU called the Nios II, which is a MIPS-like 32-bit machine that is a soft core. In other words, you write, you do a point and shoot, it writes Verilog, it compiles to a CPU. It runs C. It has a TCP IP package. It runs at 50 megahertz clock rate. And you know what that means? It's a dog. It's 20 times slower than the ARM9. So what it amounts to is that projects that don't use the ARM9 don't do advanced communication to the outside world. It's too hard. I'll add, and maybe this is a trivial point, maybe it's not quite the question that you're asking either, but it is, to be clear, it is possible to program the FPGA completely separate from the ARM. You don't need the ARM to program the FPGA. And in Lab 2, we'll do close to that. Lab 2, the drum synthesis is almost all on the FPGA with the FPGA driving audio output. Very, very little to no interaction with the ARM. So that is possible. You can work with the system totally separate from the Linux system on a chip. Maybe not exactly what you were getting at, but it's a point I haven't made, so I think it's worth making. Yeah, no, I was just curious because the kind of FPGAs that I've seen used in a lot of the industries, I feel like don't have a system on the chip. I think they do oftentimes just have a standalone FPGA system that they sort of build on top of Linux boxes. So I was just curious to see, you know, how, like our system is very different from that, but I think I understand that. If you put an FPGA directly on the bus of, say, a PC, then you can use a parallel bus adapter, like a PCIe chip, to get high-speed, reliable communication to the host. But if it's on a separate board, then that effectively means you have to use serial communication. USB is a beast. Ethernet is not bad as long as you don't have TCP IP. If you're willing to throw frames, Ethernet frames back and forth and decode those, it's pretty easy. Otherwise, you're stuck with serial communication, RS-232. Thank you. So let me show you some VGA stuff. I'm on the course website, and if you go to the university computer graphics and sound page, the project that I'm going to open and the one that I recommend folks start with for Lab 1 is this here. This is under the title or under the heading 640x4A 18 and 16-bit color. So down here between the color picker and the graphics primitive demo. Before I do that, I thought we could just take a look at the video for users manual. And there's a huge amount of stuff in here, which I'm not going to go through in any depth, but I thought I'd show you just a few things that are particularly relevant. The first being the arrangement of the pixels on the VGA, which is probably no surprise to anyone. But the pixels are arranged as shown, where pixel 00 is in the upper left corner, and then the X pixels increase to the right up to 639, and the Y pixels increase down to 479. And you can address these pixels in a couple of different ways. So these pixels can be addressed in what's called consecutive mode or in XY mode. And in a few of the different examples, you'll see both of these modes used. This is a configuration that I believe is set within Q-SYS. In consecutive mode, we have essentially what we're addressing across the screen in the X direction and then going down one and addressing across the screen in the X direction again. And then in XY mode, it's as shown, right, where we're separating out the X and Y bits into totally separate parts of the address, which is a little bit more readable. This will make some more sense when we see it in the example. Anything else I want to mention in here? I'll mention that for the RGB color space, if you're using 8-bit color, then the pixels are allocated as shown, where we have three bits for red, three bits for green, and two bits for blue. The reason being, I learned from watching one of Bruce's old lectures that apparently it is the case that we have less resolution in the blue channel in our eyeballs. So there's less need for a resolution in blue because we can't see that. I wonder if there's some evolutionary reason for that. I'm sure there must be, right? There's a bunch of hypotheses. One is that we evolved as fruit eaters, and the ability to discriminate a ripe fruit from an unripe fruit is therefore important. So red-green is important, but blue, eh, not so much. But there may be other reasons, too. Well, there's also the peak of sunlight is in the green, yellow-green. So along the wavelength, that's red-green and less blue. Sure. So this is the allocation for 8-bit, and then if you're doing 16-bit color, the allocation is shown here. Five for red, six for green, five for blue. This is essentially all I'm going to talk about with this user's manual. As I said, you can go through this if you'd like, or not if you'd like. This is an example of one of those rabbit holes to be cautious about. So with that being said, let me actually open a remote connection here and do this. Not on my PC. Okay. So let me download the example I want to use. Okay. So as I said, start with this one. I'll just download this and open it up. I think it's clearest to, I was trying to decide whether to talk about the C first and then look at the Qsys and the Verilog or the other way around. I think it's best to look at the Qsys first and then look at the C. But I could be wrong. Let's see. I'm going to open this up. When you start messing with the graphics in lab, almost certainly the webcam positions to your VGA screens will not be exactly to your liking. Just let me know and I'll fix that. I've done that for folks in the Monday section as well. So I'm happy to do that. And by the way, if you want to do weekly checkouts before your scheduled lab section, that's totally fine. We had one group from the Friday section come to the Monday lab just to get checked out. And that's wonderful. So we're happy to check you out early if you'd like to do that. I'm going to open up the Qsys here. So I'll open up the computer system. We can see what's on the bus here. And this example that I'm opening up is, as I understand, a Bruce-modified Altera example. And I'll show you the modification. Is that right, Bruce? This is modified to write to SDRAM, as I understand it. And I did exactly what you were specifying earlier. I started with the gigantic system that came from Altera, and I stripped out everything except the VGA controller. So there's no parallel ports. There's no nothing. There's just the VGA controller. Then I added back SDRAM to learn how to display video from SDRAM rather than expensive on-chip RAM. So if we look at what's on this bus, some of this stuff is familiar from the examples we've seen before. The system PLL is still here. The ARM is still here. And then I want to highlight the connections. I want to highlight the stuff that the ARM AXI master is connected to and that the LightWave master is connected to. So you can see that the AXI master is connected to the Avalon Memory Map slave input of the SDRAM. It's also connected to the on-chip SRAM. And it's connected within the VGA subsystem to the character buffer slave input. So let me start by just looking at the SDRAM here. So the SDRAM, because it's hooked up to the AXI master, it is placed at offsets from the AXI master base address, just like the PIO ports were. And in this case, you can see that the SDRAM is placed at an offset of 0, 2, 3, FF, FF, FF, FF. So at zero offset from the AXI master base address, which in a previous lecture, this is where we placed a PIO port. In this example, the SDRAM is placed at this address. So if you wanted to add a PIO port to the AXI bus, you would have to be cautious to place it at an address that does not overlap with, in this case, the SDRAM or with anything else that is on this AXI bus. So you also would not be able to overlap with this on-chip SRAM or with this character buffer range of offsets. If you had an address here that overlapped, then it would show you an error down here and you would fix that. The other thing to be careful about is, if you wanted to put something like a PIO port into this example on the AXI bus, you would have to pay attention to the span as specified in the C file that we looked at last time to make certain that the offset that you've added to include that PIO port, which is sort of above everything that's already on that AXI bus, you'd have to make certain your span includes the stuff that you just added. If it doesn't, you're going to segfault. Or you could add PIO ports to the lightweight bus, which there's less stuff on. But in any case, I want to show you that on the AXI bus, we are writing to SDRAM, which is also being accessed by the VGA subsystem. So the way that this is set up is we write pixel information to SDRAM. The VGA subsystem looks at that agreed upon chunk of memory for pixel information and goes in and accesses it to put it onto the VGA screen. We'll see that in the C code a bit more clearly, but I wanted to show you this first. And then there's some other stuff on this bus. The AV config configures the audio video. The pixel DMA module here is, what is this doing? This is communicating, this is translating pixel addresses. Isn't it first something along these lines? Nope, I think you're muted. I can't highlight the bus on the, there's a master here, which is driving the VGA subsystem. And then there's a slave, which is getting data back from the VGA subsystem. So the VGA subsystem is controlling which part of SDRAM is being read. And I believe that the DMA generator is then taking that address range and barfing it out to the actual IO interface, to the VGA RGB lines. Again, it's really tricky that the combination of the AV config, the pixel DMA, the VGA subsystem together is tricky to modify. I've done it after watching a student mess with it for a month and finally getting it right, but be careful. And the on-chip SRAM is, I don't think, used here. Is that right, Chris? That is correct. That was an artifact of the previous example where I was doing picture in picture. So I was puking video, excuse me, a more technical term. I have NTSC connection to the FPGA to put live video into a sub-window and display on the VGA. Decided that I wanted a cleaner example, but I forgot to take the on-chip SRAM off of it. Yeah, when I was experimenting with this yesterday, I did what we're encouraging you all to do, which is deselect this to remove it, compile and put it on the board to see if anything changes. And I suspected based on the connections that this was not doing anything and everything continued to run just fine. The sort of experimentation that makes sense in this case. Okay, so this is what's on the bus. I'll make the point also that in the last lecture, I mentioned that these conduits, right, we can double click to export a wire here, what looks like a wire into the Verilog for us to manipulate. In the case of the PIO ports, exporting that conduit creates one wire within the Verilog that we mess with. In this case, when we export the VGA conduit, it is, let me move my Zoom stuff here. It is adding a few wires to our Verilog. So down here in the computer system, this here is the exported wires into our Verilog from that conduit in Q-SYS, where these here, VGAB, VGAR, VGR, vertical sync, horizontal sync, these are all actual, these are hooked up to actual wires that go into the VGA connection that goes into the screen, right? So we are tying these to actual no kidding wires that are going through a connector into the VGA screen. So you could imagine writing your own module that, you know, turns on and off the horizontal and vertical sync at some point and actually writes to these wires in your own module. And you could do that. That's sort of that third option for writing to the VGA that I mentioned. And I'll treat that in a later class. But there's an example on the website of something that does that. So if you see this sort of thing and you're tempted by that, you can do it. But I'm just making the point that exporting that conduit sometimes creates more connections than just one. For the PIO4, it creates one connection. For the VGA screen, it's creating all of this, all of these connections into the VGA. Okay, so let me just, we're going to look at the C here, but let me just program this so that it's ready to go when we run this. Okay, now I want to look at the C. So I recommend that you start with the graphics primitives example here, because this is going to have all of the nuts and bolts for drawing that you're going to need for Lab 1. Let's take a look at this. I'm going to copy this into a text editor that does syntax highlighting. Okay, so let's see what's going on here. A lot of this is going to look familiar. So as before, we pound define the non-negotiable base addresses from which we have offsets to talk to specific parts of memory. I want to make the point here, because there was a little bit of confusion in the Monday lab about this. In the PIO example that we talked about, when we placed the PIO ports at a zero offset from the hardware registers base, this base address of C0s had some other name. I think it was called axibase, something like this. In the address map file that Bruce provided, it might have a different name. I'm not sure. But in any case, the name here doesn't entirely matter. In this case, we're calling this base address sdram base, because you'll recall from the Q-SYS that we just looked at, that the sdram is placed at a zero offset from that base address. So we're offsetting, and this is just a name to suggest that. It's all that this is supposed to be. sdram base, sdram end. And then this is that span that I was talking about. If you wanted to add a PIO port to the axibus, sort of at an offset that was above everything that's already on that axibus, you would increase this span to accommodate the new stuff that you put on that bus. And then we have some other base addresses. The character base. So the pixel information and the character information. So we're writing pixels to the VGA screen, and we're also writing characters to the VGA screen. Those live in different places, which means that the characters are overlaid on top of the graphics. The characters are overlaid on top of the pixels. And as I mentioned, too, you get by default one font size and color. So you're going to get white text with clear background overlaid on top of the graphics. This is another example of the same thing that we saw last time with a slightly different name. You'll recall that this ff2 is the base of the lightweight bus, which in this example we're calling the hardware registers base. So these are all the base addresses. Again, very similar to things that we've already seen. A bunch of declarations for graphic primitives. Some declarations for some colors. A macro here that will write a single pixel. So this takes in an x and y coordinate and a color. In this particular example, we're using 16-bit color. So what this is doing is declaring a pointer to a pixel, which it is setting to the pixel pointer base address plus the y pixel times 640, which row are we in, plus x. So this is from the data sheet. I talked about there being two ways that you can address a pixel, consecutive addressing and xy addressing. This is an example of consecutive addressing, where this VGA pixel pointer, we'll look at this in a moment, where this is actually pointing to is the base of that SD RAM. So if we were to write to zero offset from that SD RAM base address, we're writing to pixel 00. And then offsets from that are right to different pixels. So to be clear, the memory address is specifying the pixel that we're talking about. The value stored at that address is specifying the color. And then we do some very familiar looking things, right? Just like last time, we do this memory mapping. We're declaring a pointer to the lightweight virtual base into the pixel pointer, the pixel virtual base and the character virtual base, which, as I mentioned, are two different parts of memory. As before, we open memory as a file here, and we do our m mapping, exactly like in the PIO examples. And I'll point out here that when we are mapping the pixel virtual base, we are pointing it to the SD RAM base here, right? So as I said before, if you were to write to a zero offset from here, which is to say the very bottom of that SD RAM memory, you're writing to pixel 00. And then as you offset from that, you're talking about different pixels. Go across the screen like so. And then the value stored at that memory address is the color, which is a 16-bit data type short, right? Two bytes, 16-bit color that will get mapped to the VGA screen. And then after we do all of this mapping, it's a bunch of very normal looking C stuff. So we have a character array, a bunch of character arrays. And then we exercise some of these graphic primitives, the declarations for which we saw at the top of the page. And then I'll show you where they're defined at the bottom of the page. But making a box with corners at 00, 639, and 479 with color 0x000. What's that going to do? Clears the screen, right? So we're writing a box the size of the entire screen, black. Clear the text, write some text to the screen. And then within this while loop, we are exercising all these primitives, drawing some boxes, drawing some disks, drawing some circles, making rectangles, et cetera, et cetera. And timing the whole thing. And that timing output also gets put out on the screen. I'll show you that when I run this. So I thought I'd show you maybe just one of... All of these are pretty similar, but I'll show you specifically maybe the VGA box function here, which takes in four integers specifying the corners of the box that you'd like to draw. And then a short, which is the color that you'd like to draw. And short being 16 bits. So there's some error checking here. If you attempt to draw... If you attempt to write to a pixel that is outside of 640 by 480, you will segfault, right? So there's some simple error checking here just to make certain that the pixel that we're writing to is an allowed pixel that's not going to segfault the system. And then this loops through each pixel in that box. So within every row, it goes through every column in that row that's in the box. And writes the specified color to the pixel that's at that column in row. And all of these are very similar to this, with slightly different rules about how we loop through. You know, if you draw a disk, it's a different set of rules about how you fill in these pixels. But it's all essentially boiling down to that VGA pixel macro, using that on pixels within some range defined by the geometry that we're drawing to draw these things on the VGA screen. Okay, so let me show you what this looks like. Any questions as I'm getting this set up? Can you clarify again how the pixels are mapped to the SRAM? So are you saying then that the first row of pixels is essentially the first set of values from the base of the SRAM to, say, like, I don't know, 640? The first 640 memory locations in the SRAM are dedicated to the first row of the VGA pixels? Yeah. So if we take a look at this, when you're writing to a particular pixel here, x, y, that pixel is specified in this case, because we're doing the consecutive addressing as opposed to the x, y addressing. It is, you start with the base address, and then you offset from that according to this rule. 640 times your y-coordinate plus your x-coordinate. And it's, like, consecutive in that, like, it'll, it's all essentially, like, kind of connected to each other. There's no gap. Yes, exactly. So if this were 0, 0, we would be writing to just this base address here. If this were the first x-coordinate, it would be 0 plus that first x-coordinate. So we'd be offset just by the x. Do you guys have your IP addresses memorized yet? And then I'll, okay. And I'll compile this. Remember the dash lm? In this case, we're using math, the import math.h. So we need the dash lm. If we were using pthreads, then we need dash lpthread. I'm going to talk about pthreads probably in the next class. Okay, and then let's run this. And then I'm going to open up the camera. There's the FPGA board, and things are working properly. This is the sort of output that you can expect then from, from this example. So in terms of lab one, what I would recommend is get this example running in the way that's shown now. And then add PIO ports to get communication between this C and the Verilog. You'll add the integrator to this Verilog so that you can communicate the output of the integrator over these PIO ports. And then the final piece of this is going to be the actual drawing. And you can sort of decouple this from everything else. A recommended maybe way to do this, the way that I did this, is you could create a locally defined sine table in the C program. And see if you can get the graphics to, in an oscilloscope fashion, draw that sine wave over and over and over again. That's a nice way to get the graphics working right. And then you can, instead of drawing the sine wave through the PIO ports, plug in the integrator output from the Verilog, and have it instead draw the waveform output from your Verilog. But you can, you can sort of decouple these things. You can get the graphics running. You can work on adding PIO ports. You can incorporate your integrator into the Verilog. And you can work on the graphics pretty much decoupled, and then put them all together. It's not easy, but, you know, probably the best path forward. Okay, any other thoughts? I only have five minutes left, so I'm not going to get into key threads. One thing to point out here is that, given that the mapping is done in this program, you don't have to think too hard about how pixels are arranged in SD-RAM, because the VGA pixel drawing macro abstracts that away. You can think in XY coordinates, and, you know, offset from the left-hand side, and offset from the top corner. And then the mapping is done by that macro. By the way, that macro had a shift left in it. It was 640 times Y plus X shifted once to the left. Why do you think that was? Anybody got any idea? Well, if I remember right, I wrote this a couple of years ago, because the addressing is done in bytes. And the pixels are words, are 16-bit words. And so to take a coordinate to an actual byte count, you have to double it. Are folks starting to think a little bit about final projects? I had a few conversations with some folks in lab about these. Some people are thinking about cool ideas involving zoom augmentation by plugging audio from the PC directly into the FPGA. You can talk at the FPGA through a zoom call and do things with that audio. Some groups are thinking about video projects. If you would like for me to do things like plug in a camera to one of these boards and point it at something, I'm very happy to do that. So I could point it at a screen, for example, if you want to feed the FPGA images. Start thinking about it a little bit. Maybe one of you wants to do, oh, I don't know, a pandemic spread simulation. That could be kind of interesting. 3D visualization stuff could be cool. Start pondering. And talk to us about it, too. You know, if you want to brainstorm some ideas, that's one of the most fun things about this class. So by all means, we are happy to brainstorm some ideas. You know, level set on that, there's all of the previous projects for the since the class began are on the website. But only the last three years are using this board. And there was a step up in capability of the board that makes the older projects a little less interesting on this board. But some of them could stand to be dusted off and be done like a 3D renderer. You're in this computer graphics and you want to do a pixel renderer. That'd be fun. Reinvent NVIDIA. Write your own processor, because you can. Take an old processor from the old days of computing and bring it up. Bring up an ENIAC on the FPGA. ENIAC was the first publicly announced computer. Valentine's Day, 1946. Bring up a PDP-11. There's a dedicated group of old guys with beards and suspenders who love PDP-11s. Or if you want to get really weird, there was a machine called Thinking Machines. There was a company called Thinking Machines that built an early parallel computer called a Connection Machine. Bring up a Connection Machine on the FPGA. That's an interesting exercise. Your version would be faster than the original. 3D visualization of the Lorenz butterfly. That could be pretty cool. Maybe explorations of other fractal geometries like the Mandelbrot set that we'll treat in Lab 3. Procedurally generated fractal landscapes could be pretty cool. Yeah, that's fun to do. Or fractal generation of river networks. All kinds of cool stuff with fractals. Anyway, we're out of time, so I'll let you guys go. There's a huge amount of stuff. Plus, we didn't even mention anything really audio related. All kinds of sound synthesis stuff that you can play with. Start thinking. And I will see some of you this afternoon and the rest of you on Friday.