Lecture 04

IIT Kanpur July 201
20 followers
Oct 22, 2023
Now, coming to memory. So, memory is as I said the bit BIT, bit is the basic unit which
is take, it takes 0 and 1, 2 values 0 or 1. This is classical computer. Quantum computer
has more memories, I mean given, I mean 1 unit of quantum computer has not 2 but many
many more, it can say more numbers. Now, 1 byte is 8 bits and typical PC has, my laptop
has 16 GB RAM. So, 8 GB is typical. Now, servers have go up to 4 terabytes of RAM. Of course,
this will be expensive. So, you need to put lot of money for buying this much RAM. Now,
so this is the, right now memory is also told what technology, what kind of memory is that.
So, DDR4 is a standard now, I mean your machine will be DDR4, I believe the new ones. So,
it is called double data rate. Do not worry too much about it right now. But this is important,
how much data can be transferred from memory to CPU. Now, that is what we want, I mean
your arrays or like the vector data is sitting on RAM and I want to process it, I have to
send it to CPU or GPU. So, it can send the maximum, the peak speed is 25.6 GB. Now, remember
can do 20, 2.5 teraflops. Now, tera is 2.5 times 10.12. Now, here I have something like 100, well
more than 100 times less because by the way 1 byte does not contain 1 data. So,
1 real number normally takes 8 bytes. So, if I just divide by 8, then I am stuck with how many
numbers can it transfer. So, it can transfer only 3 giga numbers in the best possible scenario,
3 giga numbers. But that guy says, well I can do 2.5 tera operations. So, I simply am not able
to send that much data and that is the bottleneck right now. That is a real challenge in supercomputing,
memory is slower and what to do with that, that is the problem. So, you also need to know a little
bit about how much memory is required to save a character or integer. So, I have written this
table for C, C language. Python also is very similar, but I mean, but this is for the standard,
but there is one minor difference, but I will tell you right now. So, character typically requires
1 byte, like character A, it requires 1 byte. I mean, A does not require 1 byte, A is a number,
but all the keyword characters, I mean, there is something like 50 of them sitting here.
So, plus some special characters, all that put together is 256 various combinations
and 1 byte is enough, 1 byte is 8 bits. So, 2 to the power 8 is 256, 2 to the power 8 is 256
and I can store 256 characters in 1 byte. It turns out now, we can, we have a lot of
characters for different languages. So, 1 byte is not enough. So, now we will use 4 bytes to store
all possible characters of different languages. Integer, 4 bytes and the range of integer will
be minus 2 power 31 to 2 power 31 minus 1. Now, this you can figure it out. I mean,
with 4 bytes, I can maximize 2 power 31 and 2 power 32, but we have positive numbers and negative
numbers and of course, there is a 0 sitting as well. So, we divide all that numbers into this
range. Now, this is sometimes small number, I mean, 2 power 31 is basically 2 billion.
Now, I may have a number bigger than that, then I need 8 bytes, it is called C long and the range
becomes 2 power 63. Now, sometimes that 2 is not enough. So, in Python, it turns out if the number
is bigger than this, it will allocate more bytes or more thing, but it will not truncate, it will
simply keep putting more and more allocation for memory to store any size number you may need.
Now, Boolean normally will take 1 bit, but you see they store in 1 byte.
So, floating point, a floating point is real number. Float, I am going to tell you a little
bit later what is why it is called float. So, it is very interesting idea, but you wait for maybe,
I mean, half an hour or so, I will come to float. So, float requires 4 bytes
and the range of numbers is here. It can put 10 power 38 max, nothing more than that.
And smallest number is 10 power minus 38. Now, if I need a number bigger than this, then I can
use 8 bytes and that is called double. So, this is called double precision.
And this is called single precision. So, is how many bytes are used to store a number
and double precision I can go to 10 power 308 and also how many significant digits,
15 digits for double precision and 7 digits for single precision. So, my number in, this is in
decimal, not binary, decimal numbers, I can store only 15 significant digits, which is good enough
for most applications. But suppose I want to compute pi up to 1000 digits, then of course,
this is not good enough. So, you need to write the program in more precise, I mean, advanced
program to generate pi up to like 1 billion digits. Now, estimates. So, typical desktop has
4 gigabytes of 8 gigabytes of RAM or laptop. Now, what is the matrix size we can store in this RAM?
Okay. Now, we can compute, we can estimate what is the biggest size, I mean, I will not do it
right now, but I will just show you whether this matrix will fit inside the machine or not.
So, if n is 1000, so what is this number, number of bytes I need? So, how many elements are there
in this matrix? 10 cube square, because it is a matrix is two dimensional. So, 1 million,
10 power 6 numbers. Now, how much one each number require is a real number. So, double precision
will be 8. So, I need 8 times 10 power 6 bytes for double. If I want single precision, then 4
times 10 power 6 bytes for full. So, I need this, it is smaller than this, I mean, this 8 gigabytes
and I have 8 megabytes, this is 8 megabytes. So, this can definitely fit inside this laptop
or desktop. But if I make it, let us say, 10 power 5, 100,000, n is 100,000, then that becomes,
it will not fit here. Because 10 power 5 square will be 10 power 10 and multiple of 8. So,
that will be 80 gigabytes and that is not, that cannot be fit in this machine.
Another, I mean, very big computer is, very big requirement is 4096 cube.
I mean, we simulated one problem with this many grid points and that requires, you can just
do this operation, I mean, multiply and so on. It requires one quarter terabytes. So,
it will not fit in, of course, in a laptop. It also will not fit in a standard
supercomputer. You need a very big supercomputer for this operation. So, we had access to a big
machine and we could do this operation. So, as I said, there is a desktop and of course, there are
slightly faster machine than desktops because servers and even faster is called super high
performance computing clusters. This is like supercomputers. You can think of supercomputers,
but the name is not supercomputers these days, it is called HPC.
Now, these are many machines connected together. I mean, you can basically, you can just imagine
that many desktops are connected together. Of course, you do not put the desktops like
boxes there because that will take too much space. There are pretty tiny nodes,
which are stacked in racks. I do not have a picture here, but you can just do
a Google and you will find how do they look. Now, so basically there are lots of desktops,
desktop nodes. So, this is called node. They are connected together by a switch
and they talk via that switch. Now, I just mentioned two top most HPC systems in the world.
So, the first one is Fugaku. This is a top most machine in the world right now. It is in Japan.
It has how many cores? It has 7 million cores. A combined is 64 cores of a ROM processor.
It is really, I mean, it is astounding how many machines have been connected together
and it can do up to multiplied by 10 power 12. It is a teraflops. If you look at the numbers,
it is 0.44 hexaflops, floating point operation. It can do 0.4 times 10 power 18 operations per
second. So, it is basically, if you say xi is 1, so we are half to that ideal machine,
which you are looking at. So, we have not reached exa scale yet, but we are close to it.
Now, there is difference between Rmax and Rpeak. You do not need to worry about it in this course.
But yeah, so this is the power of the machine and it takes around 30 megawatts of power. So,
it is like huge. 30 megawatts is more than a power requirement of full IIT.
Now, Summit is the second machine is in US and it has 2 million or 2.4 million cores
and it is basically 0.14 hexaflops. Now, anyway, this is just for your information.
We are not going to program HPC system in this course. So, Fugaku has some more details. I
think I can skip this one. As I said, it has basically millions of cores and this is a node
and each. So, CPU of Fugaku is not a ROM processor. It is their own indigenous
CPU and that is 48 plus 4. There are 52 cores in a single node. And memory is also faster. It is
not DDR. It is called HBM. So, high bandwidth memory. Now, so we can read more about these
machines in the internet, but like these are really fast machines.
