2 Video Views·Jun 11, 2024
Electron microscopy is almost a century old, but it continues to play a role in exciting new developments that extend its use well beyond its original purpose. We sit down with Professor Sergei Kalinin from the University of Tennessee-Knoxville to discuss these exciting new applications of this older technology. Learn how electron microscopy, originally developed for imaging, is now used for atom-by-atom materials fabrication. We also explore how combining this technology with machine learning can streamline scientists' work.
This Materialism Podcast is sponsored by Materials Today (https://www.materialstoday.com/) , an Elsevier community dedicated to the creation and sharing of materials science knowledge and experience through their peer-reviewed journals, academic conferences, educational webinars, and more
This episode of the Materialism Podcast is sponsored by Cal Nano (https://www.calnanocorp.com/) , leading experts in spark plasma sintering and cryomilling technologies. You can learn more about their work and services by visiting their website (https://www.calnanocorp.com/) .
Thanks to Kolobyte and Alphabot for letting us use their music in the show!
If you have questions or feedback please send us emails at [email protected] or connect with us on social media: Instagram (https://www.instagram.com/materialism.podcast/?hl=en) , Twitter (https://twitter.com/MaterialismPod) .
Materialism Team: Taylor Sparks (co-host, co-creator), Andrew Falkowski (co-host, co-creator), Jared Duffy (production, marketing, and editing).
Keywords: Electron Microscopy
I would like to describe a field in
which little has been done but in which
an enormous amount can be done this
field is not quite the same as the
others in that it will tell us little of
fundamental physics but it will tell us
much about the strange phenomena that
occurred just below our perception in
contrast to the Natural philosophers of
the past the scientists of this field
delve into the recesses of Nature and
show how she works in her hiding places
their Quest is to understand and create
the imperceptible after all there is
plenty of room at the
bottom hello and welcome to the
materialism podcast an exploration of
the past present and future of Material
Science and Engineering my name is
Andrew falowski I'm a student at the
University of Utah and I'm joined by Dr
Taylor Sparks Taylor how's it going
Summer's here man best time of the year
it doesn't get Summer was here until we
got a winter storm in the middle of May
no joke May 8th or something I had the
best ski day of the entire season it was
completely wild man no I mean we got
like just the taste of Summer and I was
I was I was into it the day before I was
at the park on a hill on a Sunny Hill
just reading the book and then the next
day there's snow on the ground and know
and you're back in the house waiting for
sunny days I know I got bees and my bees
I I do beekeeping and they were so happy
the first couple days and now they've
just boarded it up they're waiting it
out we'll see if they come out today
okay today we have an awesome episode
we're going to talk about uh a topic
that I thought I kind of knew well and
it's clear that I didn't and to to help
us understand it um we've got one of my
favorite people one of the smartest guys
I know Sergey kenin joining us Serge
tell us a bit about
yourself so great to be here so
currently I a professor at the
University of Tennessee Knoxville so
before then I spent 20 year at orri
National Laboratory working at the user
Center so I started as the VNA fellow uh
I left when I was a group leader and the
corporate fellow slightly unusual thing
about myself is that in between orage
and the Academia decided to see how the
real world works so I spent a year at
Amazon and the special project divisions
you know it's kind of equivalent of the
Amazon equivalent of the Google a where
the idea is to see how the scientific
ideas grow at the risk and become a part
of the Amazon product line so that's
that's the short story fantastic um a TP
a sabatical at Amazon that's not typical
was it a good experience
it was super good so it's something that
I can unambiguously recommend for pretty
much everyone in the Academia so as a
scientist we are exceptionally well
familiar with the ideas we generally
want to have these ideas materialized
but in the end we think that when
something is done we have a publication
out of it for the last maybe five years
we know the Publications are not enough
we actually need to share code we need
to share data in other words science is
useless if those that come after us
cannot build upon that and paper is of
course a very very narrow Foundation to
build on but still it is totally
different from the industrial setting
when the ultimate goal is to have a
product in five years and ideally it
would be a commercially successful
product so the way that it affects the
way you think the Vues of the uh
research and the engineering inquiry you
will explore is just
amazing absolutely it is totally
different um reward structure totally
different timelines totally different
everything from Academia so it's cool
that you're able to sort of Traverse
both plus National Labs uh giving you a
pretty well-rounded perspective on the
different options out there for
science pretty much uh in terms of the
National Labs they can also be fairly
heterogeneous so the different Labs have
different uh priorities and figures of
Merit so I spent uh 20 years in the BS
part of the World which is basically
aligned with the Academia as close as
possible so we go to the same
conferences uh our primary measure of
performance is the Publications and
high-profile Journal so it's not
terribly different from uh Academia in
terms of where we are going with the
research in the BS Labs cool cool well
we've got you on today sge because you
of probably anybody that I know I
consider you like the expert in
microscopy especially microscopy plus ml
right which is going to be kind of the
topic of what we cover today and you
sent me a you know this what I at the
time thought was this super bizarre
message on Twitter saying like hey let's
do an episode on what' you say electron
microscope beam assembly of matter atom
by atom and I was like what is he been
smoking like what is he been talking
about because you know I'm I'm like a
generalist material scientist I'm not a
microscopist but like I also kind of
know about the technique I've done a
little bit of temm I've done a little
bit of sem but what you described there
assembling matter adom by adom that was
not something that I thought my croscopy
could do so in today's episode I want
you to get me up to speed and any of our
listeners that might fall in that same
category of thinking like yeah I'm kind
of aware of microscopy but say what now
help us understand what exactly um has
happened over the last 30 years in this
area of assembly via
microscopy sounds like a plan so look at
it this way so we always consider
microscopy as primarily the technique
that allow us to characterize material
right yep and uh there is always a
little bit of push and shove about how
really useful the microscop is so we
know that intellectually if we use the
microscopy to look at the single crystal
that's probably not going to be the most
rewarding experience right so we are
just going to see atoms sitting at their
own sides and that's basically it
microscopy becomes useful only if our
material is non ideal so we are sort of
expand to see the real structure defects
how close they aligned the development
in the high resolution microscope
Illustrated that we can not only resolve
atoms but to see how they displaced from
the high symmetry positions so all of a
sudden we can visualize the order
parameter Fields like ferroelectric
octahedra Tils all the other things that
make the physics of the material is
interesting but for a very long time
basically from the beginning of the
electron microsc at er ruska time till
maybe 2010 or so the Assumption was that
we want to use microscopes only to
visualize the materials right my early
interest in the machine learning stem
from the fact that microscopes generate
large volumes of data uh manually
analyzing images may be not a long-term
uh Pathway to success so we need to have
ml tools to help us do that but in all
cases the feeling was that uh the
feeling was that we just want to see
what's out there and ideally we don't
want to touch anything so the less our
Imaging condition affect the material
the better I was just going to say like
when I was in undergrad taking my very
first images we looked at some polymer
materials and I remember seeing for the
first time in real time like oh the beam
like if you let it just chill on the
sample you're actually like a blading or
whatever you're doing to it you're
destroying your sample you can see like
the little tattoo that you just gave it
and so the the idea I always thought is
like oh you want to avoid that you don't
want to interact with your sample you
just want to visualize it obviously it's
impossible to not interact in some way
but you want to minimize that and what
I'm hearing from you is that not only is
that not necessarily the case but you
can be intentional about the way that
you interact with the sample to get
thoughtful sort of outcomes exactly and
this is exactly where the Dual
background in scanning probe microscopy
and electron microscopy came in handy I
think because in scanning probe
microscop from the first days of the STM
it was known that the STM tip can move
atems around
and the way that people observed it in
early 90s was that imagine that you look
at the surface of the gold or
semiconductor sometimes you can see the
clear Atomic lce so all atoms are in
place sometimes you see the add atoms
that form kind of nice OD patterns and
sometimes the atom will look like a fuzz
ball yeah obviously atom cannot be a
fuzz ball if it is a fuzz ball that
means that it's actually sitting in some
form of potential well and when it
interacts with the stmt
it start to move around and you
basically see the not the atom itself
but the dynamic picture of how atom
jumps around on the surface within some
small volume and there were quite a few
papers by folks like um at UCLA for
example that basically said that oh we
can have molecular rotors or we can have
Dynamics and so on and so
forth and uh long time ago Folks at IBM
so in Don igler group I speculate but
there are some was that if atoms can
move under the action of the tip can we
really be deliberate about moving it so
that was a STM story and we know that
for about 30 years moving atoms by the
scanning tunneling microscopy tip was
the only way to control matter on the
atomic
level then about year 2010 so that was
the time when the aberation correction
become a a big thing and at ID there was
a big group under Steve pikok where
people started to
uh observe the 2D materials under the in
the electron microscope basically with
fairly high throut output so it was not
like you have one successful experiment
every month and then you write paper in
nature it was you have a successful
experiment pretty much every day and
they started to notice that uh sometimes
the atoms become a fast bolt so you look
at the electron microscop image some
atoms are well defined and some of them
look like they are kind of moving around
or you can even see half of the atom
right we all know that you cannot kind
of exactly cut the atom by the Electron
Beam if you see half of the atom it
means that the Electron Beam started to
interact with the atom and then the atom
shifted to a different position and it
become clear that at some point you can
even see the atom and action so you can
see half of the atom at one Atomic site
and then another half of the atom at
another Atomic side and uh what's going
on it's very clear that the Electron
Beam somehow excites the atom so the
atom jumps to the adjacent location but
from this moment it is a very small uh
step for saying that you know what if by
regularly scanning beam we can
accidentally move Atom from one place to
another maybe we can move the beam
deliberately to move the atom from one
location to another where we want it to
be so that was the thought uh probably
around 10 years ago so slightly more
than that the difficult Parts however
become that in order to do that there
are small quote unquote small technical
steps you have to take so electron
microscopes absolutely not build uh for
the user to tell the Electron Beam where
to go so in fact they have service
contracts secondly you kind of need to
know uh where the beam should be for the
atom to move where you want it to be so
you need to understand why this
phenomena happen and you need to have
the engineering control of the
instrument in order to do it the way
that you want and then the final thing
is that um you see electron microscopes
are much faster than human so human can
I mean if you play uh things like Smash
Bros or Doom your reaction rate is
faster than average but generally humans
can perform several operations per
second and uh as my experience with the
Doom suggest that actually goes down
with the time so I was never able to uh
play the last versions because it's just
too
complex I mean once you has to have this
Dam Dash jump or something else I cannot
do that I had to ask my kids to come and
help me get to the more complex Parts
but electron microscope is still fast it
can uh change things on the time scale
of micr of milliseconds which basically
means that if you want to do it at the
reasonable rate you somehow need to use
the machine learning in order to control
your microscope and uh in a meaningful
manner so that was the challenge and
that's got to be tricky too right
because if the you can only respond as
fast as you can collect information
right so even with machine learning
right are you still limited by the rate
at which you can collect an image to
know where the atom is and are things
like vibrations and other sort of noise
that comes through in the Imaging
process did those affect it to some
extent yes but the important thing is
that electron microscopes again are very
very fast right so the beam current in
the electron microscope is typically a
fraction of n
which basically means that you have a
billion of uh electrons going through
the sample per second you don't need
that many electrons to form some signal
indicative of the presence of atoms so
uh the classical electron microscope can
give you data in milliseconds there are
versions of the ultra fast detector that
can do it in microsc so the dynamic
range between how fast you can connect
information versus the range at which
humans can operate is easily multiple
orders of
magnitude and so you know Andrew
mentions that okay there's the
collection but then there's also usually
some sort of post-processing I'm
guessing with microscopy and so if
there's a step that has to happen if you
have to process it to say to get the
exact position of bom you're going to do
a representation of it now that could be
also slower so it doesn't matter how
fast your ml is because it's waiting on
that post-processing step from the raw
image anyways is that the case like
where are we at or maybe it's the same
with compressed sensing right you don't
you're not going to measure every single
Pixel you're going to measure some of
them and fill in the gaps with the
others um are these slowing you down in
your ml pipeline as well so for the time
being that has not been the um has not
been the uh bottleneck so the for the
time being the bottlenecks are actually
quote unquote uh boring engineering
things like the positioning of the beam
so we know that uh the details of the
Electron Beam interaction with the
nucleus and the processes that Electron
Beam induce in the material very
sensitively depend on where the beam is
with respect to the atom so for example
if your beam is going directly through
Atomic column you in some sense all your
momentum is perpendicular to the atomic
plane if you talk about the 2D material
so that at we cannot expect the atom to
move to uh any specific Direction so
after a lot of trial and Errors By uh my
colleagues and Nod we figure out that
the atom actually moves towards the
Electron Beam if the Electron Beam is
halfway between the atom and and its
neighbor so the exact mechanism by
itself is fairly complex and we
speculate that the way it works is that
we almost excite a single chemical bond
from the bonding to anti-bonding State
then we have the isomerization of this
Bond structure and this the result it
relaxes to the new Atomic
configuration uh that being said from
the point of view of application the
exact theoretical mechanism maybe is not
that important what you need to know can
you move it exactly can you move it and
uh how precisely and uh with which resp
reproducibility can you move it that's
ultimately what matters from the point
of view of the uh fabrication electron
microscop and ml yeah so in one of the
papers you sent um you had sort of a
standard setup that you calibrate
against I can't remember was like 90
Pico amps or something like that and in
the paper you mentioned that it's over
the 5 Seconds or whatever that it's on
the or maybe it's five millisecs um
you're dropping like something like
three billion electrons which
immediately made me wonder like okay
this is a really cool thing that you're
showing me but can this scale um are we
going to see some sort of dramatic
change in architecture of microscopes
that would allow us to do I I never
thought the EB lithography be a thing
and it is right are we going to see
something similar where we be able to
move individual atoms to actually make
devices or is this only going to be in
the proof of concept phase that's a
super good question and that depends on
uh uh kind of how uh how useful it is
you right that the key question here is
what are the downstream application
because strictly speaking people knew
that Electron Beam can modify solids on
essentially Atomic or Atomic plane level
for at least 40 years so 40 years ago
you can find papers what that time the
electron microscopy was uh much less
developed than it is now but there are
papers where people say look we can have
a morphos silicon or a morphos gum
arsenide the Electron Beam can make it
crystalline and you can observe this
process of crystallization essentially
one Atomic plane at a time so it's
almost like a 3D printing but 3D
printing when the electron be Ser the
role of the exactly high high energy
excitation and at that time it was
curiosity and if you go through the
literature you can easily see that
people have rediscovered this phenomenon
many many times so and if you talk to
the folks who have been in the electron
microscopy Community for a while uh they
basically say hey we tried it it was
never useful
so once we go to the atomic fabrication
the ultimate goal ultimate question and
that's something that I learned when I
was at Amazon who is the customer so
what part of the quantum science or
Material Science or condensed
metaphysics will benefit from this
capability and uh in microscopy
Community there are excellent examples
of the technique such a crym where there
was a very sharp inflection point before
it was something interesting but not
particularly useful so until maybe 2014
PR electron microscopy was typically
referred as blab at the conferences
which is not a very endearing term and
the reason was that yes you can observe
the biological molecules frozen in ice
but you don't learn much and then
because of the progress in the electron
microscopy in general the resolution of
cry electron microscopy pass the
Threshold at which you can start to VIs
visualize the molecules with the
resolution sufficient to draw some
conclusions about their biological
function and lo and behold the cry
electron microscopy becomes a key
technique to explore the structure of
the
biomolecules drug Target protein
interactions and so on and so forth and
now you have startups that invest in cry
Asen buys multiple cry electron microsc
stations so in other words now there is
a downstream application and this
Downstream application is the
biology and
medicine so same thing is going to
happen for the atomic fabrication if
this is a proof of concept
experiments uh it will remain proof of
concept experiment for quite a while are
there reasons to be
optimistic I mean I'm obviously is
supposed to be optimistic about it right
man sell me sell it to me I would I
would otherwise I wouldn't be doing it
for 10 years uh but uh there are few
examples I can think about so the most
obvious one and probably the lowest
hanging fruit is the fabricated
nanopores for the protein sequencings
the Precision medicine so this is the
field where there are already companies
like Northern nanapur that essentially
make the device one device has only one
nanopore so it's not like you have a
fabrication threshold and you need to
make a billion for that to be useful the
key question is it the case that the
atomic precise Nur will give you better
results for the uh for the protein
sequencing that the one that's formed by
the dialectric breakdown if it is the
case then there already is a market if
it is not the case uh then we don't know
but the key part here is to figure out
whether this is the case or not so we
don't know yet the
answer another example where it may be
useful is uh uh things related to the
quantum communication so whether the
quantum Computing will emerge and
whether the quantum computers would be
based on the solid state cubes or
something else is pretty much in the a
Quantum communication is very much the
very much the thing now so being able to
fabricate the single defect complexes
that can work at the single Photon
emitter that might have a market so I
don't know whether this Market will be
realized in three years and or 10 years
I don't know how it will compare to the
existing techniques out there but at
least there is
connection I I could see it so it sounds
like the proof of concept is there and
there might be an application but we
need to get better at the actual
movement of atoms which is what the
paper is about that you sent to us so I
I like the early on in the paper you say
like you know the people that have been
doing this for a while by hand turning
knobs right actually moving the beam
around have sort of realized that if you
want to first off we're talking about
graphing you've got a sheet of graphine
and we're going to put a single silicon
in there right and if you want to move
that silicon from one spot to the next
what theyve done the past is they just
hit the beam on the on the carbon next
to it basically and that caused it to
jump over there and it sounded like what
you guys wanted to explore is are there
other rules right instead of just doing
this sort of simple humoristic can we
come up with a model are there better
ways to move things are there unexpected
things and so it sounds like you tried a
bunch of different scenarios you planted
the beam on many different spots not
just directly on the atom but at
different sort of X and Y positions away
from it and then Quantified what
happened which was not easy because you
know things shift and move right your
sample it sounds like you've got the
specimen drift problem and not all the
you know the dopent and the car the
carbon don't have the same contrast so
this sounds like a tricky problem to
pull off um do you want to tell us a bit
about
it so in this case most of the credit
goes to my uh colleague Kevin Raa at
ORNL so uh I started this uh line of
inquiry uh more than 10 years ago by now
so Kevin is last in the uh line of
several group several uh teams that ORNL
that explored this direction and before
maybe half a year before I left ORNL for
Amazon I buil the uh collaboration
between our team and the folks at the
Google brain who were interested in
building the reinforcement learning
framework to make this process
automated so Kevin basically essentially
spend
every uh hours on time when he was in
the microscope with the Google folks and
know the first to make sure that they
speak the same language right so it's
not the easiest connection between folks
working in Google and electron
microscopist no I bet not secondly
figure out how we can in principle
control this process so what is the ml
mechan algorithm for doing that how to
deal with the fact that amount of the
data coming in the experiment is usually
much smaller than the ml folks I used
for it's like when you play the Mario
game or chess in silica you can do it as
many times as you want and then you can
always get extra computer power if
necessary so whatever is done on the
instruments uh takes a lot of time and
effort and the experiment does not scale
it is what it is uh it is a linear time
of on the instrument so the difficult uh
that being
said uh it is doable the difficult Parts
appear in the fact that the real world
and the idealized environment are not
the same thing so for example the
biggest problem for Kevin was to make
sure that the Electron Beam goes exactly
where the control algorithm tells it to
as opposed to be a little bit off
because of the non-ideality of the
scanning system right and this small
thing ended up being a separate half a
year project of how to actually pinpoint
the beam and sort of Target the atom
exactly the atom that you want because
if you're off then even a small
discrepancy between the intended
position and the real Position will
throw off your algorithm and we are
talking about the positions being off by
10% of the atomic radius this is just
wild that you can actually do this that
microscopes are that good as well as
that they let you do that too right a
lot of a lot of instruments and it's
becoming more available and it it seems
that that was essential in making this
research work as well usually lock a lot
of that stuff down they they don't want
you to actually go and turning the dials
manually they want it to go through a
controlled interface so well probably so
that you don't break it um absolutely
true so we started to work on the
automated microscopy noage uh somewhere
around 200 8 2009 so first it was
scanning probe microscopy and the reason
for that was ultimately scanning probe
microscopy at that time tended to have
more open architectures and secondly the
price of error was much smaller how what
you mean
uh in SPM if you something goes wildly
wrong this is a 2300 K tool and
generally it is very unlikely that the
damage that you can incur will exceed
few whatever 10K to fix electron
microscopes are complete relatively more
expensive so usually they come at the
cost of 5 mil a pop they typically have
a service contract so if you start to
work with the electron microscope you
need to be prepared to lose the service
contract protection then we talk about
hundreds case hundreds of CAS per year
so we were lucky because uh at the early
work in this area in something like 2014
2015 uh we worked with folks like and
lupini who was one of the original Three
inventors of the aberation character so
he knew the electron microscope
architecture inside out so at that time
the progress was slow because you have
to have a guy in the room who knows the
electron microscope Hardware the person
who deals with the custom controllers
and then the person who actually runs
the experiment so it was slow things
started to change uh about four years
ago when the neon the company that build
the microscope started to provide the
API for the python interface so it has
become possible to take the python codes
and deploy them directly on the
instrument and I remember very well the
earlier Co days when one of for
mentioned Kevin was actually working on
the microscope from home so he goes to
the lab lows the sample make sure that
the sample is in the microscope then he
goes uh to his uh house and basically
spends the whole night working with the
microscope trying to figure out how to
deploy the Deep kernel learning how to
make atom moves and so on and so forth
and uh what's interesting is that um at
that time the progress still took us a
while so it was maybe two years when we
could say that even with the API we are
comfortable running the ml algorithm
directly on the live microscope and of
course each time you try to new to do
new things you need to develop the
hardware and control component which is
not a part of the ml workflows directly
but what's very remarkable is that the
rate at which we can um drive this
development by now accelerated by order
of magnitude so I started to work at UT
uh barely a year ago so by now we have
the U thermop Fisher microscope which is
integrated with the University of
tenesis supercomputer and that did not
take us three years so at this point it
took us three months and in the same
amount of time we were able to build
interface between the scanning probe
microscope and super computer so again
you can run the experiment you can
stream data to the superc computer you
can stream the control signals back
again so first time it took forever
second time
it was already a finite amount of time
and it was done well and the third time
it looks like that this type of
capabilities are becoming available at
the uh microscop center next to you it's
not it's still difficult but it's not uh
not outstandingly difficult one question
thinking about this is with any sort of
manufacturing process or any sort of uh
scientific approach there's usually a
question of reliability right like how
you even if you find the optimal place
to put your
how reliable is it that you can actually
move the
atom so that's an excellent question and
let's look at it a little bit from
different perspective so for the time
being we discussed that the primary
thing to figure out other iend use
applications so these are not cheap
experiments there are fairly risky
experiments uh you can write Hy profile
paper about doing something like this
kind of maybe only once there is nothing
wrong about it but ultimately this field
worth investing time and effort if and
only if we have the end
use however let's assume that there is
an end use case so there are people
coming to us and say guys look we this
particular Atomic configuration can make
a Next Generation quantum computer so
what we need is to
scale how how far can you go if you have
a sufficient uh how far can the
technology allow you to go if you uh can
make a sufficient
investment well in order to answer this
question let's look at uh what
determines the rate at which the
structures can be fabricated in the
electrone microscope so the ultimate
answer is the number of the high energy
electrons going through the system
because uh that's actually what
ultimately causes the
reaction this goes to the very
interesting point that the beam induced
phenomen in solid very much depends on
the energy of the electron so if the
electron has a large energy then it is
prone to produce the knockoff effects
and move things around if the electron
has a lower energy then it is very
unlikely to destroy the material so
there is still damage but it usually
emerges due to slightly more complex
mechanisms so the question becomes if
somebody says look just let a trip what
would be the ideal system for uh
performing this type of manipulation and
what are the pathways in which you can
develop the technology
so my first answer is that currently we
are limited to the system where we have
to maintain the balance between the
observation so getting the image or
figuring out where the beam should go
and
manipulation therefore we try to
manipulate in the regime where our
Imaging is relatively slow and the high
energy events which move at them around
are fairly rare So in some sense we are
limited to manipulation on the level of
the seconds of frac ction of the second
because if we increase the beam current
or the beam energy we will not be able
to monitor material we will be just
modifying it are we stuck there
absolutely not so there is absolutely no
reason why we cannot have two electron
beams going in the same point in the
solid one beam having the fairly low
energy and we use it for non-invasive
Imaging another beam coming with a high
energy and then we use it for
manipulation that's a cool idea so
that's entirely doable it's not going to
be cheap so if you try to build the
electron microscope that's for example
interfaced with the ion beam facility so
you can observe the internal beam and
use Dynamic this tools this reminds me
of like what Teresa Pollock's doing with
her tribeam um it's not the same but
it's a similar idea that you're you've
got right you've got your ablation
you've got your ebsd you've got your
Imaging all of these in the same
instrument exactly so there is a record
of building this tools first time we bu
one is not going to be cheap you
probably wouldn't start it with less
than 20 mil budget and very good
collaboration with the electron
microscopy manufacturers right but the
technology for doing there is there and
it's available there are other tricks
one can play so for example once we have
the Electron Beam uh there is no
reason why the Electron Beam cannot be
multiplexed simply because there are a
lot of electrons it's not a problem to
go from billions to 10 tens and hundreds
of billions of electrons per second
multiplexed electron microscop exists so
Z and other companies actually build
them for the characterization of Wafers
when you want to scan a large volume
really I didn't that there
obviously uh I discovered it few years
ago and that was kind of pretty
surprising but there is a lot of very
cool electron microscopy development
done in the context of the semiconductor
characterization yeah I'm not surprised
and uh it's actually if you think about
it
uh when it comes to the character
semiconductors these guys can afford
pretty hefty price TXS right and and the
defects that they're looking for are
well suited for microscopy right like
they're looking for the types of things
that you can find there this is the
right tool for it now just need to scale
and so Multiplex sem I'm learning new
things every day this is wild so then if
you try to uh just think about whether
these two concept can be combined
imagine that you have 64 beams sort of
large multiplication you have pushed
your dynamic range for another three
orders of magnitude because you use high
energy electrons for manipulation and
low energy mic electrons for for
observation so you can argue that
getting to the level of uh thousands of
assembly events per second is actually
doable with the technology that we have
beyond that point is difficult to
predict because you know we typically
can predict technologically next maybe
next two steps we're already talking
about combining two steps right but this
should be doable whether there would be
some fundamental mechanism that like
heat accumulation or what's not that
will prevent uh doing things at this
rate we don't know
so we can speculate in term of in terms
of the possibilities but implementation
has to be done step by step with the
specific Milestone and Benchmark at
every at every moment there's no other
way to do that
so what we haven't yet talked about um
that might be an interesting thing to
shift to then is if this is possible if
we have microscopes that holy cow we can
python control them and we can we can
shift on the Fly what we're looking for
um how does that change the way that
scientists interact with microscopes and
other instruments for characterization
right I feel like there's a paradigm
shift coming we've talked about machine
learning for property prediction for
structure prediction that stuff's become
very well known in the last couple years
we've we've seen generative models like
our group's been working in that um but
there's a couple things that have been
under uh investigated one of them is
processing uh tying the the sort of
nitty-gritty boring aspects of okay when
you process materials trying to use ml
for that but then another one might be
with characterization we've seen a
little bit like there's automated re
felt refinement a few things like that
but I'm curious what's your take on what
the future will look like in something
like microscopy what will be the role um
will be the way that scientists interact
with it will it be you know scientists
running it will it be turning it over to
the ml something hybrid between the
two you know it's a super interesting
question and um two years ago I would
say that um if we use machine learning
and automated microscopia we can do the
same things as humans only faster and
there is a figure to Merit to that right
so you can argue that currently we using
microscopes maybe 8 10 hours a day and
then the microscop rest if we have the
ml algorithm working and doing the human
level task then we can use it
24/7 and that increases productivity and
uh they need rest time less time to make
specific decisions so we can get easily
maybe factor of 10 of the output maybe
even factor of 20 but probably no more
than that but some of the developments
that uh happened in my group in the last
year and with my colleagues at oan
Pacific Northwest make me think that uh
I was unduly pessimistic about it let me
explain uh why it is the case both from
the point of view of
opportunity and from the point of view
of how human works and in terms of what
it takes to make it happen so in terms
of opportunity the situation is actually
very simple that uh our microscopes Now
operate in the regime of the double
technological Deb so we use the same
scanning as people used in again in rusa
time and this scanning gives us the
rectangular images which are very
convenient for the human eye to
interpret so that's why it's double
technological maybe it's kind of one of
this component is technological and
other is
psychological that being said
microscopes can first of all operate
much faster so the electron microscope
can easily generate 32k by 32k image
human eye can recognize features maybe
in one or two key images so if we have
image of the size we will have to spend
considerable amount of time of trying to
go through it by ey and find what's
interesting
secondly typically when we run the
microscope or we do the microscop
experiment we actually have a reasonably
good idea about what is that that we
after so what we want to understand and
very often the things that or object
that we're interested in don't have the
uniform Distribution on the material
surfaces so ideally the microscope will
not just scan everything and then human
looks through the data and try to make
sense out of that but the microscope
will actually Target the objects or
behaviors of
Interest so this is the opportunity so
we already know that for electron
microscopes there is at least three or
four orders of magnitude of acceleration
that is possible if we learn to analyze
the data dynamically for SPM methods
which are relatively slow we don't have
three orders of magnitudes but we can
run experiments much uh in a much more
smart way so to increase the ratio of
useful observations to the observations
in
general so the second part uh which is a
very big bottleneck at this point that
it's uh that the humans are
exceptionally good in doing what the
humans are doing right so if you look at
the performance of the large language
models or the or the classical Benchmark
for machine learning all of them start
to basically saturate slightly above the
human level
and saturation slightly above the human
level average human level means well
below the human expert so for the time
being the classical supervised machine
learning models uh if you deal with the
area that you're not familiar with that
can be help but you don't expect it to
know things better in your own domain so
that's a fundamental limitation of the
supervised learning methods they cannot
uh magically jump out of the
distribution or
extrapolate but then the interesting
thing becomes whether rather than trying
to use the supervised machine learning
we can Define the goals of the
experiment in the way that makes sense
and then we'll let machine learning
algorithm to follow those goals as oppos
to rely on the data that we have created
in the previous experiments so this uh
mindset is very different from the
classical or wellestablished mindset in
the ml Community but it seems to be much
better tuned
to the way that the experimental
Sciences work so in other words rather
than trying to use ml to assist me in
doing things that I'm already good at
and then ml only helps me save time I
want to impart my reward structure on
the ml algorithm and let it pursue it in
the way that it sees fit given the
access to the instrumental control that
it can access in the instrument so
what's missing there and you've call out
in your paper like The Missing Link like
the real thing that we're where this
community needs to spend our time is
then translating between what the human
knows what they're looking for in the
experiment and the actual reward that
you program into the algorithm right it
has to have something that it has for
feedback loop and we know like if I'm
looking at a material and I'm trying to
find like a cause of a defect like what
caused this material to break I kind of
have an idea of what I'm looking for I'm
looking for something that's out of the
ordinary I'm looking for certain things
that maybe I've seen in the past that
caused that so how do you translate that
high level thinking about the
application right the end application of
what you're doing to something direct
that the microscope can look for in real
time like I don't know how we're going
to solve that any ideas so uh so this is
super difficult and we started to have
this idea maybe uh a year ago so that
was the time when uh I talked to Folks
at uh Google um uh and uh at some point
I ask them a very simple question like
look guys I tried to understand
reinforcement learning for several years
so I went through several books uh
that's not the most uh trivial area no
but uh let's say I kind of know enough
to I went through several Elementary
examples by now I understand the B
Basics uh I can roughly say uh and write
by memory how the Bellman equations look
like so that's all great but where do
you guys get the reward functions
because uh I understand where the reward
functions comes in games I mean that's
what makes gam super cool because the
reward is almost instant and you know
what is the long-term reward but then uh
when I start thinking about okay
reinforcement learning is great how do I
go how am I going to apply it for the
real world scenarios like microscopy or
material synthesis or what's not and
then I start to realize that for any
real problem that I have uh I can reduce
it to the level of the myopic
optimization when the reward function is
instant but it's very difficult for me
to formulate the real world problem as
the reinforcement learning problem where
the reward is somewhere in the end of
the
process and then I talked to the Google
folks and that was slightly before the
time when this all concerns about the X
risk from a started to surface my
question was uh where do the rece come
from because I kind of feel like this is
the part of the reinforcement learning
that I don't understand and then they
told so am I stupid no you're absolutely
not stupid that in fact the most complex
part of the reinforcement learning so if
we don't Define find the reward
correctly then the algorithm starts to
do something weird so my analogy is that
imagine that I'm playing the chess with
the Boston Dynamic robot and the robot
is told that it has to win so if you
don't specify things better the robot
can easily stand take the uh take the
chessboard and hit me on the head
because win is a win so another example
uh I guess if you read the uh Persian
fairy tales of the thousand one NES and
so on there's always this issue of how
you interact with the genius right so if
the Genia you liberate a Genia and the
Genia is supposed to fulfill through
through three of your wishes you have to
be very very careful about how you
formulate them because if your
formulation allows for G takes a
shortcut that's good for Genie but not
good for you the genie will do that you
always want to be happy one of the ways
to get there is to lose your mind
because then you're always happier and
in some sense the problem with the
reinforcement learning and generally
reward driven methods is exactly the
same as with the genius so if you don't
formulate what you want to accomplish
exactly the algorithm can find the
shortcut that is not something that
you're interested in and then I start to
look at it a little bit broader like
what are generally our rewards when we
deal with the machine learning algorithm
then I realized that intrinsic rewards
are very very bare cupboard we have lost
functions we have different ways to look
at the for example symbolic regression
and value the predictive error and
parimon of the of the expression but
these are very very narrow and very very
specific the real world scenario we
actually deal with the much longer term
objectives which can be fairly complex
So This research on interface of how you
define the reward functions that
translate our long-term objectives in
the short-term rewards that can be
pursued by the automated experiment that
I think would be super
important yeah and thinking to my own
experience with microscopy a lot of the
rewards aren't static either right you
go into uh some sort of microscopy put a
sample in the the sem start looking at
it and you think you're looking for one
thing and then all of a sudden you find
something unexpected something that you
weren't looking for and that starts to
become a new challenge because now all
of a sudden the reward structure has to
be reformulated right yeah you see
you're like well shoot why don't we do
EDS map oh look at that line scan shows
this oh maybe this is what's going on
right like human is very much there
deciding that stuff so how do you get a
algorithm to have this high-dimensional
way of of considering all the different
options and all the different reward
path is when it's not completely
straightforward and notice that this is
actually the cool thing about the
machine learning when you apply it to
experiment because when you apply
machine learning and Theory effectively
at some point you get to the point we
just we need to scale once you start to
apply machine learning in experiment you
inevitably start to think why are we
doing science in the first place what's
our goal so what determines our decision
making because you can take several
people give them the same sample and
different and same instruments and the
way they will structure the experiment
workflow would be totally different so
in some sense the automated
experiment is almost like Humanity in
the uh sort of in the toy model of the
humanity when we need to understand why
we're making decisions and what drives
us and then if you start to do a little
bit of the introspection or in my case
it was
introspection uh forced by the necessity
to build the ml workflows for the
instrument you realize that even during
this single experiment as you mentioned
uh we can pass through multiple rewards
we start with optimizing the instrument
we we then press to explore the material
based on the um for example uh
observation of if we have some areas in
the material that have roughly similar
structures it's natural for us to look
at the majority regions then we start to
do things based on the Curiosity for
example you see weird patterns out there
you try to zoom in on them or try to
understand what goes on there and
generally of only after that we start to
think about some specific the the or the
high level thinking and of course while
we are doing all of that we also keep
eyes open for the serendipitous
Discovery so very often things are uh we
observe things that we didn't expect and
then we are driven by Curiosity on how
to do that so the interesting part is
that all these uh considerations except
for the hypoth even including hypothesis
driven Discovery in some sense can be
introduced as the part of the uh in the
part of the automated experiment
so these are well-defined rewards they
can be casted in the language that the
machine learning algorithm understands
and in principle we can uh run the
automated experiment when the machine
learning agent runs the microscope and
gets the data and issues the commands to
the microscope and the human operator
observe the process and sort of shift
the reward Target for the machine
learning agent and that seems to be the
first case where we can do something for
fundamentally different from how we have
been doing microscopy before so rather
than us thinking about what to do and
issue the commands we specify what is
that we care about in terms of physics
and then the machine learning agent
tells the microscope what to do to
pursue those um targets most effectively
this is already not a speculation that's
actually practically works but the more
interesting part becomes when we go from
the microscopy problem
to the outter context problem so
ultimately we are not doing microscopy
for the sake of microscopy even though a
lot of my colleagues probably will
disagree ultimately we are doing
microscopy starts in the chemistry lab
so we synthesize something we bring
sample for the microscopy we expect to
learn something about the physics or
chemistry of material and the reason why
we need it is because ultimately this
material supposed to be a part of
something else and connecting this
outside context problem
with the experimental planning that
would be a huge thing for the microscopy
and characterization in
general so thinking about reinforcement
learning and in my own work with uh
optimization in general I know that
reinforcement learning takes quite a lot
of data to training to get right and the
more I've played around with
optimization I've come to find the value
of actually simulating optimization
campaigns to see how things work out and
use that to help maybe guide initial
model selection is there role for that
in terms of generating fake microscopy
data to maybe test out different reward
structures or to maybe ensure that the
model is best aligned in advance of
putting it towards real data absolutely
so I think that uh the key constraint in
any experimental Sciences is that our
experimental budgets are limited so
there is nothing we can do about it and
uh effectively we cannot
scale uh even if we try to connect all
the microscopes in the world and some
common ecosystem that allows share ing
the data and
algorithms that will help but it will
not help that much because all
scientists like to do different things
so by definition we all want to go where
no one has gone before which means that
working on totally different data sets
uh is not the case where the big data
actually can help us accomplish anything
so we need to be very cognizant of the
small experimental budgets and
practically uh my feeling is that our
starting point is dealing with the uh is
uh would be the introduction of very uh
uh very myopic workflows when the reward
is available in each step so in this
case we don't have to deal with the
large data requirements of the
reinforcement learning So eventually if
we go for the longer experimental
campaigns that may become an issue but I
think that the primary trick would be uh
not to run more experiments but to be
much more responsible and the cognizant
of how we Define the reward functions
and uh how these reward functions
connect what we can do in microscopy to
the general materials Discovery so
that's where the trick is Sergey you're
amazing um this has been a fun
conversation uh very eye openening for
me somebody who thought I kind of knew
this field um I clearly did not so thank
you for sharing with us man thank you
for inviting and uh that
was very pleasant experience to and the
opportunity to share some of the crazy
things that we've been working for the
last few years okay yeah especially
being an Academia you only get exposed
to these instruments on a very
superficial level right the sem takes
images you know there's certain other
characterization techniques but you
never get that chance to really dive
into and see how how deep all of these
different techniques are and all the
things you can do with them so it was
really eye opening and insightful to
read through these papers and chat
today this episode is sponsored by
California Nano Technologies or Cal Nano
if you've listened to the podcast you
know that we are big fans of them we've
done a couple episodes with them and I
as a professor have used their services
multiple times over the course of my
career they have some really great
Services they've helped us with spark
plasma centering at an extremely large
scale way bigger than what you're
thinking they can make some huge
components I've always had great success
with them they turn out high density
Quality Parts two order and pretty quick
turnaround um and they also have some
other capabilities including cryo
Milling check out our episode on that we
think that they're an awesome company
and we're proud to stand by them as a
sponsor of the show The materialism
podcast is also sponsored by materials
today visit material today.com to stay
up to date on the latest happenings in
the material science field and read some
fantastic articles they have published
you can head over to el.com to find out
more about their journals books
conferences and related
programs thank you as always for
listening to this episode of the
materialism podcast if you have
questions or feedback you can send us
emails at materialism. podcast gmail.com
Yeah we actually love reading those I
love reading messages from fans and also
if you would do us a huge favor
subscribe to our show you can find us on
iTunes on Spotify we used to be on
Google Play but it is now on YouTube so
if you look up Taylor Sparks on YouTube
you'll find them on YouTube as well
there um if you leave us a review we
think that will help other people find
our content and that would be pretty rad
and then finally you can check us out on
Instagram we have the ATM materialism.
podcast page um you can connect with us
there leave a comment WE Post lots of
behind the scenes stuff and videos and
pictures we think you'd like
it and lastly we'd like to give a shout
out to alphabot and kabite for making
the music for the podcast they both make
a ton of really cool synth wve music and
you can check them out on Spotify and
YouTube catch you next time the
inventors of fire electricity magnetism
iron lead glass silk cotton the makers
of tools the Capt of lightning the
architect the engineer the musician are
all beneficiaries of the materials of
this world and are bound only by their
imaginations in manipulating those
materials