noc19-cs33 Lec 30 Parameter Servers

IIT Kanpur July 201
20 followers
Dec 3, 2023
[Music]
[Music]
parameter servers previous content of
this lecture in this lecture we will
discuss the parameter servers and also
discuss the estelle synchronous parallel
model in machine learning systems
especially the scalable machine learning
algorithms that we are seeing for big
data computing requires the abstractions
for the scalable building scalable
systems and here we can see that there
are three different the machine learning
systems landscape we have to see that
there are data flow systems which are
built over Hadoop and stack platforms
using the graph system algorithms and
shared memory systems and these machine
learning algorithms which are nothing
but supported as the dataflow systems
such as naive A's and requires an
abstraction from the underlying systems
like Hadoop and spark and for graph
systems it requires abstractions from
the graph lab and tensorflow and
similarly for shared memory systems and
these are motion and DMT K also requires
the parameter servers so what we will
see here is that in in a simple case the
parameters of machine learning are
stored in a distributed hash table that
is accessible through the through the
network which is there in the shared
memory systems so parameter servers
which are used in Google Yahoo and also
is used as an academic work by Smola and
zing and now we have to see the
parameter server and its how it is going
to be used
the framework for machine learning so to
build the scalable machine learning
systems so skill machine learning
framework requires the use of parameter
server and it distributes a model over
multiple machines and offers two
different operations one is called pull
that is the query part of the model and
the push means update part of the model
so in any machine learning update
equation which is shown here a wi which
is updated SWI plus little delta is to
be handled into the model by push that
is in the form of plus operation so in a
stochastic gradient descent and all in a
collapsed Gibbs sampling for a topic
modeling this aggregate that is the push
updates is done via the addition
operation so whenever this addition
operation is there so these parameters
are to be pushed into the model and that
is handled using the parameter server
now this spark machine learning is done
with the parameter servers so here in
this particular scenario we see that
spark driver and spark workers the
together work and support this parameter
server notion
similarly this kind of parameter server
notion is also supported in the glint
and in the glint we can see there is a
spark there is a parameter servers and
in a spark there is a spark driver and
different spark workers are there so we
can see here in this particular example
that that these parameter servers are
either maintained in the glint as the
queues as the distributed queue and in
the spark as a and Driver worker
combination
so parameter server is used in the
training phase also so in the training
state is stored in the parameter charts
and asynchronous updates are being
handled so this is the example figure
which shows that the the models are
stored as a part of parameters charts
parameter server charts and whenever
there is an update so it has to be
updated at all the charts which is which
is managing the parameter server so the
parameter servers are maintained in this
big data scenario that is through the
through the clusters that we are going
to understand here in this part of the
discussion so parameter servers are
flexible and here the number of number
of model parameters which is being
supported is quite large and number of
cores which are there which supports
this parameter server is shown for
different machine learning landscape for
example topic modeling and matrix
factorization and CNN that is
convolutional neural networks and DNN
deep neural networks all requires
flexible parameter servers and huge
amount of parameters are complex
parameter servers are being managed
efficiently therefore it is shown here
in this example figures so let us
understand about the parameter servers
so parameter server now comprises in its
park prices of the server machines and
the worker machines and it models the
parameter servers as are stored in the
parameter server machines and accessed
via the key/value interface that is in
the distributed as per the distributed
shared memory so extensions of it are
multiple keys are available for matrix
operations and multiple channels for
multiple sparse vectors and multiple
clients for the sims for the same
servers so the extensions which are
available here in the parameter
server is in the form of push-pull
interface to send or receive most recent
copy of the subset of the parameters and
the blocking is optional and the
extension can block until the push pulls
with a with a clock which is bounded by
t minus rho is completed so let us see
the data parallel learning with the
parameter server and here the parameter
servers are shown in this particular
picture that they are that they are
being exchanged through a different part
of the model is available on different
servers and the workers they retrieve
the part of the model as it is needed
so therefore the abstractions are
provided which are internally
implemented as the data parallel
operations in the parameter server such
as the key value API is for the workers
that is they can get the key and also
that means they may get the model values
and by gate key operation similarly I
had key with the wither Delta to be
updated in the parameter server and this
particular addition can be performed
here using add operation so let us see
how the iterations in a MapReduce is
supported using the parameter server so
this particular iterations in the
MapReduce is often required to implement
the scalable machine learning and this
requires the training data to be
provided to the different map phase and
which are to be read in two different
splits or in the form of the shots and
then they will redundantly save the
output between these stages in the
MapReduce function and therefore
increase the cost of this iterations of
the MapReduce
and therefore the parameter servers are
very much needed to make this machine
learning scalable machine learning every
support so this particular parameter
servers requires still synchronous
parallel model we will understand about
them so here we have to see these model
parameters are stored on the on the
parameter server machines and and
accessed via the key/value interface
using distributed memory so iterative
machine learning if we see the the
workload which comprises of these
parameter transformations that is the
data is transformed and by the worker
nodes and in between they are accessing
the model parameters and for example in
the topic modeling and matrix
factorization support vector machine and
deep neural networks they all are
iterative machine learning algorithms
wherein the workers requires these
parameters to be updated parameters
means model parameters are to be updated
so therefore a parameter server has to
maintain this model parameters and the
all the workers will now communicate
with the model parameters server and
these values are either read or that
means they will either get or they will
put these values on this particular
parameter server
so here we can see that in typical
MapReduce model where the data model is
the independent records and the
programming abstraction is in the form
of MapReduce and the execution semantics
which is used as bulk synchronous
parallel in the MapReduce to support the
map to support the iterative machine
learning
now if we compare this with the
parameter server here the data model in
the parameter server is also independent
records and the programming abstraction
is not the MapReduce but it is simply
the key value store that is in the form
of a distributed shared memory to be
provided now what is the execution
semantics is it the bulk synchronous
parallel or some other synchronous that
we are going to see now the problem here
is that the networks are slow that means
when these parameter servers which are
maintaining the parameters of the model
and using gate and adding the keys are
the operations which are to be performed
on the network there becomes quite slow
as the network is slow compared to the
local memory axis so we want to explore
the options for handling this with the
help of the caching now the caching will
introduce a problem that is to be
handled using the cache synchronization
and here we can see that the the the
cache which is maintained in and
whenever sparse changes to the model is
happening and repeat is done through the
cache then these parameter servers
requires this particular cache to be
synchronized
that is called cache incrimination so
having implemented the cache
synchronization in the parameter servers
the problem of accessing the parameter
server across network also can be become
efficient if the access is done through
the cache the second solution to
implement the efficient access of the
parameter server on the network is about
the asynchronous operations so here the
machine one machine to machine three
they will perform another computation
they will iterate and now they have to
wait till others other iteration they
complete their their iterations and then
they will interchange their
communication and then the computation
can go on
so the in every iteration we see that
there will be a barrier after the
machine finishes the iteration and start
the communication and before all the
communication completes so that the next
iteration will begin and these are the
barriers and if we collapse these
barriers now we can introduce a much
efficient way of implementing this a
synchronous asynchronous here in this
case so enable the more frequent
coordination on the parameter servers so
if we support this asynchronous
execution environment with the parameter
servers then the the scenario will look
like this that the efficient the the
operations in the parameter server will
become more efficient for example here
the machine one machine to machine three
they access different parameters W 1 and
W 2 and W 3 from a different parameter
servers which are shown logically W 1 W
2 W 3 they are different parameter
servers and they keep on accessing by
different machines and they internally
they communicate the results back to
this particular server so this
asynchronous operations are if it is
there then let us see the problems in
this asynchronous execution so
asynchronous execution lacks the
theoretical guarantees as the
distributive environment can have
arbitrary delays from the network and
stragglers so therefore what we see here
is that if we see a particular
implementation of the scalable machine
learning algorithm let us say that we
say the delayed stochastic gradient
descent algorithm here we will see that
F is a function loss and X is a
parameter which is being computed in
this algorithm and which required to be
updated and this particular updation let
us see what problem will create during
the using asynchronous communication and
therefore we require to make about
execution semantics and decided so that
this particular a synchronous operation
should not be a bounded unbounded so it
has to be a bounded synchronous
operation which has to be supported so
the parameter servers will use the model
which is called still synchronous
parallel model where in the global time
is let us say T and the parameter
workers will get and can be out of the
date but cannot be older than t minus
tau so tau is the tolerance which is
introduced and therefore it is called
bounded synchronous so tau controls the
stillness also called as stale
synchronous parallel so still
synchronous parallel execution semantics
we are going to see here which is being
supported in the parameter servers so in
the stale synchronous parallel we see
here is that they will these are workers
1 2 3 & 4 so these workers now they will
be changing the parameters as
synchronously and the stillness is also
a bounded by s is equal to 3 that means
now the here we can see here is that the
worker 2 will be blocked until worker 1
reaches the clock of clock of 2 and so
that means the updates will be
guaranteed and this is black means
updates are guaranteed and green means
it is visible to the worker 2 and blue
will indicate that it is in complete and
incomplete updates sent to the worker 2
and not yet visible
updates not yet sent to the worker too
so therefore this particular slots which
is divided and stainless bounds s is
equal to three which is being calculated
so this way the interoperate interpolate
between BS p and the as synchronous
modes and they subsumes both allow the
workers to usually run at their own pace
and the fastest Orlick slowest threats
not allowed to drift the clock UI that
means more than s clocks are parts so
efficiently it implements the cache
parameters in this particular manner so
consistency here matters because
stillness has to be bounded by BSP which
is a strong consistency but we relaxed
the consistency and stainless stillness
is bounded so therefore suitable delay
will gives a bigger speed up here in
this particular case so it still
synchronous parallel if we introduce
then we will see that it is it is giving
the better result and ensuring the
problems of the asynchronous operations
so in this lecture we have discussed the
parameter servers and the is still
synchronous parallel model of operations
which is supported in the parameter
server thank you
[Music]
[Music]
[Music]
