Route to IEC 61850 (2017): Managing substation communication networks via IEC 61850 services

OMICRONenergy
9 followers
Oct 16, 2023
Dr. Martin Jenkner of Moxa talks about managing substation communication networks via IEC 61850 services.
Recorded during the OMICRON Power Utility Communication Tutorial & Workshop 2017 held in Germany.
www.omicronenergy.com/puctw
I'm with MOXA. A couple of you may know MOXA already, probably from something totally different.
A couple of you may have used our serial products already. At least that's what I learned this morning.
Ah, MOXA, you were doing device service. That's indeed true. MOXA is a 30-year-old company.
We are going to celebrate our 30th anniversary this year.
The company was founded in 87, in the late 80s, basically just developing multi-port serial boards at that time
when the industry discovered that there are so many things to be controlled with a PC.
At that time there was no Ethernet, but the people were starting to automate and use communication.
MOXA has made its way since the 80s, following the track consistently from these multi-port serial boards
moving into the upcoming Ethernet, away from host-based architectures to network-centric architectures
and has ramped up a switch port for you in the last 20 years.
Switches and device service, or I should say serial or even what we call edge connectivity oriented products
continue similar revenue to our business in 2017.
We have been adding for the last 10 years additional product lines, starting from IP video cameras
going specifically into embedded computers.
Also having wireless products, for example, in real applications as well as I.O. products.
MOXA is serving industry only. You won't see us neither in the consumer space nor in any enterprise area.
We are focusing on vertical markets, namely power and rail, but we also serve oil and gas, factory automation,
intelligent transportation systems, marine.
Today I will show you what we can provide for the power market.
When I sat down with Fred to agree on the title, it was so important for Fred.
He said, you need to say communication networks, because you are in the power industry
and if you talk network, they are going to think about something totally different.
But excuse me, please, I will make it short.
Whenever I talk about a network, just think about an IP network.
And what this is about today is basically talking about managing networks.
And again, after I had agreed with Fred, I said, the topic is clear, but if I would do this,
I would be finished in three minutes.
Because I would tell you something that you already know.
The bits and bytes on how the things that I am going to present work is very well known to you already.
And when I had a second thought on this, I thought, why is it still not so popular,
what we have been proposing for a couple of years?
And it came to my mind that whenever I talk to customers promoting the MMS server capabilities
of the switches we have for the power market, there are two types of customers.
One customer says, oh, that's interesting, I can integrate this into my power scaler system, how beautiful.
I will show you how it works later.
The second customer tells me, go away, I have a network management software.
And sometimes it may be, it's yours, you have been selling me the network management software already.
And indeed, Moxa does also do network management, generic network management software.
So what I want to discuss today is basically what to choose, when, what are the differences, what are the benefits.
And that's how you see the agenda.
I will first, in the first half basically, I will walk you through what is general network management about today,
what tools are used, what is customers' expectations.
And then the second topic is basically the core.
It's about what I call scalar integrated network management,
because this is what IEC 6150 services for network management is about.
And you will see an example that I give you why I made such a kind of generalization.
You will see an application example, which will lead you also to the demo that we will do in the workshop tonight and tomorrow.
I'm here with Patrice Roussel, that's our application engineer, who will help you on the technical questions.
And in the end, I'm going to give you a comparison and outlook, and actually no recommendation,
because as you all know, it's eventually down to the customer to choose what and when.
So why network management?
Well, there are a lot of reasons, and if you ask people what is network management about,
they come up a lot of, very often with things like, I need it for troubleshooting, quick error detection.
Overall, my business scenarios, I need lower down times.
Networks are a critical part of infrastructure, blah, blah, blah.
It's so important, I need that.
People are focused on errors and issues.
But once you look into popular network management software, you will see that network management actually starts way earlier.
It starts from the deployment phase.
And when I saw your slides this morning, I realized that the PowerPoint clip art you were using is very similar to the ones that we have.
It's kind of an agreed way of how network management looks.
I will show you in a second.
And when I listened to Thomas this morning, I also understood that we're looking into distributed networks these days.
The classical pyramid is, I wouldn't say it's over,
but it has to be realized if you look into the definition of industry 4.0, there's one term they say that it needs to be horizontal integration.
So we're looking into inter-substation communication.
We're looking into remote networks where you are not part, but you need to know what's happening in those networks.
So these are all new items when you look into network management.
So let's start about what is network management, general network management, about.
And I dare to say general network management is life cycle management.
Because you start from the installation phase.
All those tools, they cover installation, operation, maintenance, and diagnostics.
And what I will show you in the next couple of slides is basically you can sort that into each of these buckets, basically.
So installation, yes, easy configuration, operation efficient, monitoring, easy backup, quick troubleshooting.
These are the items that customers are looking for.
So let's start with the easy one.
That's what everybody thinks when you look into network management.
It's about visualization, of course.
You want to know how does your network look like, who is talking to whom, who is who.
You need to understand the nodes, what kind of switch,
what kind of end device actually is linked to which other device.
And then in the second round, you want to look into how, what's going on.
So you want to have kind of a live view on what's going on.
That is kind of, you want to see, understand the bandwidth node, for example, and the link.
You may want to look into the current configuration of a certain node.
And even more, when you go down to troubleshooting, basically, you want to know if there's anything wrong.
You want to see this highlighted.
And very easy, if a link goes down, a network management software, at least the visualization part of it,
should show that this link is red, broken, whatever, to show here's the issue.
Now, typically networks have the characteristics that if one link breaks down,
you will not see anything which is behind that link.
So how can a network management software know what's happening in all those devices when this link is down?
And in fact, this is one of the major challenges when you want to visualize the network,
because before you could do this, but that's not interesting.
So you need to know what happens when.
And in complex networks, a single failure can cause a cascade, an avalanche of succeeding failures.
And of course, a smart guy who knows the topology inside out will know,
ah, okay, if this part of the network is dead, there's a problem with this link.
But that's an experienced operator.
That's actually the task of a network management visualization tool.
So typically in the old days, you were looking into log files telling you in numbers and text who was doing what and when.
And there's even here some visualization in terms of how severe is what happened.
That is totally distracting, actually.
But that's the core databases.
What we have been implementing in our general network management is a timescale where you can run up and down
and see what happened at which data.
So you see the network status locked for a while.
And you basically visually run through the events that happen in a network.
And when you try this, it's really easy.
You can easily identify this link broke down and you lost this and you lost this.
And the thing is done.
That's about visualization.
I talked about links and device.
And of course, if you have a device, you want to click on it.
This is the time of tapping, clicking.
I want to see how it's configured, what's working.
I want to see the details.
Once you do this, you easily get into the next challenge and the next requirement.
Well, if I can see what's the configuration of this node, I want to change it.
So, of course, you can have an online change of the configuration.
And then, well, next things are, as you have the configuration, as you can see the configuration of each node,
the next step is, of course, you just sum up all the configurations and you can save them in one file.
Or in the deployment phase, you have already the configurations pre-configured and you let the network discover itself and then you deploy it.
Various ways of doing this.
But in general, network backup automation, that's the key word, is automatically backing up the configuration of the network.
Now, step by step, you get into something which is even more comfortable.
Assume you have a healthy network at a certain point in time.
We can take a snapshot, basically a dump of all the network configuration in terms of topology and configuration,
save it for the time when your network has failed.
Visualizing that something failed is easy, that's what I showed you before, but finding the root cause is not that easy sometimes.
So, what you do, basically, is you compare the networks.
And in this case, you can figure out between the two snapshots that you did, that in this case, for example,
an access port was reconfigured to become a trunk port and that's why the network failed.
That's basically state of the art, what you do with networks.
And I want to go now into a new topic, which I think is very important these days, and that's cyber security.
Cyber security is everywhere.
Every conference I'm in, every customer I talk to, at a certain point in time, cyber security comes in.
The issue is, everyone needs it, no one really knows how to define and how to comply with.
There are a couple of standards.
There's IEC 62443, 624, 62351, there's the BDE W white paper, there's NERC SIP, okay?
And they're covering all different aspects of cyber security.
What I picked here is a 62443, which is a nice standard that defines cyber security on different levels of applications,
starting from the production of a device, going to, number four, the device itself, going into systems,
going into deployment, and some basic things.
So, IEC 62443 has a couple of chapters, and the most interesting chapter for network deployment is chapter four,
which deals with the device itself, basically the cyber security status of the device,
that says, is the user interface protected with a password?
Is the password according to the rules that have been set up?
Is the access to the device based on user rules?
A lot of things which in 62443 have been combined and put to certain levels.
You can comply on a, in chapter four, your products can comply to a cyber security hardened status level one,
and also a level two.
And what we did is basically, we integrated this into our network management software,
that scans all the devices, at least the Moxa devices, and looks into how much is that cyber security level,
how much do they comply to, okay?
So, in one snapshot, you get a quick overview of the cyber security status of your network,
and of course, you can then, if you find some trouble, okay, you can check the security wizard,
you can go into the device, say I want to comply to four, to level two,
and then you see what you need to change in order to comply, in order to come back to a healthy status.
So, high, medium, and basic are the ones that we show here as a status,
and you see what you have to comply for, on which level, in order to achieve that level.
So, this is what a general network management tool can do today, that's quite new, honestly speaking.
And the last word on that is, I already mentioned Industry 4.0, distributed remote networks,
that's also part of network management, because the network that you want to monitor is not always the one where you're currently in.
For cyber security reasons, it is better that you're in the network in order to see what's going on,
but at least you want to be notified in a remote substation if anything goes wrong.
So, remote access associated with a secure access, typically using VPN, is the way to go,
and you see that is today already possible to do this on mobile devices.
We have actually an app developed that will access the server where we run our network management software,
which will send, through a VPN tunnel, the notifications of what's happening in the network.
On a very high level, this is not intended to manage the network with all the details that I just presented.
It's more like a quick information and high-level overview on the status of the network.
So, this is the beauty that you can do with a general network management suite.
So, what's the benefit of a SCADA-integrated management? Let's look into this.
And I'm going to give you an example, and this belongs to the part that we should open our view,
which is happening in factory automation already. What's happening there?
Factory automation has a very famous protocol that's just unit IP.
PROFINET is one of the most famous protocols being used especially in Europe with a high growth rate.
And PROFINET works like there are PROFINET controllers and there are I.O. devices.
The I.O. devices are basically part of the field level.
The controller is on the management level, typically on the control level here,
and they exchange data in an interesting manner.
They have roles, and actually there are parameters of the devices that have to be monitored.
And in order to show the capability of the devices, PROFINET requires that devices have a GSD file,
which shows the capabilities of the device.
What does this have to do with switches?
Basically, switches are supposed to be transparent for PROFINET.
You can use any kind of switch to transfer PROFINET. That's the architecture of PROFINET.
But if you want to get a qualified PROFINET certified switch,
you have to comply with a couple of things that you do in managed switches,
like quality of service, queuing stuff to ensure that the performance of a PROFINET certified switch
is according to what you need when you design a PROFINET network.
Not too hard.
But what we have been developing for the last years is,
we said the switch should show up in the network as a network device, as an I.O. device actually,
to make it visible to the SCADA system.
So far, the SCADA system will only see the meters, drives.
What we said is, in more complex PROFINET networks, you better also visualize the status of the switch.
So what we did is, we provided a GSB file for the switch that shows the capabilities of the switch,
what you can monitor in terms of SNMP, link up, down, power status, RX, TX,
typical values that you would scan with an S via SNMP.
And with this, the device becomes visible in the SCADA system.
And that's what I call a SCADA integrated network management.
Now, if you look into the orange part of that slide,
I think for many of you, especially when you see GSB files,
the bells are ringing because that's very much what IEC 61850-based services network management is about.
The same story. We have a protocol, not PROFINET, but we have MMS.
We have end or process level devices, which are defined as IEDs.
They have properties which are defined by logical nodes.
And they show their capabilities using IED files.
It's the same story in very different models.
So there must be something behind it.
So it's not a fancy proposal just to use 61850 in order to do network management.
There's something behind.
There's a reason why people want to integrate network management in a SCADA system.
Now let me just quickly go through for some examples what you will see in switches.
This is the object model of 61850, very well known.
You will see this kind of data, and I invite you to the demo.
You can use the IED scout from Omicron later tonight or tomorrow to browse through the demo box that we have.
We do not have only switches.
We also have redundancy boxes in the network to monitor PRP-HSR status.
And that's what you will see.
We will use MMS, of course, as a communication protocol.
So the status of the switch is stored in an MMS server, which will be approached by a client,
which is typically the SCADA system, to either read out what the switch is doing
or even to write something in order to configure the switch.
So that's how it looks like.
That's what you're going to see.
Not tonight.
This is now the general view on that.
You see basically that not only IEDs do have an MMS server but also switches.
So they appear in the same manner.
And here you see a screenshot from Copadata Xenon that they kindly helped to do for us,
also in the demo box tonight and tomorrow.
And in detail it looks like this.
And I think it's worth to have a quick look at that screenshot
because here you see some of the benefits of a SCADA-integrated network management.
The key focus is this user interface.
First, it's very structured.
And it's not structured to anything which is related to the network.
It's structured according to the structure of your substation.
There's a process level.
So you see already which redundancy box in this case, or even a gateway, is on process level.
You see on bay level we have two red boxes which work in coupling mode.
What they do here is they couple the HSR.
These are the boxes which are in coupling mode.
They couple this purple HSR into a higher level PRP network.
This PRP network, that's something that's very important for us to tell,
can hold RSTP structures, can hold other redundancy structures like the Moxar Turbo Ring,
very similar to Hyper Ring architecture.
We have another two red boxes which are also added to this PRP network.
And last but not least, you see our high-performing substation computer
which is 6150 certified, i7 platform with a quad-core processor,
which is running the PowerScaler system.
Actually, it's so powerful that it can do even more, not just running the Xenon.
It can also do data logging, data processing.
That's what it's good for.
Be invited tomorrow to check it out.
This is the same thing if you look into MXView, which is our general network management software.
You see the same structure. You see the HSR ring, two red boxes in coupling mode.
You see Turbo Ring here and there with standard switches.
You see two red boxes attached to LAN A and LAN B in the PRP network.
And you see the computer, which is hosting a hardware-based implementation of PRP-HSR.
We have a dedicated card that runs PRP and HSR as a PCIe card in our computer.
And it's attached to LAN A and LAN B.
You see the network is different. It really focuses much on the properties,
which port, the links, and you don't see much about the underlying architecture.
You can move this around, and Patrice did a good job in arranging the nodes
in a way that resembles exactly the structure that you saw on the Xenon interface.
But honestly speaking, a network management tool doesn't really care.
If you auto-plot the topology, it will just plot you a topology which is right,
but will not resemble what you have in your substation.
So, that's a comparison.
And the last comparison I want to do is still on browsing level.
If you compare the MIB file, which is holding all the SNMP data,
you see this structure comparing again to the IAD Scout.
And please be invited again for the demo.
You can do it yourself. You can browse the structure.
You can see the difference and experience yourself the details of it.
In fact, the same data is available in both types of views.
So, what's it all good for?
This is the time of redundancy. I talked already about PAPHSR.
Also, Belden mentioned this. Very important for substations.
And it's about something that I call the spare tire topic.
The ones of you who have just bought a car, have you?
Who has a spare tire still?
Or do you use the spray? Yes, I think many of you.
Who checked the pressure of the spare tire when you bought it?
Or did you ever check in any car that you own the pressure of your spare tire?
Honestly, I don't.
And I'm always lucky when I have a breakdown, a flat tire,
I get into my trunk and I find it's enough to get to the next petrol station.
So, that's redundancy.
You have five tires, four are used, and one is in the trunk.
This analogy is not adequate to PAPHSR.
Because in a PAPHSR car, the tires would permanently be swapped.
And actually, you would never really know which tire is currently in use.
Actually, if you have one connection from A to B,
you would like, when you see this with TSN,
you would have actually five tires available, only one being used.
You would never know which one is currently in use.
But that's important, especially if you don't have the luxury scenario of five over one,
but only two over one, because it's so expensive.
That's what I call the redundant network pitfall.
Because the idea of redundancy and of those networks is that they provide a black box.
Well, you don't care, you just have it in and out.
And if you look deeper into it, you see, of course, that's a very simple PAP.
The easiest, the smallest PAP scenario you can ever do.
You have two parallel redundant paths.
And if you look into the pressure of your spare tire,
you would see that only one path is physically healthy,
while the other one has, for whatever reason,
a broken or degraded fiber connection or a bad connector, a success rate of maybe 60%.
This doesn't matter to the operation, because you have enough bandwidth,
that's a concept of your network, to drive your operations.
But what happens if exactly this path breaks down for whatever reason?
Then you get stuck, and then you figure out, you open the trunk,
and then you see, I have another flat tire.
So, that's what I call the challenge.
And that's what you should do, is you continuously should monitor your tire pressure.
Not in your car, I think that's just overkill,
but in substations you really want to do this, because this is critical infrastructure.
Now, what does this have to do with network management and PAP-HSR?
First of all, let me just tell you that in 62439-3, which is defining PAP-HSR,
all the parameters are in there to help you to monitor the statistics of the packets
that have been duplicated, and especially, which have been received or not.
Typically, you don't care. In a healthy situation,
the redundant packet will be just dropped by the receiving ID or redundancy box,
and you don't care, because it doesn't hold the necessary information.
But you should know, if you're dropping 8 out of 10,
because the time you're going to need the other part of the HSR ring, you will fail.
So, the information is available, and how do you do this with 61850?
Well, basically, you just make this data also available,
not for 61850 services, and that is using MMS service.
So, here you see a very kind of wild scenario.
Again, kind of the same thing, LAN A, LAN B for PAP-HSR,
which is, according to the standard, in blue and green.
You see an HSR, basically two HSR rings.
You even see RSTP chains added to it.
And every item, not only the IDs, hold MMS service,
but especially the red boxes, which work either in coupling mode or in SAN mode.
They have this capability.
You also see a PAP-HSR-capable switch, which is also part of the demo,
which you will see tomorrow.
So, that's what it's about, the benefit of SCADA-integrated network management.
And as I promised in the last slide, it's kind of an attempt to compare
what are the pros and cons.
If you look from the graphical user face point of view,
the generic or the general network management is very much
oriented to the needs of network management.
Remember the screenshot of MXFuel.
He doesn't care about phase and process level.
While the Xenon one is very much tailored to the needs of the operators.
In Xenon, you can just simply click on another tab,
and then you're in the single line diagram.
That moves network management very close to the operations monitoring of a substation.
It is not intended to make operators do network management in substations,
but the idea behind that is to lower the threshold of looking into the network.
In other words, we want to extend the time when an operator has to call the IT department
because the network is down.
Well, communication protocols are to be used.
That's interesting, yes, of course.
In generic network management, you use SNMP, you use LLDP,
which are the most widespread and universal protocols to do topology discovery
and device discovery.
Of course, you're limited in SCADA-based systems too.
The communication protocols, which are the process-oriented protocols,
it is indeed a little bit of a limitation.
And the data structure, of course, yes, that's a general one.
And this is here based on profiles.
And that looks also like a restriction,
but eventually profiles have been invented not only in 61.850,
but also in Bluetooth to make the life of users simpler.
If you have a profile, you already have some presets.
If you tackle a problem, you typically have application-specific requirements,
and these are typically part of the process.
Last but not least, of course, this is application-independent.
That's why it's called a general network management tool.
It's for any kind of application, so it can be used everywhere.
This is application-oriented, but still it has the drill-down capability.
It merges, it's a unified approach to look into your network.
Yeah, that's my point of view for the comparison for the future.
You already saw me mentioning cybersecurity when we look into SCADA
integrated network management.
I think there's something ahead of us.
Also visualize cybersecurity-relevant information in SCADA
integrated network management software.
Thank you.
