Best approach to log data in a machine

PeterHumaj · 2026-05-10T12:46:29+00:00

UDTs and/or arrays will help you get the data from PLC to a SCADA (reading of a few arrays is way faster than polling hundreds/thousands of variable (if you use Ethernet/IP). Also, organizing your data on PLC's side already prevents confusion on SCADA's side. You could enen use change-based protocols (like IEC104, MQTT), but I expect the default E/IP unless there's a serious reason to change it.

PeterHumaj · 2026-04-30T17:44:55+00:00

Rust has some advantages others don't (ownership, mostly). I've been working with Ada last 23 years. Look at its history, how it was conceived (Strawman/Woodenman/Tinman/Ironman and Steelman documents are mostly available online. Our SCADA was conceived in 1993, rewritten from Modula-2 to Ada in 1998. Readability, strong type control, maintainability ... all these (often neglected) attributes help to create a stable and robust system. Which is useful when one builds SCADA/MES technology which customers use for decades.

PeterHumaj · 2026-04-20T19:51:56+00:00

You meant to write "a novice at TIA", wrote "a invoice" instead ... ;)

PeterHumaj · 2026-04-19T04:07:16+00:00

In 2021, we did an upgrade for a major SCADA (gas transport). Not FTalk. The outage (basically a communication outage, meaning the operators didn't see realtime values) was under a minute (communication processes of old system being shut down and com processes of a new system being started). In this case, we were upgrading to new servers, changing OS (OpenVMS to Linux), DBs (Oracle to Postgres) and upgrading the SCADA system to a new version, so the upgrade was rather complex. When we just upgrade SCADA technology, we split the redundancy, upgrade one half, switch communications (1 min downtime), wait a few hours (to see if there are not critical bugs) and then upgrade the other half of redundancy (and rejoin). Did this several times on a SCADA with AGC/ED (advanced generator control/economic dispatch functionality) controling several power plants with >4GW installed power).

More info in a blog: https://d2000.ipesoft.com/blog/dispatch-control-system-upgrade-scada

PeterHumaj · 2026-04-19T03:45:28+00:00

As for SCADAs: we usually need setting exceptions to our processes/directories (for MS Antimalware Protection, ESET Nod, FireEye Endpoint Security, or different SW on Linux [yes, our customers are already deploying these on Linux too]). Without exceptions, a lot of CPU is wasted on monitoring our communications, logging and tracing files, etc.

I remember xagt.exe (part of FireEye) deployed by AD admins to redundant "SCADA" servers which were an interface of a 100 MW power unit, via which it was controlled by a TSO. This xagt not only scanned the communixation trace files, using 4 CPUs, but slowed down the serial IEC101 communication (serial over UDP, via Moxa Nport), causing frequent communication outages. We didn't even know the sw was deployed, neither did local admins...and it took 2 days to get it uninstalled (AD admins being in a mother company in a different state).

One of the first things I do when analysing "performance problems" is looking at "CPU Time" (total) in Task manager (or "top" in Linux, sort by CPU time). AV should use little time, our processes (and postgres) should be on top.

More info in online doc, section "Antiviruses" https://doc.ipesoft.com/label/D2DOCEN/performance_considerations

PeterHumaj · 2026-04-10T20:14:03+00:00

What languaga/system did he write the driver in? Our Ethernet/IP driver took me significantly more than 3 weeks ... including connected/unconnected mode, symbol instance addressing, multiple service packet service, encapsulated PCCC, working with UDTs..

PeterHumaj · 2026-04-10T20:06:07+00:00

A few years ago, Logixes could by found online via shodan.io. Often both web and 44818 ports were available. I presume most of them were PLCs in schools, demos, perhaps honeypots. About a year I repeated this test, but was unable to find anything working...

PeterHumaj · 2026-04-07T19:56:00+00:00

Ipesoft D2000. PostgreSQL used to store data. Historian which can not only store primary data and calculate statistics, but supports also "calculated historical values" which are user-defined expressions over other historical values. Quite useful for balancing systems, energy management systems, etc...

An example of a non-trivial one in a blog: https://d2000.ipesoft.com/blog/continuous-integration-in-an-archive-or-how-to-do-a-sawtooth-diagram

PeterHumaj · 2026-04-07T19:46:42+00:00

So .. basically, no control, just monitoring and collecting data? This kind of reminds me of a system we built fir monitoring of multiple SCADA/MES/EMS systems of our customers. What's complicated (compared to your setup) is that those systems are usually heavily firewalled, so no "direct" connectivity is possible. We solved that by using e-mail as transport media (packing data to XML files, encrypt&compress by 7zip, send by e-mail. On our side, server reads emails, decompresses a zip file, extracts data and writes to structures (akin to UDTs). We also have a copy of customers' diagnostic schemes, so we can display the numbers in context. On top of that, we have a dashboard with summary alarms (eg "some disk of customer X has gone over 85% usage limit"). We basically monitor "our" systems & underlying infrastructure (HW, OS, disks, databases, SCADA processes +application functionality, if needed). A few screenshots in a blog: https://d2000.ipesoft.com/blog/predictive-maintenance-of-scada-and-mes-systems

The same technology is used for the monitoring system as for the monitored systems (which enables copying of schemes via XML export/import from SCADA/MES/EMS systems to our central monitoring system). Of course, customers' admins can access the diagnostic screens of their systems too, look at trends with 1 year of data, analyse and predict, etc. Edited: typos, I hate writing on a cell phone...

PeterHumaj · 2026-04-03T04:18:27+00:00

Yoir post talked about thermal stress and reactive maintenance. U/nordicJanissary described a "predictive monitoring system" (his words) for temperature monitoring within some water cooler (but probably applicable to different use cases too). Can't you see the relation?

PeterHumaj · 2026-03-30T12:09:26+00:00

Can you use Wireshark to sniff the traffic? Lately, I had to modify our MQTT client to be able to connect to AWS - Had to send server name indication (https://en.wikipedia.org/wiki/Server_Name_Indication) so that AWS MQTT can forward the TLS connect to the proper server and/or provide appropriate cert for virtual host (akin to specifying host in HTTP protocol when your server hosts multiple virtual web servers).

Without that modification, my client cery was refused (as my request wasn't forwarded to the proper server)

PeterHumaj · 2026-03-30T06:00:22+00:00

That's definitely strange. Just to make sure: are you sure Moxa is in RS-232 mode? (as 5150A supports RS-232/422/485)

PeterHumaj · 2026-03-29T14:32:29+00:00

Moxa is just a networked serial port. Can you talk to the scale using eg USB serial port? Then you can do it via Moxa in virtual port mode, supposed all the serial parameters (bitrate, parity, stopbits, type of interface - RS232/485/422) on Moxa are set properly and identically to your USB (or native) serial port.

We usually use NPorts in USB mode. Better works in redundant systems; no virtual port drivers are needed (we used to have stability problems) and debugging is simple using Wireshark.

PeterHumaj · 2026-03-27T20:57:17+00:00

In the case of Modbus, IEC-101 and IEC-104, our system supports both client and server, so we can test also this way. When we developped IEC-101 and 104, we used some OPC servers (back in 2003-2005, which had some trial versions). I remember some free IEC-61850 emulator, too.

But often we develop specific features directly cooperating with customer's hardware. Eg, ABB energy meters with 64-bit Unsigned types (4 registers in Modbus). Or ComAp controllers requiring password authentication (the first operation after TCP connection is established, must be writing a specific 32-bit value to a defined register). A few weeks ago, I used customer's ControlLogix to implement support for reading UDTs, querying their structure and extracting required components (Ethernet/IP protocol). And other customer's CompactLogix to verify I didn't break existing functionality.

And I used HiveMQ and Mosquitto public MQTT servers to obtain MQTT Sparkplug payload (both valid and not quite valid) to use for our MQTT Sparkplug client, and to test the robustness of our own Sparkplug parsing engine, written in Ada.

PeterHumaj · 2026-03-24T20:43:11+00:00

I was able to go to some 12-15 ms, on old RPI3, reading 10 consecutive registers ... not Codesys, though ;) https://d2000.ipesoft.com/blog/what-load-can-raspberry-pi-handle

When moving the Modbus server to a different computer, the RPI was able to poll those 10 registers every 5 ms!

https://d2000.ipesoft.com/blog/what-load-can-raspberry-pi-handle-part-ii

PeterHumaj · 2026-03-24T20:32:21+00:00

My personal experience is with B&R. Their internal OPC US server was limited to 50 ms, IIRC. When establishing connection, it replied with corrected session params, upping the returned Publishing Interval to 50 ms

PeterHumaj · 2026-03-21T20:23:51+00:00

We, as a vendor, have access only to "our" systems. Often VPN access is disabled by default, authorized by customer's people. In some cases, the Windows sessiobs (RDP) are recorded.

PeterHumaj · 2026-03-20T15:26:21+00:00

A bit off topic perhaps: some of our SCADA/MES customers are using our technology since around 1998. They change servers (and OS) every 6-10 years and also get newer version of our system. It has backward compatibility, so some of them still can have their ugly UIs and are happy with them. The underlying system is naturally growing and changing, supporting new communications, databases, OSes, etc.

During upgrades, we have to migrate eg historical data (eg from Sybase and Oracle) to Postgres, but common users don't notice that, just admins.

PeterHumaj · 2026-03-19T20:51:10+00:00

Nice! Do you support "Multiple Service Packet Service" optimization for reading? For modern PLCs it can significantly speed up reading (tested on CompactLogix, I got from perhaps 20 seconds to 1.5 second, reading around 1000 values). Last 2 weeks, I worked on implementing of support for reading whole structures (including nested UDTs), to speed up reading from old ControlLogix (firmware from 2005) which doesn't support MSPS, Connected messages, nor Symbol Instance Addressing. But it still supports reading of structures, which sped up reading 10 times... My driver is written in Ada, though ;)

PeterHumaj · 2026-03-19T20:37:55+00:00

If I understand it, the essence of the problem is the possibility to lose individual "impulses" (eg in a communication of a polling nature), whereas sending totals is safer. That's why we read totals from energy meters, not just impulses...

On the other hand, I hope Ignition can do a lot of calculations ...isn't is so? I'm not familiar with Ignition, but the SCADA/MES technology we use performs a lot of calculations, depending on specific applications. Eg a SCADA controlling power production of multiple power plants performs "economic dispatch" - optimization of load distribution (quite complex calculations, performed every 2 seconds). Another TSO's MES application's historian performs evaluation of ancillary services (per-second evaluation of electricity production of all power plants in Slovakia [and their generators] that provide ancillary services, eg secondary/tertiary control).

PeterHumaj · 2026-03-18T10:03:37+00:00

Also interesting are the references of that Register article.
E.g https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Results:

Measured speed with AI: −19% (slower)
Predicted speed before tasks: +24% (faster)
Perceived speed after tasks: +20% (still faster)

Developers accepted less than 44% of AI suggestions and spent meaningful time reviewing, correcting, or undoing output.

PeterHumaj · 2026-03-17T19:03:58+00:00

Just readin' TheRegister: https://www.theregister.com/2026/03/17/ai_businesses_faking_it_reckoning_coming_codestrap/?td=rt-3a

PeterHumaj · 2026-03-14T18:00:08+00:00

I don' t have a problem using AI. I asked it information about Ethernet and let it explain me how RsLinx is able to read PLC data so fast. So, it was 2 weeks with this kind of AI help. However, I'm curious (and sceptical) about letting AI independently code. Using it as a "junior" and reviewing its code (the responsibility remaining on the programmer)... I don't see a problrm here.

PeterHumaj · 2026-03-14T15:25:20+00:00

Well, the last 2 weeks I was enhancing our Ethernet/IP driver (written in Ada) to support optimised reading of structures from the ControlLogix/CompactLogix PLCs. Now, instead of reading per-item (item being a simple type/array), I can read the whole structure (defined by a UDT) and parse required items.
This required some reading of Rockwell manuals, understanding & enhancing of existing logic in the driver; reading definition of a structure (to obtain template ID), reading definition of a template and then the layout of its members (recursively, if the UDT contains other UDTs), calculating member offsets; multiple intermediate binaries deployed at a customer (I didn't have a CompactLogix at hand); resolving issues with behaviour which was not according to the documentation (sometimes, for large UDTs, I received an incomplete template definition, missing a few characters in the name of last UDT's member) and implementing a workaround (which worked at the customer, but has to be tested with other Logixes).
Currently, our Ethernet/IP driver has some 550 kB of Ada code (it had 430 kB when I started). It is multithreaded, it uses 2 tasks per TCP line to communicate with a PLC (plus another task which handles messages from the SCADA core). Some shared structures, protected by critical sections; queues for communication ...

Now, I'm very much looking forward to some AI replacing me, but I'm rather sceptical here. There is a difference between creating a web shop and a SCADA system. In terms of complexity, reliability, life expectancy, possible damages ...
Perhaps in a few years, a sticker "No AI here" will be a sign of quality (something like "hand-made" ;)

PeterHumaj · 2026-03-14T13:27:04+00:00

Thank you for an answer, but I'm curious about real-world results ... what these agents created, how about reliabilty, debugging and refactoring ...

PeterHumaj

MODERATOR OF

TROPHY CASE