SCADA and telemetry
The RTU Kept Disconnecting. The RTU Was Fine.
By Arch. Dany Dandachi, ALMAFOR, Abu Dhabi, United Arab Emirates
Published
The most expensive faults on an RTU integration are the ones that are not in the RTU. We spent real hours on one: a remote terminal unit that kept dropping its connection to the supervisory system, over and over, with nothing wrong in its configuration and nothing wrong in its logs. The RTU was fine. The problem was where it was plugged in.
During integration testing the unit was connected through an office switch, on the office network, which sat on a different network domain from the one the telemetry was designed for. Everything looked connected. The link light was on, pings worked when we tried them, and the session to the master would establish, hold for a while, and drop. Again and again. The moment the RTU went onto the network it was actually meant to live on, the disconnections stopped and never came back.
That is the whole story, and it is embarrassingly simple in hindsight, which is exactly why it is worth writing down. An office network and a telemetry network can both be Ethernet, both hand out link lights, and still be different worlds. Below is the checklist we now run before anyone opens the RTU configuration to look for a fault that is not there.
Why an office network quietly breaks telemetry sessions
None of these announce themselves. Any of them can produce the same symptom: a session that connects, survives for a while, and drops.
- Different subnet or broadcast domain. If the RTU and its master are not where the addressing plan expects them, traffic may be crossing a router or gateway nobody designed for. Sessions that establish through an unplanned path inherit that path's timeouts and failures.
- IT-managed switch features. Office switches are configured for offices: port security that shuts a port when it sees an unfamiliar device, storm control, spanning tree reconvergence, energy-efficient Ethernet putting quiet links to sleep. All good policies for laptops and printers. All capable of cutting off a device that talks rarely and expects to be listened to permanently.
- DHCP against a static plan. Telemetry equipment lives on static addresses. An office network with DHCP can hand the RTU's planned address to something else, and a duplicate IP produces exactly this pattern: intermittent connection, intermittent loss, nothing conclusive in any single log.
- Idle timeouts in between. Firewalls and routers on IT networks silently drop TCP sessions that stay quiet longer than their idle timer. A telemetry session with slow report rates looks idle to an office firewall, and the RTU only finds out its session is dead the next time it tries to speak.
The checklist, in the order that finds it fastest
- Ask where the cable actually goes. Not where the drawing says. Follow it. If any part of the path touches the office LAN or an IT-managed switch, move the test rig onto the dedicated network or a plain standalone switch before doing anything else.
- Confirm subnet, mask and gateway against the addressing plan on both the RTU and the master, and confirm they are in the same domain the design intended.
- Check for an address conflict. Unplug the RTU and ping its address. If something answers, you have found your intermittent fault.
- Watch the session, not the ping. Continuous ping can pass while the protocol session fails. Watch the actual telemetry session establish and hold on a protocol test tool for longer than the longest quiet period the site will ever have.
- Only then open the RTU configuration. If the network path is clean and dedicated and the session still drops, now it is worth looking at keepalives, timeouts and the device itself.
The rule we took from it
Telemetry equipment gets commissioned on the network it will live on, or on an isolated switch that imitates it, and never on the office LAN because the socket was closer. The office network belongs to the IT team and is tuned for people. The telemetry network belongs to the plant and is tuned for devices that must never be forgotten about. Plugging one into the other produces faults that read like equipment failures and are nothing of the kind.
It is the same lesson our timestamp investigation on the BMXNOR0200H taught from a different direction: on RTU projects, the fault that presents at the device is very often a fault in the arrangement around the device. What an RTU actually does, and why its connection matters more than its uptime, is covered in our plain-language RTU explainer, and how these projects are scoped end to end is in the RTU and telemetry guide.
Common questions
Questions we are asked about this
Why does my RTU keep disconnecting from the SCADA?
Before touching the RTU configuration, check the network path. A different subnet or broadcast domain than the design intended, an IT-managed office switch in the path, a DHCP address conflict with the RTU's static address, or a firewall idle timeout between the RTU and the master all produce the same symptom: a session that establishes, holds for a while, and drops. In our own case the entire fault was an office switch on a different network domain; on the correct network the RTU never dropped again.
Can I test an RTU through the office network temporarily?
It is the single most tempting shortcut on an integration bench and it costs more time than it saves. Office switches carry port security, storm control, spanning tree and power-saving behaviour tuned for laptops, and office networks carry DHCP and firewalls tuned for people. Commission telemetry on the network it will live on, or on a plain isolated switch that imitates it.
Ping works but the telemetry session still drops. How?
Ping only proves the address is reachable at that instant. The telemetry session is a long-lived connection with its own timeouts, and a firewall or router in an unplanned path can silently discard it as idle while pings continue to pass. Watch the actual session on a protocol test tool for longer than the longest quiet period the site will see.
How do I check for a duplicate IP address?
Disconnect the RTU and ping its address. If anything replies, another device holds the address, and every symptom you have seen, including the intermittency, is explained. This happens easily where telemetry equipment with static addresses is plugged into a network that also runs DHCP.
Does Almafor commission and troubleshoot telemetry networks?
Yes. We deliver RTU and SCADA integrations across Abu Dhabi and the UAE, including the network between the station and the control room, and much of our troubleshooting work is on exactly this boundary: equipment that is healthy, on a network arrangement that is not.
We want to hear from you
Tell us about the system you need built, upgraded or kept running.