# Long range data over radio



## mburtis (Sep 1, 2018)

Just wanting to discuss what everyone has seen and used for long range data over radio applications. We have 2 plants that are 13 miles apart and our intake is about another mile past that. Currently the intake shoots to town while the two plants have a pretty unreliable t1 line between them. Supposedly the city hall IT guy is coming up with new radios and going to change it all up. I don't know much about radio stuff but figure I better start learning so I can fix it when it doesn't work. We also have lots of tanks and other remote sites that are on radio. Right now we have a mixed bag of fun as far as equipment, primarily old Motorola and newer phoenix contact stuff. So anyone want to share there wisdom on radio products or technologies?


----------



## BeardedBrother (Jan 10, 2021)

mburtis said:


> Just wanting to discuss what everyone has seen and used for long range data over radio applications. We have 2 plants that are 13 miles apart and our intake is about another mile past that. Currently the intake shoots to town while the two plants have a pretty unreliable t1 line between them. Supposedly the city hall IT guy is coming up with new radios and going to change it all up. I don't know much about radio stuff but figure I better start learning so I can fix it when it doesn't work. We also have lots of tanks and other remote sites that are on radio. Right now we have a mixed bag of fun as far as equipment, primarily old Motorola and newer phoenix contact stuff. So anyone want to share there wisdom on radio products or technologies?


When you say radios are you talking like ip based technologies or some other type of radio based technologies. If it's ip based I have a bit of experience.


----------



## mburtis (Sep 1, 2018)

Feel free to share any information. I'm open to learning all areas. I don't think we are using IP but that might be what the IT guy is looking at.


----------



## Flyingsod (Jul 11, 2013)

You’re looking for a data stream as in computer networking and not radio linked inputs and outputs?

Either way do you have line of sight capability with towers? That will make a huge difference. 


Sent from my iPhone using Tapatalk


----------



## gpop (May 14, 2018)

900 mhz is a no go at that distance which means you would need a radio study and to buy a frequency around 200mhz and a 85' tower (may be even higher). If you are in a hilly area it might not work which is why you would require a study.

If you have cell signal you could use that for the 13 miles (small monthly charge) and maybe cell or radio for the intake. Fiber would be the best option but also probably the most expensive.


----------



## drsparky (Nov 13, 2008)

Radio links can be cheap or expensive depending on location. There is no one size fits all. Licencing is another factor, you just cant hop on a frequency and start using it. As far as working on radios that can take a lot of training and very expensive test equipment.


----------



## SWDweller (Dec 9, 2020)

An unreliable T1 that is a pretty big pipe for moving data. I would want to exhaust the reason for unreliability before I jumped into a new tech. Maybe it is time to investigate the options of upgrading the T1 line. 

I have never been a fan of wireless transmission of data, the security go out the window and the information can become public. Which could be the end of days.

When I worked at the open pit mine we had radios that would cover the 10 mile distance to town from the mine. Monthly we were all schooled/reminded on what not to say on the air waves. There were people in town that for what ever reason wanted the mine closed and to hell with the economic benefits the mine provided.
We had a death in one of the pits during active operations. Someone put it over the air and with in an hour every news person in southern New Mexico was camped outside the gate clamoring for information. A lot of the anti mine people were there as well. The mine got the state police to allow us to exit the mine and get to our vehicles in the parking lot. The information in the beginning was he was electrocuted. Which would have been on my watch as I was the head electrician for the shift. I verified that no electrical device in the circuits in is area were tripped. Our ground faults would have caught something like this. Meeting after meeting with all of the people claiming authority. I had to sit there and repeat, what I knew and what had happened.


----------



## just the cowboy (Sep 4, 2013)

I just signed a contract for 1.75 mil to redo my radio network and get off of my T1's, it is not cheap.
If you start passing a lot of data cell is not cheap, and don't fit into our cybersecurity plans.
We are doing a high speed licensed microwave bidirectional ring with redundant paths to each plant (5 plants). And we are going with 220 MHz tail ends to the remote sites (82 sites). In this day and age you should be looking into a WATER network completely separate from the IT network, do not let them talk you into shared transport.
My guess is IT is sharing your T1 line and bogging it down. My T1's are 99.9% up with no issues, my only problem is speed. As @SWDweller said a upgrade of the T1's would fit you best for now.
We have had very good luck with the Xetawave brand of radios. We are using their 900MHz, 450 MHz, Spread spectrum and I/O radios and are going to end up with all 220MHz due to speed and flexibility of coverage. They are available with built in routing and protocol convertors, what this does it lets me program remote sites the same with out other routing hardware or that IP in and Modbus out.


----------



## emtnut (Mar 1, 2015)

A properly engineered Radio system can be quite reliable, but adds a lot of complexity to troubleshooting.

I agree that looking into the issue with the T1 would be much better.
If it's bandwidth, then upgrade to T3, or install more T1 lines (they can tie multiple T1's together)

At the water plant I worked at, voice lines, DVACS lines, and T1's were often an issue. We'd have to call out Bell Canada (tel company) to check the lines. Worthwhile to at least get a butt-set to listen to the line, you can often hear the ground hum, or notice it's a low level.
With intermittent problems, after Bell was out and couldn't fix it, we'd megger it 😁
It usually fixed it, sometimes it broke it, but then Bell would come and actually fix it.

Another thing we had to push for was 'sealing current' . The small signal on those lines allow crud to build up on the contacts, and eventually they go open circuit. The sealing current fixed that.


----------



## mburtis (Sep 1, 2018)

The T1 line I believe is dedicated to between the plants, shouldn't be any of our IT stuff on it. We don't have an issue with speed so much as it just dropping out. It's a pretty old line and has been patched together so many times and the service for it isn't great. It's running 13 miles out of town along what was once a very rural highway, building houses like crazy out there now. Chances of getting anything upgraded is probably pretty slim. 

Our IT guy is coming up with all kinds of stuff that scares the hell out of me. His long range plan is to apparently bounce clear across town and tie in with the rest of the city radio network. His short term plan is something about a VPN to push our data through the existing internet through the city hall firewall. I figure I got two options, try to learn enough to make an intelligent argument against it or wash my hands of it and stick to motors and PLCs so I don't get blamed when it messes up.


----------



## emtnut (Mar 1, 2015)

mburtis said:


> The T1 line I believe is dedicated to between the plants, shouldn't be any of our IT stuff on it. We don't have an issue with speed so much as it just dropping out. It's a pretty old line and has been patched together so many times and the service for it isn't great. It's running 13 miles out of town along what was once a very rural highway, building houses like crazy out there now. Chances of getting anything upgraded is probably pretty slim.


If it's dropping out, then it's up to someone to go after the tel. company and make them fix their **** !
You're paying for it.
Not any different than calling poco for a connection that drops out. They HAVE to fix it.


----------



## mburtis (Sep 1, 2018)

emtnut said:


> If it's dropping out, then it's up to someone to go after the tel. company and make them fix their **** !
> You're paying for it.
> Not any different than calling poco for a connection that drops out. They HAVE to fix it.


That's my thoughts but we are in wyoming and apparently there are like 2 techs that cover half the state.


----------



## drsparky (Nov 13, 2008)

Everything we do over the air is encrypted and the routers are secure, can't have the Russian hackers shut you down.


----------



## just the cowboy (Sep 4, 2013)

mburtis said:


> That's my thoughts but we are in wyoming and apparently there are like 2 techs that cover half the state.


My T1 lines are not cheap to rent, it is one reason we are getting away from them. BUT you are paying for them as @emtnut has said. Pull utility rank critical infrastructure and go up the line. 



mburtis said:


> Our IT guy is coming up with all kinds of stuff that scares the hell out of me. His long range plan is to apparently bounce clear across town and tie in with the rest of the city radio network. His short term plan is something about a VPN to push our data through the existing internet through the city hall firewall. I figure I got two options, try to learn enough to make an intelligent argument against it or wash my hands of it and stick to motors and PLCs so I don't get blamed when it messes up.


This is what I was warning about, it will be shared transport and open the door for ransomware. Your IT will say "no we have firewalls", but so does everyone else that got hacked.
Since you are a one man show basically I would go with your ending statement of stick to the control end, but let it be known in writing that you don't agree with it.


----------



## emtnut (Mar 1, 2015)

mburtis said:


> That's my thoughts but we are in wyoming and apparently there are like 2 techs that cover half the state.


As Cowboy said, escalate it !

Then one of those techs will be looking at your T1.

I remember being on a conference call with my boss and service provider. They gave a similar excuse as yours (we only have xx techs)
He responded with "Hmmm, you're problems are interesting, however they're not mine, fix my equipment ! "
Very hard for me not to LOL 😆


----------



## mburtis (Sep 1, 2018)

I've actually been emailing the IT guy discussing this stuff all day. One of those keep your enemies close type of things. Doesn't help he is tight with the big boss down at city hall. At the end of the day I don't care how data gets from one plant to the next as long as it's reliable and I know generally how it works. We have had 3 security problems in the last year so whatever he does can't be much worse than what we have.


----------



## MikeFL (Apr 16, 2016)

drsparky said:


> Everything we do over the air is encrypted and the routers are secure, can't have the Russian hackers shut you down.


This is a great topic to seriously consider.

If you're on the city network, any city employee can open a malware e-mail and start clicking links and the bad guys can hold your utility ransom.
Get your own system and you won't be subject to that risk.

ETA: I see someone already mentioned it. I'll leave my comment here so OP has more feedback when consulting his peers across the nation.


----------



## mburtis (Sep 1, 2018)

So I asked similar questions today about how he was going to ensure our stuff was isolated from the general IT traffic. Being an IT guy of course his response was that everything will be isolated virtually via the firewall and switch settings and the plants will basically be on there own virtual LAN (whatever the hell that is) and he will basically create a virtual airgap by not allowing our stuff to talk to the internet external port. Difference between electricians and IT I guess, I'll trust hardware solutions over software any day while IT prefers software. Software works great until staff changes and the bill doesn't get paid or the settings get screwed with, or it goes out of date. 

He is always "so busy" so we will see if he ever gets around to doing anything or if we hang out in limbo forever. On that note does anybody have any suggestions for monitoring a T1 line. Something to monitor and log when it drops out so I can prove there is issues. Our problems are intermittent normally so by the time they get there it's working and they just shrug and get back in their truck.


----------



## emtnut (Mar 1, 2015)

mburtis said:


> He is always "so busy" so we will see if he ever gets around to doing anything or if we hang out in limbo forever. On that note does anybody have any suggestions for monitoring a T1 line. Something to monitor and log when it drops out so I can prove there is issues. Our problems are intermittent normally so by the time they get there it's working and they just shrug and get back in their truck.


No one has confirmed that the T1 line is the problem, and IT is just changing equip ??
Doesn't make sense, but the parts cannon solution seems to often be prevalent 

Depending on the T1 bridge you use (or NTU as the software guys call it), most have diagnostics in them that will show you lost connections/packet loss, and any error codes. Some even have a built in BERT tester.

Someone has to figure out why it's going down before getting the cannon out !


----------



## mburtis (Sep 1, 2018)

Blasphemy you say.... pretty sure it's in the operating manual somewhere to replace all known good parts with untested new tech before looking for the actual problem. It comes right after the "anyone caught thinking for themselves will be severely reprimanded" statement ....


----------



## Jlarson (Jun 28, 2009)

Best thing you can do to start is cut the IT ****s out of it and get a consulting firm that has actual SCADA and radio telemetry experience. 

IT should be worried about if the front desk secretaries computer can google stuff, not process stuff.


----------



## splatz (May 23, 2015)

mburtis said:


> That's my thoughts but we are in wyoming and apparently there are like 2 techs that cover half the state.


People act like a T1 is a great thing but IME a T1 is probably the most unreliable circuit the telco sells. 

Problems with the electronics are common but once they replace them - usually just replacing modular cards - it will be a long time before you have more trouble. Problems with the cable infrastructure are common too, and harder to get them to resolve. Complaining to the PUC is the way to go but it's not a slam dunk. 

If they only have two techs, you and the PUC don't care, they still have their obligations as a public utility. Understaffing is not an excuse for not meeting their legal service commitments. 

If they have ratty old cable from one or both plants to the telco company office it might take an act of Congress to get them to fix it. You might be able to get them to give you fiber at the T1 price as a settlement, if they have the fiber.


----------



## splatz (May 23, 2015)

mburtis said:


> I figure I got two options, try to learn enough to make an intelligent argument against it or wash my hands of it and stick to motors and PLCs so I don't get blamed when it messes up.


I would push for an outside consultant. Realistically it might be pretty hard for you to come up to speed to argue with the IT guy. From the sound of your post, he wants to try some new things and apply some of the regular corporate IT network practices to your industrial network, annexing your network to his. This is exactly how things go horribly wrong. 

For that consultant - I don't think you want a vendor selling a product, you want someone that specializes in industrial network security. Someone that can point to the many cases where going all-IP and extending corporate networks to industrial environments has led to some very public embarrassing and alarming breaches. 

My opinion is that although corporate networks can be very secure they are a totally different animal from industrial networks. 

* Breaches to corporate networks can leak sensitive information and destroy data but if your backups are in order the loss of data should be limited. Industrial networks have the same risks but also control things that can do tremendous property damage and kill or injure people. The stakes are astronomically higher. 

* Industrial networks may incorporate devices and systems that can't be secured to the degree required by corporate practices. These devices and systems don't have a $1000 price tag and a five year life cycle, replacing them with something that complies with security practices can cost thousands or millions. Auditing the industrial network to just get a handle on this situation is a huge can of worms. 

* All of the above for security also applies to reliability. The systems that the industrial network controls demand higher reliability. (You can't turn off the sewers overnight on Saturday for maintenance.) 

* This completely changes the design and deployment. Just as it would be hard for you to come up to speed on the network side, it's not realistic to expect the IT guy to come up to speed and understand industrial automation. You're basically both DIYers with this and you ought to get a pro involved.


----------



## gpop (May 14, 2018)

There is nothing wrong with using IT hardware as a point to point transfer network. Hardwired, port locked, mac address works fine. 

The trouble starts when someone decides that they would like remote access to the plc system which is generally running on older versions of windows that have not had security updates due to conflicts with the software we use. If you never allow a bridge to be built between the networks the risk is very low.
You should already have off-line back-ups of all plc and hmi programs and a written sop of when and how these back-ups will be maintained.


----------



## splatz (May 23, 2015)

gpop said:


> There is nothing wrong with using IT hardware as a point to point transfer network. Hardwired, port locked, mac address works fine.


MAC address? I think it can't hurt and it helps accidental connection mistakes but it is too easy to spoof to do much for security.


----------



## samgregger (Jan 23, 2013)

Maybe Starlink?


----------



## mburtis (Sep 1, 2018)

Thanks for all the input on this. This thread kinda went a completely different direction than intended but it's been a good discussion. I'm glad to know we are completely screwed. 

At this point his planned involvement is limited just to getting the two plants to talk to each other. So far he has not made a move for the in plant networks for i/o etc, nor will I let him have those. I'm already moving forward with adding stand alone HMIs around the plant to improve the local control and log important data in another place outside of the computer. At least that way if SCADA does go down you just pull the local copy of data and continue on with being a fat, dumb, and happy city worker. 

If I have to I'll hook the computer up to the control network via old school serial... hack that.


----------



## just the cowboy (Sep 4, 2013)

mburtis said:


> On that note does anybody have any suggestions for monitoring a T1 line. Something to monitor and log when it drops out so I can prove there is issues. Our problems are intermittent normally so by the time they get there it's working and they just shrug and get back in their truck.


This is a great learning opportunity for you. You said you are getting a PLC and HMI, make it 2 PLC's.
You can do this without going into your plant PLC code this way. Build what is called a heartbeat in each PLC. You pass a message from one PLC to the other reading the other ones time clock value. Compare last time with new time, if different T1 is working. If the same, start a timer and log timer value to the HMI. The C-more has a built-in datalogger that can go to a USB stick then a spreadsheet.
You can build and test this on the bench by unplugging 1 PLC then after you proofed it just put one PLC at each plant.

Good luck
Any questions ask

Cowboy


----------



## mburtis (Sep 1, 2018)

just the cowboy said:


> This is a great learning opportunity for you. You said you are getting a PLC and HMI, make it 2 PLC's.
> You can do this without going into your plant PLC code this way. Build what is called a heartbeat in each PLC. You pass a message from one PLC to the other reading the other ones time clock value. Compare last time with new time, if different T1 is working. If the same, start a timer and log timer value to the HMI. The C-more has a built-in datalogger that can go to a USB stick then a spreadsheet.
> You can build and test this on the bench by unplugging 1 PLC then after you proofed it just put one PLC at each plant.
> 
> ...



Thank you for this. I figured something along this line. Most of this programming should already exist in our main processors. I know each processor monitors the other one to make sure it's still running and there are comm alarms and watchdog alarms. I haven't dug that deep into the programming yet. I kinda like the idea of setting up something independent though just to keep it seperate and as you said a great learning experience.


----------



## just the cowboy (Sep 4, 2013)

mburtis said:


> Thank you for this. I figured something along this line. Most of this programming should already exist in our main processors. I know each processor monitors the other one to make sure it's still running and there are comm alarms and watchdog alarms. I haven't dug that deep into the programming yet. I kinda like the idea of setting up something independent though just to keep it seperate and as you said a great learning experience.


What brand PLC is in the plant? PM me if you don't want to post it.


----------



## mburtis (Sep 1, 2018)

They are all controllogix. We just updated them about 3 years ago which is where most of our problems started. We were supposed to get an upgrade that included rebuilding SCADA. The bean counters got a hold of it and what we ended up with was new pics and a program from 93 migrated to new hardware.


----------



## DragnUp (Jun 18, 2021)

I had a network of 45 serial point-to-point Freewave radios sending data to a single master on a rohn tower that was over 100' tall. Two farthest nodes were about five or six miles out from the master, in opposite directions. Flow meters, tank gauges, temperature data, sales meters, etc....

I was using the Freewave FGR and FGR2 radios. Their computer software for configuring the radios and monitoring network efficiency was straightforward and easy to use. Their techs were quick to pick up the phone and gave me practical, straightforward info I could use to get comms backs, but also encouraged me to educate myself in the physics behind it all. They gave both bare-bones quick fix solutions, but also could nerd waay out on radio propagation theory.

Average age of these things was over 10 years old, for the FGRs, and they continued to just work. These radios are little 2.5" x 6" electronic boards that are built tough. Whenever I was troubleshooting comms, i checked the radio _last_ because the Freewave was rarely the problem.

Two things: 
1) Backup and double-backup all your existing radio settings, network settings, instrument settings, etc....

2) If you're looking to buy some radios for industrial use, the radio company may come out and do a site survey for free. They'll analyze your needs, the topography and a dozen other variables (the presence of pine trees for example) and make a recommendation.


----------

