When Infrastructure Gets Hacked

[@PracticalEngineeringChannel]

🌉I have Practical Engineering hats in the store: store.practical.engineering 📰Compare news coverage from diverse sources around the world on a transparent platform driven by data. Try Ground News today and get 40% off your subscription: ground.news/practicalengineering

[@midbc1midbc199]

The only person I know with the name Grady is also a civil engineer.......must be name related lol

[@edstercw]

Watch "Dan Tentler - Defcon 2015 - Comedy Inception Panel" for 100 industrial devices he found exposed on the internet. Petrifying.

[@rogerscottcathey]

Why is there is a platinum ball hanging by a chain in a water tower.

[@Enstrayed]

That Ad transition was so smooth it’s criminal

[@Marin3r101]

Hey Grady, the wording you used to describe zero-day seems a little odd. Defining a zero-day: a zero-day vulnerability is a security flaw in software or hardware that is unknown to the vendor and has no patch available. Cybercriminals can exploit these vulnerabilities to steal data, compromise systems, or launch other cyberattacks before the vendor is aware of the issue. This is an exploit previously unknown to anyone, but it was found by a bad-actor. Sometimes its still not even known by consulting groups either. Once they get caught (if ever) they spill the beans. Digital Forensic investigations require systems to be brought offline so they can inspect the systems that were compromised. Most of the time, especially with a water system like this video, is just not possible. So they only noticed it after and then started to investigate further. If these bad-actors compromised the system and then laid low leaving access open to it in the future they would have had a far more nefarious goal in mind. Clearly not military like you suggested. I would like to see where you got that bit of information from.

[@Joeyzoom]

I've been a cybersecurity consultant for the past 4 years and I cannot stress enough on the importance of managing privileged access to critical infrastructure. I've worked with several organizations big and small that have limited protections in place to prevent unfettered access to their internal resources - it's scary, even. Even air-gapped environments have their weaknesses and I'm glad you alluded to it. An organization's security posture is only as strong as the least prepared end-user. This is a brilliant demonstration, thank you Grady! 🎉🎉

[@Soapyeiew99]

Hey that’s what I do! Access audits are a necessity I hadn’t considered before this job

[@FlesHBoX]

I've been absolutely astounded by how many companies fall flat in this area. Hell, even data protection is often overlooked. The number of clients I've had to tell "I'm sorry, without proper backups, that data is just gone" and they're out tons of money because someone in the chain decided that some software license wasn't worth the cost because they had never had need to use it...

[@breakupgoogle]

whole world is a house of cards.

[@coutxxStatix]

@@breakupgoogle It doesn't have to be, we have lots of systems to back up and protect cyber infrastructure but it's a hard thing for a lot of people to see so they don't spend the money on it, despite the fact that the services can cost $10-100,000 and ransoms cost $1-10 million

[@goosenotmaverick1156]

​@FlesHBoX I've not seen that exact thing but I've seen small mistakes by low-mid level folks cost companies thousands and thousands per individual incident and still not correct the access levels that allowed it to happen. Even simple stuff like that gets ignored. Much less real security issues.

[@Welgeldiguniekalias]

For anything related to infrastructure, the question should be: "Can it run without being connected to the internet?" If the answer is yes, it should not be connected to the internet. You can still automate it, you can even build your own network, just make sure you don't put in any unnecessary internet connectivity.

[@dank_69_420_memes]

Airgap the world

[@TealJosh]

Say hello to Industry 4.0

[@Mineral4r7s]

this. thats just logical if u have the slightest clue about how computers work. u can run seperate networks without endangering one or the other

[@Marcus-p5i5s]

NONE of it has to be on the public internet. Also NO controls for the critical infrastructure should be connected to any computers that have a route to the internet and machines with any wireless infrastructure. And NO MS O/S's at all. Only hardened nix machines.

[@KellySmith-kw2cl]

This is wrong and would be extremely expensive. Your idea would require someone sitting at a control panel all day monitoring the equipment. The reason for automation is 99% about monitoring, not about control. But you need control in order to act immediately on monitored output. What they need to do is assume that they will be attacked and harden the connections through VPNs, firewalls, encryption, and 2-factor authentication.

[@ecospider5]

A guest to our company tried putting their own wifi router on our network so they could access it from their hotel across the street. They were actually confused when we took it down. They didn’t see the harm. Security is a very human problem just as much as a technical one.

[@ghost307]

We run a full review every year to verify that our SCADA is still air gapped. Each year we find new breaches that have gotten installed. On one new project I told the manufacturer's rep that he was NOT allowed to connect his equipment to the internet. He pointed out the value of his factory being able to monitor the equipment 24/7. He thought I was joking when I told him to call me any time of the day of night that he wanted to come onsite and check his equipment. He never did get his internet connection.

[@57thorns]

The only ones I head of that actually lock down their physical cat 5 outles are defense industry and the university. You can still attack that if you spoof the Mac address, but it is a little bit harder than just getting access and plugging in your own device.

[@moos5221]

that's hilarious...until it isn't.

[@pileofstuff]

I found a WIFI device on our office network that one of the secretaties had put there so she could use WIFI on her phone in the lunchroom...

[@saiv46]

@@57thorns Universities don't know how to properly secure their network.

[@bulwulffcristole3235]

I'm a licensed ethical hacker and network engineer of over 20 years and have several water districts as my clients in Texas and I can say that most do not have the cybersecurity support or network security in place to properly secure their infrastructure. A lot of it comes down to IT departments not knowing what they don't know about cybersecurity and thinking they do, which in turn means they reject specialists like myself who provide oversight and ensure that high level of security. Because this attack wasn't done through social engineering, that means the network security wasn't anywhere near where it should have been. The only threat a network should have is through social engineering because if it's a technical weakness then someone isn't doing their job.

[@jaymacpherson8167]

I agree as that was my experience working as a public water system regulator for almost 14 years. Yours is the most salient point relative to other comments herein, as air-gaps can still be penetrated if an unsecured thumb drive with a virus is connected to the air-gapped network.

[@reahreic7698]

@@jaymacpherson8167 That's why you fill all the USB ports with superglue. It's only going to get worse as the market is now flooded with devices that use USB-PD to charge.

[@goosenotmaverick1156]

I find your field fascfascinating. I listen to a podcast where they talk to folks like you and others in the industry. Sometimes it's hackers that have even been convicted of stuff. It really gave me a cool insight into your industry. It's way over my head but it's fascinating. Simple USB drives and devices seem to have begun to pose a much higher risk over the last handful of years or so.

[@goosenotmaverick1156]

Oh the podcast is called Darknet Diaries, I find it interesting, maybe someone else will too.

[@Mike80528]

Cybersecurity Architect here - industrial control systems are a serious issue. It's scary how outdated (and insecure as a result) many of these systems are.

[@TravisNewton1]

As someone who works in IT: it's absolutely bonkers that my home network has better security than a lot of critical infrastructure. There's too much stuff on the Internet that shouldn't be on the Internet. I can scream this into the void, but C-levels don't care until it's too late.

[@kevinortega6688]

This is currently changing. I work as a SCADA engineer in the water industry and there has been a huge emphasis in cyber security for critical infrastructure. It is not obviously up to par as it needs to be, but examples like Mulshoe and the colonial pipeline has really shaken the industry to take this stuff seriously.

[@kenneyshepard4511]

Seriously it's like they haven't heard of 2FA before.

[@sathyanarayanan4171]

UFW deny

[@haxalicious]

Yep. And better than corporate networks. They all use spyware that the C-suites got duped into using to spy on everything done on employee devices and try and stop malware, instead of properly using VLANs to stop said malware from spreading far once it inevitably evades detection.

[@3nertia]

If it costs more money to fix than it makes then capitalism doesn't care heh

[@MyBrandonTV]

I work as an operator in a small west Texas town and not for nothing, we have opted not to connect our SCADA system to the internet specifically for these reasons. Great video!

[@williamchamberlain2263]

Good move.

[@danl6634]

Next step is social engineering training for anybody with access, including everyone up the chain. I've seen some very confidently wrong c-suite execs who would plug in that usb from the parking lot in a minute.

[@Morridin19]

Additionally should setup least functionality and least access controls. Having seen first hand the affects of someone breaking an air-gaped system unintentionally and the resulting incident when someone found out the system had little to no controls in place to limit access.

[@JSMCPN]

I'm a licensed operator of a rural water system (volunteer) as well as an IT systems engineer. My PLC isn't even capable of being networked... short of disassembling it, installing a GSM modem, and adding function blocks. Changes must be made via physical access and RS232 serial, from a laptop that contains the original decrypted program layout.

[@DrLoverLover]

You cavemen! that will lead to recession and economical chaos

[@ugh212]

In my old job, I would inspect water systems for vulnerabilities. The best SCADA system was so old that there was no outside access. On a non-windows system.

[@RichardsNickname]

lmao thats why i loved my old windows xp i used it for so long that i ended up being forced to use windows seven. never even had a antivirus, just a computer too old to get sick

[@Cyanfox3006]

​@@RichardsNicknameSorry, this malware isn't compatible with your operating system

[@TheDankFarmer]

@Cyanfox3006 Please run this virus in Compatibility Mode

[@Senthiuz]

​@@RichardsNicknamekzbin.info/www/bejne/bKa2h4l5orR7a9k

[@diggernash1]

My former utility had our analog tone system in place until the early 2000s...but then the telecom went digital and encouraged their old school techs to retire...and our system became entirely unreliable. Now, I believe that the US DHS will no longer allow truly secure communications systems and software to exist; even in the United States.

[@capn_shawn]

As an Electrical Engineer, SCADA/HMI/PLC programmer, Network Admin and electrical system designer, I can assure you... most issues are caused by employees breaking the air gap either intentionally, by sheer stupidity or by over-zealous network admins that can't stand the presence of a workstation on "their site" that isn't under their control.

[@_ata_3]

How an overzealous admin is to blame?

[@jfbeam]

In every system I've seen, things are ALWAYS setup with remote access. In over 20 years, I've not been in a Class A office that didn't have complete remote access to everything... fire alarm, security doors, elevators, HVAC... the f'ing clock in the lobby... (The scary part... a _nuclear power station_ can be remotely controlled. Protected only by an RSA SecureID token.)

[@Buddydaneable]

Or remote maintenance vendors / third party

[@capn_shawn]

I didn’t say secure connections don’t exist. I’m saying the issues exist because of people. Just because some aspects or overhead systems can be remotely controlled, I assure everyone there are still air-gapped, physically separated or unbridged networks in most facilities… even nuke plants. You think you are going to reconfigure the motor drives or vibration monitor calibrations on the reactor jet pumps from your home office, you are mistaken.

[@Mordecrox]

​@@_ata_3because they want it to be under THEIR control - read, they can't stand the idea of an asset they don't have full time physical and remote access, so they take measures to "take ownership" of said assets. Be it have it somehow connected to the network with "I put this on Network isolation source trust me bro" or badgering management so they gain unrestricted physical access. Last serious place I saw we were told someone earned a writeup for just requesting access to The Vault, for they should know better.

[@bmenrigh]

I’m a professional cyber security engineer and this video is very well done. No fear mongering, no bullshit, no misinformation. Excellent job researching and presenting!

[@g0d5m15t4k3]

Amen!

[@MrNicoJac]

Do you have to be a civil engineer or a coding engineer to be able to do that? I find cyber security really interesting and am thinking about maybe doing a traineeship in it after graduating university - but my strengths are definitely more in critical thinking, planning diagrams of systems/layers (not actual coding), and human interactions. So I'd love to explore whether it would be something for me, but I'd absolutely go insane if I just spent the entire day looking at lines of code that _somehow_ refuse to compile.... 🥲 Anyhow, thanks for youe hard work and keeping us all a tiny bit safer ^^

[@Mo1stKevlar]

I live in Calgary and have been watching your videos for years! Was neat to see my city in a video! The feeder main pipeline that failed has many more repairs needing to be done, something like 21 additional repairs, and we are currently under water restrictions until later this month so other repairs can be completed. I believe the pipe diameter ranges from 1.5m to 2m, is 10.4km long and carries 400 million litres of water. It dumped millions of litres of water during the initial break. It was certainly catastrophic. As a plumber, it affected work hours to a degree as we were not able to shut down and drain buildings for various repairs during the restrictions. Great video Grady!

[@GideonMesser]

If anybody is wondering whether the lead/lag pump configuration causes the lead pump to wear out sooner, you should know that lead/lag pump controls are typically programmed to swap which pump is the lead pump every so many hours of operation. Additionally, for critical pumps, there's often a spare on the shelf, or a third pump that is rotated so there's always a pump in the system. This is also a common configuration in HVAC systems.

[@zyeborm]

He did cover that in the sewer video too

[@joshyoung1440]

No, but I do kind of feel like a slacker now for not wondering that.

[@FlowMeterGuy]

I have been in the flow meter business for 40 years and personally watched the change over from manual operation to automation on running a facility. Today our customers all want to monitor and control everything on their cell phones… So security is an ever growing issue. I enjoyed your video.

[@chrispowell1224]

You’ve done my industry justice. Normally people chat a lot of nonsense, especially KZbinrs, but you’ve done a good job.

[@N0Xa880iUL]

Which is

[@GregorVDub]

Grady is the real deal, he is great.

[@mathieufavreaux8682]

I understand the subject at hand and I concur. This is amazingly explained, proper vocabulary is used and simply introduced, incredible. This is giving me so much confidence in how this channel covers other areas and fields I know nothing about. Well done.

[@kindlin]

Did you just find this channel? That's standard. Same with about another couple hundred YT'ers I follow. I swear, most of my sub list should be the default sublist for any new YT accounts, at least if you like science and math.

[@kylecossette5044]

As a calgarian, on your ad at the end, I certainly hope you cover the watermain break in a future episode. We're currently in restrictions again for the past month as they've found extensive degredation in the same line. Textbook case of what a lack of maintenance over time results in.

[@iamdegroot9025]

So true

[@DirtyDanMunicipalMan]

I work in municipal water. I think it’s both a blessing and a curse that the public doesn’t know what we do. On one hand, we’re doing our job well enough that the public doesn’t have to think about what we do, but on the other hand it allows so much misinformation and bad info to spread. Thanks for being an excellent source of factual and accurate information.

[@bborkzilla]

15 years ago I worked on securing DNP3 - which is used by a lot of SCADA devices. We received a lot of pushback from utilities who were more worried about locking themselves out of their systems than they were about intruders getting in.

[@rhouser1280]

I work as a control room operator in a power plant. Talking about looking at screens like a video game forgetting that it’s hooked up to actual equipment. That happens when you’re training people, it’s almost like a disconnect that happens. Everything looks so clean & nice on the screen, it’s easy to forget that the picture of the pump or turbine you’re operating is not a nice clean turbine, it’s a 70 year old machine covered in oil and dust.

[@KonradTheWizzard]

That's why I try to get my trainees into the production floor for a tour. I also encourage colleagues who haven't been there in years that they should join.

[@Jimmy_Jim_Jim_1234]

I'm a board operator at a Refinery. I always think of my outside guys first before i make any moves. Especially on natural draft furnaces.

[@Smokeisprogress]

Sounds like people that aren't qualified hiring and training people who aren't qualified.

[@rhouser1280]

@@Smokeisprogress couldn’t be further from the truth, guys work 20+ years sometimes just to get into the control room. You’re put in charge of hundreds of millions of dollars of equipment that unlocks stored energy & converts it to extremely high pressure superheated steam to drive turbine/generator units & power the grid with hundreds of Megawatts, sometimes Gigawatts of electricity. It’s a dangerous environment. Nothing unqualified about us. We’re humans, humans become complacent, it’s in our nature. I was just saying I can relate to what he said about it & I bring it up to anyone I train in here.

[@nitehawk86]

Have there been accidents where someone thought they were running a test/training system, but it actually ran on the live hardware? I think every computer company has managed to accidentally do this at one point or another.

[@FuncleChuck]

So the pipeline thing at 9:00 was completely self-inflicted. They screwed up their preparation, security, emergency response, restart, paid the terrorists, and blamed everyone else? Those execs should be in jail.

[@moos5221]

i remember images of idiots filling up plastic bags with gasoline at the gas stations 🤦‍♀

[@Tathanic]

@@moos5221 lmao

[@cmdraftbrn]

self-inflected stupidity is a c-suit trait.

[@EebstertheGreat]

@@moos5221 The weird thing is that the most famous video actually came from 2019 and had nothing to do with the pipeline. Some wackjob filling plastic bags with gasoline and putting them in the trunk for no clear reason.

[@joppedegroen6918]

I fully understand why you would shut down all your systems as soon as possible when you discover you are infected. The risk of critical hardware getting damaged (pumps running dry, pressure buildup, oil overflow) is just not worth it. Especially since there is always some slack in the supply-line which gives you time to deploy emergency solutions such as wheeled transport.

[@adamchurvis1]

7:05 It also DEFINITELY means the software engineers didn't even bother to include Sanity Checks on input values, which I'd bet my bottom dollar also means they didn't have sufficient test coverage of ANY layer of the HMI software, either.

[@bastiannenke9613]

Exactly what I wanted to comment right now. Can't believe those systems have no protection against accidental or intentional operator error. It's like using 2 contractors to change the direction of a motor and allowing both to be powered at the same time. Or allowing a traffic light to have the option "all green*. Absolutely unacceptable.

[@neosenshi]

The problem here: the HMI is often generated by engineers who are NOT specialists in software. Even 15+ years ago it was easy to build a custom HMI interface using tools provided by the manufacturer of the HMI. That means the process engineer ( who probably has little to no software experience) can easily set up the HMI to do what they want, and they probably won't be thinking of sanity checks.

[@bastiannenke9613]

@@neosenshi the PLC should have the checks, if you only have them on the HMI, the part between two machines or machine parts also won't be protected. When I had PLC stuff during my apprenticeship, the teacher always told how important it is to prevent wrong inputs to cause dangerous situations. We also had to consider breaking wires for safety features and other things. I absolutely agree with you that the operator always should be able to configure the HMI in a way, that makes sure they can do their job in the best way and that the safety of the machine should be their job. The machine should already be safe enough by default.

[@Juno_Eh]

As someone who live in Calgary, I would LOOOVE to see a video on the Bearspaw Supply main break! And the subsequent repairs and second round of (currently ongoing) repairs.

[@CalgaryBen]

I'm hoping that @thecityofcalgary and @Editorpurenews (Pure Technologies) will share some of the findings of the PipeDiver and how they correlate to what they uncovered during excavation (e.g. wire snaps in the prestressed concrete pipes). Through all these water restrictions, I was thinking it would be awesome for Grady to cover this topic in a video. He was bang on the money -- when the infrastructure is out of sight and out of mind, we (as Calgarians) didn't give it much though; little did we know we'd learn so much about our the source of our drinking water and how it's treated/delivered, and now, I'm eager to find out more!

[@MechTechMax]

Seconded. It's crazy because that pipe was only halfway through it's service life, which makes you wonder how much other critical infrastructure is close to failing without notice.

[@Darryl_Frost]

As a SCADA systems engineer, I worked on that hack in Queensland that you mentioned, he was an ex employee of the company I worked for. we had to patch the software and install new firmware in all the RTU's. SCADA systems engineering is a good career choice.

[@Mastadex]

Security is an afterthought and I'm so happy to see you did an episode on this. I work in cybersecurity doing audits of these types of systems (e.g. I'm the so called expert you mentioned). There are many, MANY attack vectors, ranging from your simple password reuse to supply chain attacks (such as the pagers that were used in Lebanon recently). This field will become huge in the next decade, and so will the complexity of attacks.

[@hightechredneck8587]

Absolutely Legendary video Grady. I will be sharing this video with my team. We work in a control center for utilities. Cyber security is a constant threat for us and the real world examples really drove the point home.

[@DasGanon]

There's a related fun security thing that happened in Wyoming. They ran an update on a Dam in the middle of nowhere (that wasn't supposed to get an update) and the update knocked off the management computer from all connections. Someone noticed quickly but it took 4 hours to drive out there.

[@MushookieMan]

So was it windows? yuck

[@sauercrowder]

When I was a network guy, we put a lot of stress on out-of-band management to prevent this kind of scenario. In my current job, I've noticed they don't bother with that, and as a result I have had the network guy call me and ask me what the equipment was doing. For something critical, you should have a way to get in when all else fails. It really isn't all that hard to do. Even for a management computer losing all network connectivity like this, there are solutions. You have some second device that is a console server connected by some other means (say, a serial connection that is impossible to misconfigure), and you give that device its own external network connection -- preferably dial-up or something else that doesn't share any dependencies with your usual network access. A four hour drive makes for a good story, but there are cases where a mistake like this could involve getting on a plane.

[@jeffspaulding9834]

@@MushookieMan Not a lot of options in the SCADA world, unfortunately. Linux is becoming more of an option these days, but you'd still be hard pressed to build anything beyond a basic system on just Linux. All the software runs on Windows.

[@Elrog3]

Was that tied in with the Crowdstrike incident?

[@DasGanon]

@@Elrog3I can't answer that for obvious reasons lol

[@chrissistrunk]

Thanks Grady for another fantastic video ...this one is near and dear to my heart because it describes my job. I've been doing ICS/SCADA/OT security for over a decade now and our security community have been working hard behind the scenes to protect critical infrastructure. Protecting small and even tiny water utilities is a challenge, but one that we are tackling now.

[@ectomorphosis]

Thank you for this Grady. I’m a SCADA / Controls engineer of 20+ years working exclusively on municipal water systems and it’s always been a challenge to describe to others what it is that I work on. This video sums it up so succinctly and highlights the greatest challenge that I and others in my field have been facing the past serveral years: cybersecurity. Long gone are the days of connecting a SLC 5/03 to a 56k modem on a leased line to a PC running RSView32 and calling it a day.

[@TJ-vh2ps]

If your organization cannot afford good network security, it cannot afford to be on the Internet. If it can in any way cause physical harm or extensive damage, excellent network security is required.

[@Stealth86651]

Okay, so you've just about took 95% of businesses and critical infrastructure offline. What's your plan now?

[@marksprings3493]

Probably hire a network security engineer or consultant or what have you ​@Stealth86651

[@Llortnerof]

@@Stealth86651 Only because they're cheap.

[@pendlera2959]

@@Stealth86651 Change the norms (regulations?) so that businesses invest in cybersecurity. Just because things are bad across the board doesn't mean we can't or shouldn't improve them.

[@dontkilImejay]

I work as a cyber security engineer for a lawfirm and this video is fantastic. It is a constant battle against malicious actors constantly poking at you and trying to break through, high stress, but I find it very interesting and fun. I absolutely LOVE the stuxnet story.

[@zyeborm]

And only some of those malicious actors are your own users 😂

[@narrator69]

We're still under water restrictions in Calgary, the damage is more extensive then originally thought. Every time they get a section back to pressure another pipe bursts further down the line. The maintenance on that specific conduit was apparently overdue by a few years but we have so many infrastructure projects going on that some seem to be on the "Worry about it when it fails" list.

[@lostwizard]

Actually, no. There haven't been additional pipe bursts. Those "wire snaps" reported are not pipe breaks. (They aren't good, but they aren't breaks.) Water restrictions were lifted for a while and then fully reinstituted near the end of August when they took the pipe back out of operation to do *preventive* repairs at additional locations they identified as weak. Additionally, they were already gearing up to take the pipe out of service to do a full offline inspection at the end of 2024 (presumably when water usage is lowest) so it wasn't just "worry when it fails".

[@joeeeyyyyyy]

​@@lostwizard 1/4 (at least) of ALL CANADIAN WATER INFRA in major cities is past its useful life and due for replacement. We didn't have acustic monitoring or other systems in place to properly monitor the wires snapping on the main/feeder - this was preventable. We were sold equipment which has a USEFUL lifespan of 50-70 years but only managed 49 - which is expected.

[@joeeeyyyyyy]

​@@lostwizardWe hadn't inspected the bearpaw in over a DECADE. It's okay to admit we need to do better instead of licking boots.

[@pendlera2959]

@@joeeeyyyyyy Stating facts =/= licking boots.

[@Ciubix8513]

Cyber security of a whole bunch of infrastructure is so scarily bad, there are a lot of systems that straight up have an open vnc port (allowing anyone to log in and control it however they like)

[@nUrnxvmhTEuU]

I think I remember you from Mastodon!

[@ado3247]

It's kind of crazy. It's like leaving the door for a control room open, and front facing the road. These are no "hackers" it was most likely just some bored kid scanning the internet for open VNC ports and messing with the values. This is no sophisticated attack, this is quite literally someone walking through a door that was left open. They should be punished by all means, but come on, leaving an unprotected VNC server open on critical infrastructure is such an easily preventable mistake...

[@Ciubix8513]

@@ado3247 yep, exactly

[@haxalicious]

​@@ado3247Similar to the many instances of open MariaDB server with default authentication (no password) and then blaming "hackers" once some kid uses Shodan to find it. How Equifax got in zero trouble for this is ridiculous.

[@zenja42]

yea - worldofvnc and vncresolver are fun to watch... and scary at the same time

[@GeoffCostanza]

Every utility company and business in America needs to see this video

[@suburbanbiology]

Water security is an important topic. Thanks as always for your work!

[@andyvdr]

Hello from sunny Calgary, Alberta. After three months of water restrictions aiming to reduce strain on our remaining treatment plant while repairs on the feeder main were underway, it’s expected we will see the end of restrictions this weekend.

[@Jakeurb8ty82]

I still remember a documentary where they interview a power plant owner talking about how stoked one of his managers was because they had set up his power plants to run via remote from his phone. Owner made said manager rip out the entire system immediately.

[@jeffspaulding9834]

Lots of vendors are pushing that sort of thing these days. "Manage everything from your phone! Monitor the state of your plant while you're having dinner with your family!" The important thing to remember is that those vendors don't care about your security, they care about their sales.

[@JarheadCrayonEater]

As a former SCADA engineer I've designed and maintained several WTPs/WWTPs across the country. There's always one thing people seem to forget. IT security is one thing, but physical security is lacking nearly everywhere. I was the controls engineer overseeing the pump stations in and around NOLA for the USACE and NOLA S&WB, and nearly every station is vulnerable to physical breach at any time, without a lot of effort.

[@davide803sc]

Love the handle

[@JarheadCrayonEater]

@@davide803sc thanks Brother!

[@nitehawk86]

A high-vis vest, hardhat, and a clipboard will get you far. :)

[@JarheadCrayonEater]

@@nitehawk86 no doubt! Act like you belong, and they'll think you do!

[@zyeborm]

​​@@nitehawk86you should try a ladder, you think clipboards get you places. A ladder people will hold the door open and you don't even have to ask. If you really need in somewhere though, a wheelchair. Nobody is going to tell the disabled person no.

[@desmond-hawkins]

I wonder how many millions Ground News has spent on KZbin sponsorships so far. Almost every single channel I've watched over the past few weeks has been sponsored by them.

[@killsode4760]

Do wonder where they're getting the money for it. The service doesn't seem like something that'd make much on its own so they're getting donations or investments from somewhere.

[@LoganChristianson]

@@killsode4760 Probably venture capitalists. Same thing happened with VPNs only a half decade ago.

[@jameshisself7375]

@@killsode4760 They work on a massive problem that the avg joe has no idea is happening, I expect there will be more awareness and more revenue in it in the future. I'm concerned that the bad actors will start a misinformation campaign that they are secretly controlled by one political side or the other. That could even happen organically amongst certain types just due to their constant conspiratorial expectations.

[@jonasstahl9826]

​@@LoganChristiansonYep, spending all there money they earn on more adds to get more customers. So they generate lots of cashflow, which increases they value of the company and than sell the company for a few 100 million to something like Google or Facebook

[@sauercrowder]

@@LoganChristianson If that's what's going on, the promise of neutrality won't hold true for long.

[@UncleBadT]

Im born and raised in Calgary, this issue is still going on. They keep finding more and more problems with that pipe.

[@seangardam1779]

Thank you Grady - I’m an automation engineer in water/wastewater and with the first six minutes of this video I can finally explain to my wife and kids what I do for a job! 😂 keep up the great work.

[@Time2Warp]

Like others have pointed out, most of these are not “hacking” per se, but it is definitely a great intro to opsec.

[@jenaf4208]

That romantic "hacking" rarely ever happens anyways.

[@alexatkinson1023]

Your videos are always well put together, but as a controls engineering student I especially enjoyed this one. A very good description of SCADA and the risks that can come from a poorly secured system.

[@wcntech]

Not a high value target but low hanging fruit.

[@wcntech]

Also, a network's worst security threat is it's own users.

[@nitehawk86]

@@wcntech "Hey I found this usb stick in the parking lot. I'm going to plug it in to my computer at work." lol

[@ikocheratcr]

The issue is that many of these who cares low hanging fruit, when you add them up, it stops being who cares situation.

[@fakename287]

@@ikocheratcr not only that, but successfully picking low hanging fruit might teach you a lot about how to pick the fruit that’s higher up on the tree

[@johnopalko5223]

A lot of these attacks on low value targets are either proof of concept or for refining techniques before going after higher value installations.

[@spiritedgareth]

I worked for a market leader in SCADA software... And it's terrifying how vulnerable the systems are. If you're on the network, you can basically do anything. Even spamming ports randomly could cause alerts to go crazy and shut down a plant. I hope it's gotten better since then... That was 7 years ago.

[@ecospider5]

In a high security building I would disable the usb ports on most computers. They are a massive security risk now. Just a phone charging cable plugged in can hack your computer. Even with no phone attached.

[@ghost307]

Those ports can be physically clocked with specially designed locks. A contractor told me that he could just disable the jacks using software, to which I pointed out that they could also be re-enabled using software.

[@BeTeK11]

This is actually normal process. Put crazy clue to usb ports so no devices can be inserted

[@howichangeyoutubehandle]

@@ghost307 or epoxy

[@Senthiuz]

Specially designed locks... or glue.

[@saiv46]

@@ghost307 Some motherboards have a setting to disable USB ports, which is a little bit harder with other measures like BIOS password and Secure Boot.

[@jadenfurtado4376]

This was a great video! I've worked on the OT side of cybersecurity. To say that things are bad is an understatement. Thanks for raising awareness about this problem!

[@dionh70]

One of the problems I see is that the higher-level executives and bureaucrats that set funding priorities simply don't accept or even understand the most fundamental aspects of security, be it cyber or physical, and thus they simply aren't willing to fund the necessary expenditures until AFTER something goes badly awry. Then, they sit their and send out nasty emails and hold meetings, looking to pin the blame on some underling, refusing to accept responsibility for their failures.

[@Staymare]

I'm from Calgary, and you definitely should do a story about our water pipe break. Our system was built with an intended 50 year lifespan about 50 years ago, and we are now dealing with massive infrastructure network that is aging out while the city failed to plan for any replacement ahead of time.

[@Genesis8934]

~ 9:50 It's almost like BSG (the 90's version) was prophetic for its time. Galactica being the only ship of the humans that had no networked systems on board to allow the Cylons to exploit lol.

[@marilyntaylor8652]

These are such well written stories. Some, I already knew about and others are new to me (before my time). The narrator is so very good. Thank you for doing these stories.

[@thepunisherxxx6804]

These systems should have constraints on those fields where you know there is a cap or upper limit that would never be crossed, it shouldn't allow a 11,000 parts per million entry for the salt for example. No field constraints is just begging for a mistake, and it looks like that's what the FBI concluded too.

[@twestgard2]

Okay but it can’t be imposed by software. Something physical like a bushing that physically can’t transfer more than a certain amount, or a pump that physically can’t move more than a certain amount of material, or have the equipment run on a trickle charged battery that shuts down if it runs extra long.

[@schok51]

​@@twestgard2ideally there would be redundant protections, both software and hardware. Hardware costs a lot more to change, so software is easier, but yes less safe/more vulnerable to hacking.

[@Arcidi225]

I would guess that those checks were in place. It was just within the limits. Maybe there are occasions where this amount is needed. Or maybe you need it for very short period of time. I don't know to be honest. But what I do know, is this is such a trivial thing to add, that it for sure was there. Although... Maybe nobody implemented it, as it wasn't specified in docs. Yeah, probably specification and communication hell, as it's common in projects like this.

[@thepunisherxxx6804]

@@twestgard2 Why can't it be imposed by software? The hacker had access to the user, not direct access to app parameters. If those software constraints existed or set off alarms or required group approval to exceed maximum threshold that would have prevented this. Your solution is way more complex, expensive, and cumbersome. Machinery isn't built to hit specific limits, that's not reality. You can certainly have more physical fail safes, but for complexity and cost and long term maintenance I think software constraints and an escalation system to go over those limits in weird situations is more than enough IMO.

[@Smokeisprogress]

@@thepunisherxxx6804 Physical limit implementation is hardly complex, expensive, and cumbersome, it's basic design.

[@mattm7220]

Australian here, with friends who live in and around Maroochy (or Maroochydore, I should say when talking about the (quite large) city - Maroochy is the name for the wider shire area). After hearing you mention the hack to the treatment plant, I asked them if they remember anything about it, and apparently it was never even reported on by local news - just completely ignored. Even trying to search for it only brings up international news articles. I imagine the lack of reporting on it has a lot to do with it being a huge tourism area, and no one wanted the bad publicity.

[@greedyProphet]

5:40 for me, part of the nostalgia also comes from the color choices. The grey backgrounds with CMY colors just hits different.

[@kevito666]

Best casual definition of "0-day vulnerability" I've ever heard on KZbin- delivery and all.

[@peterdieleman303]

What incredible timing to publish this today.

[@NetTopsey]

I have been saying what you have made this video about to my students at the college I work at for years, often using the same examples that you do. Even to what you say at the end that what "we" do is all that stands between us and some kind of cybersecurity incident. Thanks so much for helping raise the awareness of these issues

[@falconmick]

Great job with the video. Wasn’t expecting such an accurate and well informed video on cyber from a non info sec channel

[@forgotMyUsername100]

I program computers for work & recently got a factory job using SCADA & HMIs -- was cool to hear about this & I may have to show your video to our project-manager to help get a security I found fix-prioritized 😂 Love your channel, this is video is the most personally applicable I've seen yet.

[@sdracklryeg]

Absolute impeccable timing.

[@logiciananimal]

Love to see other professionals in other fields championing cyber/digital/etc. security. Thanks from all of us in the profession.

[@Luzgar]

The key to security is reducing the attack surface. Have a single door, and put a good lock on it.

[@haxalicious]

Well, have a single outside door. Have multiple internal walls too, so that someone can't just do whatever once they get in.

[@Xannthas]

I think generally the law regarding fire safety gets in the way of that, like, imagine if someone lost control of their truck and crashed into it, or some bad guy blew it up, or a fire breaks out and now that area is a firestorm, that kind of thing.

[@Luzgar]

@@Xannthas Well, I was thinking it more as an analogy for cybersecurity (don't connect everything to the internet). Don't have more access than strictly necessary to critical components.

[@zyeborm]

That's awesome, Iran had that. They had no access at all. Didn't really help them much did it.

[@Luzgar]

@@zyeborm When your threat model includes the CIA and the NSA, you are indeed in trouble. (Though not allowing thumb drive would have helped.)

[@JustinDrentlaw]

I was really hoping you were going to mention Stuxnet and you did! I remember reading about that back in 2012(?) on Ars Technica and thinking that it was one of the craziest freaking things I'd ever heard about! And perpetrated by my own government, no less. Really just wild that happened.

[@rallias1]

So, I feel like it's important to mention... the ransomware didn't kill the pipeline, just the billing system. They could have run without the billing system, but they'd have to document and bill after the fact.

[@nicholasjackson2833]

As a retired civil engineer (a British Civil Engineer - the best of course 😊) Grady always gives me warm glow. Grady you talk a language I feel at home with. Watching your videos are just the best.

[@cn8191]

"Any man who must say, 'I am the king,' is no true king"

[@Recovering_Californian]

Software Developer here: The biggest challenge as it relates to cyber security are users. In plain english, you all are lazy and complacent. Users create weak passwords. Users use the same password for everything. Users bring USB drives home and use them at work (and vice versa). Users share passwords. Users ignore security protocols. Users allow their kids to play on work computers. Users install unauthorized software on work computers...and on and on ..

[@lehpares]

Your videos, besides the excellent engineering showcase, are relaxing and soothing.

[@kingofthend]

Interesting timing considering the current news :D

[@Xxshadowman11xX]

Electrical engineer here. I work closely with the SCADA engineers at different utilities to ensure we are prepared for these sorts of events. There is a surprising amount of redundancy built into control systems for power grids. There are often at least two and sometimes three completely isolated control rooms in different areas of the state for local grids (think 10-50 Gigawatts) in case of natural disaster, terrorist attack, cyber security event, etc. While local substations may only have one RTU, anytime there is more than a couple hundred megawatts flowing through a station there are multiple RTUs connected to different networks over different physical routes back to different control rooms. One could be compromised and we would still be able to control the system just fine.

[@gopropeterchockey]

I can just picture Homer Simpson adjusting radiation levels at the local nuclear power plant with this system.

[@keiyakins]

Season 7 episode 7, "King-Size Homer".

[@Smokeisprogress]

More accurate than you might imagine, the chemical plant control rooms in S Texas are scary places, Luckily the systems are smarter than the operators these days; but fires, spills & explosions are still regular occurrences behind the gates, only reported when they can't hide it.

[@mattbennion779]

Calgarian here - we are STILL on water restrictions because of that poorly maintained feeder main. Our other water treatment plant will likely require extensive maintenance from having to hold the fort down for so long, I imagine.

[@glennac]

Grady, I miss the intro music. 😢 Along with the swinging hard hat and garage demos, the jaunty intro music is one of your trademarks.

[@MrModTwelveFoot]

His most recent video prior to this had garage demos though?

[@MeriaDuck]

I think that present day KZbin videos tend to start immediately. The intro music may lead to less retention, and therefore watch time.

[@themacker894]

Congratulations on nearly 4m subscribers! I'm also a nerd and I salute your accomplishments in the social realm!

[@MrHeavy466]

Pirate Software's Thor used to work for the federal government probing critical infrastructure like this for weaknesses. I always thought that would be such a cool job.

[@knight_lautrec_of_carim]

If you're a fan of Thor NEVER look up Maldavius Figtree

[@Reverend_Salem]

not only is it a cool job, its also important.

[@johnm5928]

The idea of workers physically opening valves and checking floats, etc, can be applied to our own homes. So many people are getting smart home devices to increase convenience (I guess). Just like a connected critical infrastructure system, your connected home is now vulnerable to literally the same hackers. Not to mention giving them additional ingress points to your personal sensitive data. IMO there's really no need for a smart refrigerator.

[@tsbrownie]

I have an "elevated storage tank" that is max security and totally immune to remote hacking. There's a pump, float switch, and solar panels. ;)

[@FirstLast-jl6fr]

Time to get my paint bucket dispensing drone out then!

[@webluke]

I worked at a power plant simulator company. It was cool seeing SCADA discussed because I spent much of my time reproducing many different manufacturers' HMI systems so they could be connected to the simulation model built by the engineers. IT was always paranoid with our simulator computers, which were always on a separate network. It was also hard sometimes to get the data from the plants so we could simulate the various systems accurately. I did do much of the computer work for them too, and had to go to a few sites that were never connected to the internet, and Windows update would corrupt trying to get updates killing the networking, so I got a few quick trips out to random places to fix Windows.

[@viaticknight8813]

3:51 why is the water so acidic?? 😂

[@bmxican1023]

Lmaoooo pH of “2”. Good catch!

[@bruce-le-smith]

Great video as always Grady, thanks! Cyber/Digital security has been important for a long time now, but each year it's increasingly important. Even though there are risks as you say, we're lucky that most of our infrastructure is still on the ground... The more we put into orbit the more there will be to fall on our heads during a criminal attack.

[@texastaterbug5395]

I think it is worth reminding folks that Colonial could still operate their pipeline. They just couldn't get paid, so they stopped operating it.

[@sucraloseUncle]

Excellent video Grady! This video should be included in every new employee orientation package for many companies.

[@baylinkdashyt]

It's funny you picked this week to release this piece... as I just got done rereading Red Storm Rising... which starts off with just this sort of attack on a Soviet refinery plant.

[@michaelashley2855]

Those Russians !

[@robertb3409]

What a great video. In my old electrical company. They had a contract to monitor and repair a green waste facility. The new owner of the electrical company saw an internet bill was a large amount. He cancelled it and ended up shutting down the whole scada system

[@donalddodson7365]

Thanks!

[@JazawaToad]

Thank you for the education, Grady. I always enjoy learning in these videos.

[@Deja117]

I thought you were about to plug a sponsor for an antivirus there for a moment. Yes, while there are paid tools that are very good at stopping infected USB's from even running on your computer, there are also hundreds of free tools as well. The sensible thing to do, is obviously not to plug in a random USB you find on the ground, and use some common sense navigating the internet. Even that's not a guarantee though. That said, I have found certain free extensions really useful for blocking malicious websites. They're not too hard to find online.

[@local-admin]

As a cybersecurity professional. Thanks for pushing this to your viewers!

[@Artoooooor]

I hope other similar systems are checked for that vulnerability now.

[@FuncleChuck]

They never are. There’s no money in prevention and maintenance. It’s all wasted on those giant scissors for the ribbon cutting on the big new thing

[@keith_5584]

Bingo, always keep in mind the obvious thing could be a distraction.

[@schmechel6888]

everything you post is just gold Grady, salute sir 🫡

[@msromike123]

It seems like these systems would have hard coded constraints on high and low limits to prevent HMI errors or hacking? Why would 11,000 ppm even be allowed as a set point? On another note, would you not back up your billing computer so it could be restored to a known good state (like hourly)? Seems like all of this can be mitigated and is easily foreseeable. It makes me wonder if we as a nation are adequately implementing anti-hacking mitigation into our infrastructure proactively.

[@CaptainZavec]

For systems designed now, that is a thing people consider! I've seen a few talks on it at conferences, generally under the name "cyber-informed engineering." A good example is the sodium hydroxide thing mentioned in the video: if the tank that holds the concentrated hydroxide is small enough compared to the amount of water in the mixing tank that even if it were completely opened up it wouldn't reach a dangerous level then it doesn't matter if somebody hacks the HMI to open it up (as much)! A big part of the problem though is how long these systems live. Infrastructure lives for 20, 30, 40+ years so even if we start building everything with perfect security now it's still going to be a long time until all the old stuff is phased out.

[@Renegade605]

The problems with easily foreseeable and mitigated are: 1) We have the benefit of hindsight, which makes it hard to be objective about how easily it really was foreseeable. 2) 10,000 easily foreseeable things aren't easily foreseen anymore unless you have the time to think of 10,000 things, which we almost never do.

[@msromike123]

@@Renegade605 Thus why I asked my real question and gave examples. Are we as a nation being proactive enough?

[@Renegade605]

@@msromike123 no, no one is. But also, what is "enough"? Enough to prevent any possible attack from ever happening? Well that probably isn't even possible. Enough to prevent loss of life due to a cyber attack? That didn't happen here, so by that metric it was enough, right? Just like every other engineering decision Grady ever talks about, there has to be a balance between cost and risk.

[@christianmotivates]

I’m from lubbock and this was such a cool watch. Thanks for this body of work!

[@Anthony-fk2zu]

Air gaps would be good it sounds like.

[@erictucker5633]

It would be good to go over the differences between IT and OT related to cybersecurity as it relates to industrial plants. Great video, by the way!

[@mooodeuce]

Grady dropping this at the same time as Israel's latest cyber attack is wild.

[@oolureoo]

What happened this time? 😭

[@CaptainZavec]

@@oolureoo probably the pagers exploding in lebanon

[@oolureoo]

@@CaptainZavec Yeah, just read about it, it's crazy

[@leenevin8451]

Looks like it’s non-cyber

[@mooodeuce]

@@leenevin8451 Maybe no hacking involved as it originally appeared. Either way they still detonated them all remotely at the same time using the network so it's still cyber.

[@CM-kl9qh]

You’re speaking my language. For most every analog value (ex. tank level) there were seven values: Hi Hi alarm Hi warning Hi limit Field measurement reported Lo limit Lo warning Lo Lo alarm The field measurement was what the sensor in the tank was feeding into the PLC (Programable Logic Controller). Limits were start and stop points of the operation. Warnings would send an alert to the licensed operator, but the system could be left alone to try and bring the field value back within limits. Alarms were sent to operators, management and support staff to take action and may prompt the PLC to take further corrective action.

[@xspager]

A lot of this attacks are """educational""" if you know what I mean. I would be 100% fine with them if wasn't for the geopolitical implications.

[@Sage2291]

As you well demonstrated, the best security can be compromised by a seemingly innocent action. Unfortunately, one person's innocent is a hackers golden key at times. And there are times that pressure placed in the system to accomplish a goal immediately can lead to these kinds of breaches. Not always clear until after the incident and the analysis... Thanks Grady!

[@galacticmechanic1]

The scarier thought is not that someone might try to break major city infrastructure, but breaking into smaller towns infrastructure that is possibly less protected. just wait until the small town a lot of the trucks pass through to get to a major city like new york has their water overflow or power goes out and then all the trucks stop.

[@McTroyd]

Excellent illustration of applied cybersecurity, Grady. Thank you! 👍

[@JBrinx18]

Beard is looking good. Ironically, even though you said you didn't use it often, I did end up getting that aerospace razor you were sponsored by. It is excellent, definitely a close shave with no nicks

[@Numerlon]

I have to mention that I love the very insightful animations! They are very useful :)

[@jerrys.9895]

This video should be mandatory viewing for any employee that interfaces with a critical system. It's so easy to assume that your credentials are not high-level enough to do any damage if compromised.