Back
Tales From the Road: OT Security Assessment of Water Utility Illuminates the Path for Widespread Industry Improvement
Our recent OT security assessment at a private water utility illustrates how no industry is safe from security threats. Regardless of sector, all industries, from education, to finance, to water treatment, are susceptible to bad actors taking advantage of their internally-overlooked vulnerabilities. While water utilities have historically lagged behind other industries in OT security, the private facility we recently assessed illuminated the ways in which the industry as a whole can bolster its OT security efforts.
Preparation, Evaluation, & Identification
To our pleasant surprise, the private water utility we were asked to assess “passed our test,” so to speak, with at least an A-. We found that their OT security systems were adequately segmented and hardened, save for only two identified vulnerabilities. While we commend this particular utility’s OT security strength, the industry is still lagging overall, and even this fortified utility was found to have opportunities for improvement. It’s this and one other main reason why performing OT security assessments in this industry is paramount: four years ago, the American Water Works Association (AWWA) reported that government intelligence confirmed the water and wastewater industries are under threat from a foreign government’s intrusion campaign, criminal actors, and threat groups.
The bottom line is this industry needs to have its OT security systems evaluated and bolstered across the board as we are well-aware of the very real threat it is under. Thanks to our recent OT security assessment at the utility being discussed today, we are now better able to shed light on how utilities throughout the sector can improve their own OT security systems.
Observed Strengths & Vulnerabilities
Let’s take a look at the strengths and vulnerabilities of the water utility we recently evaluated.
Identified Strengths
- Unsuccessful Password Spray Attacks: No accounts were compromised thanks to this utility implementing best-practice account lockout settings.
- Proper Network Segmentation: We observed appropriate restrictions between different network segments as well as proper firewall configurations.
- Account Lockout Policy Properly Configured: Proper lockout controls were found to be in place, which are key for preventing user accounts being breached by brute force attacks.
- Robust Active Directory Implementation: The Active Directory implementation followed industry best practices.
The above identified strengths at this utility pointed us in the direction of examination rather than widespread amelioration — at least at this particular location. Thanks to their stringent adherence to the majority of key OT security system standards, we were able to pivot from an overhaul of their systems to an evaluation of what they have been doing right. This information can help us bolster water utility OT security across the entire industry.
Identified Vulnerabilities
- Device Lifecycle Management: We discovered end-of-life devices without a formalized end-of-life matrix for OT assets.
- Documentation: The equipment installed in the field was found to have inaccurate documentation and lacked documented policies and procedures regarding network security protocols.
The above identified vulnerabilities highlighted some common areas of weakness in the water utility industry.
The Philosophy of a Robust OT Security System
As we identified this particular utility to have adequate OT security systems in place, we’re focusing on the “philosophy” of a solid system; how to maintain and enhance it, rather than how to achieve it in the first place. In this section, we will go over how we advised the utility to address the few vulnerabilities we found in an effort to help to enhance the already existing OT system. This section will mainly focus on their relations with third-party vendors and clients, a point of mild weakness that we found but one that can open any organization up to attack.
First and foremost, it’s important to note the level of cruciality any utility operates under on a daily basis. This utility provides water to a nearby town, meaning that a breach in their systems could affect an entire community’s drinking, cooking, and bathing water — essentially, utilities do important work that influences whole communities and cannot afford to be compromised.
We found that their provision of a third-party vendor to access their OT system was a point of potential weakness, even though they only allowed the vendor entrance in view-only mode. To address this security gap, we advised the utility to push this area of work to the cloud and require the third-party vendor to remain outside of the network.
The next piece of advice we shared with this client was the necessity to bolster their documentation. We found that their security team was not properly documenting policies and procedures, leaving the door open for a breakdown in security maintainability. Ensuring that their policies and procedures are adequately documented moving forward is the best way to guarantee that anyone could uphold security measures when needed, such as in the event that an employee unexpectedly falls ill or quits.
Identifying Other Ways to Fortify Water Utility OT Security Systems
Investing in continuous patching, updating, and maintenance of OT security systems is crucial across all industries. Some of the recommendations we made for this water utility to bolster their OT security are as follows:
Assisting With NIST CSF
The NIST Cybersecurity Framework is a set of guidelines published by the U.S. National Institute of Standards and Technology intended to simplify organizational cybersecurity assessments. We found that the head of security at this water utility was previously unaware of NIST so we were glad to help by introducing them to the organization. With this new knowledge, the entire security team will be better able to assess what tier they are operating at with each of their controls.
Maintenance
Our maintenance-bolstering suggestions are somewhat lengthy, so we will summarize them in the most effective way possible.
- A “Test Bench” — Whenever an operation is directly related to public health and safety, we double down on our common recommendation that security updates and patches be run in a test environment previously implemented in the main system. A “test bench” involves a separate set of servers and a logic controller to establish an isolated network environment.
- Communication — Effective communication between the IT and OT departments are crucial to the success of a smoothly functioning operation.
- Investing in a SOC — SOCs are an extremely helpful addition to any security system that can bolster the usefulness of monitoring software.
- Documenting and Practicing Policies — We mentioned this one previously in this post, but it’s well worth reiterating: proper documentation of security systems can be a life-saver when unexpected situations come up. Document, document, document!
Key Takeaways from Our Water Utility OT Security Assessment
In the end, there is always room for improvement, even at operations that are doing a fantastic job of maintaining their OT security systems. Continuous examination of OT security systems (and IT systems, for that matter) is crucial to the continued success of the location we evaluated and other water and wastewater utilities across the map. This particular OT security assessment revealed the possibilities and the need) for widespread improvement in the industry and it is our hope that this report will aid other organizations and utilities in bolstering their efforts.
For professional assistance in improving your utility’s OT security system, contact us today or call us at 1 888 720 4633.
10.16.23