Securing America’s Largest Water Utility Infrastructure

Take a moment to imagine the vastness of New York City’s Department of Environmental Protection (DEP), with plant operations spread over 2,000 square miles, including 300 facilities and thousands of regional centers. Then consider how critical its mission is.

More than 8.6 million New York City residents depend on the DEP for the most essential element of life: clean drinking water. Additionally, the DEP manages 21 wastewater treatment plants that process 1.3 billion gallons of sewage daily. Both of these are revenue-generating utilities that also task the DEP with collecting money from the NYC rate payer. Finally, the DEP is, as its name suggests, an environmental protection agency. As such, it’s responsible for air and water, including hazardous materials and noise pollution—and all of it must be secured.

It’s an enormous undertaking for any team—particularly in a pandemic—but Business Information Officer Cecil McMaster and his team in the DEP’s Business Information Technology (BIT) group are taking it in stride. In fact, as a result of their efforts, the DEP has already received commendations for quickly pivoting to support a remote workforce, along with other commendations for improving efficiencies while securing the DEP through its digital transformation. It’s becoming a model for agencies around the nation.

The DEP’s primary mission is to manage and protect a sprawling critical infrastructure from physical and cyberattacks that could damage the environment and physically harm the 8.6 million customers that the agency serves. Within the DEP, the IT team operates as a business leadership unit responsible for continuous improvements.

Since early 2019, the DEP’s BIT group has been leading an agency-wide digital transformation by steadily upgrading its legacy systems to cloud-data center and customer applications. Then came COVID-19, which sent 3,000 nonessential workers home in March of 2020. Despite the COVID risks, the DEP’s essential services couldn’t just stop. If anything, the pandemic drove more security requirements around the sprawling, statewide water and waste systems serving the Big Apple.

The DEP is part of New York City’s massive interconnected network of over 100 citywide agencies supported by a single security operations center (SOC) among them. That means the DEP’s small IT team (BIT) operates mostly independently of the citywide SOC but also in coordination with it. It also means that, without proper protection and segmentation, any of these interagency connection points into DEP could allow malware or unauthorized users to enter the DEP’s data center operations—or worse, allow access to the water and waste system controllers themselves.

To enforce the Zero Trust security model, the BIT needed to migrate from legacy firewalls to next-generation secure gateways that would carry out machine safety checks and enforce access protocols for all interagency connection requests. The BIT also had to implement rules for protecting connections made by workers in the field—who are using their hand-held devices to take water and air quality measurements, for example—and transmitting that information to the data center for analysis.

As it migrates more applications to cloud-based services as part of its digital transformation, the DEP must continuously scale protections to support new third-party business connections, such as its third-party cloud-based customer billing application that will be going online in mid-2021. “When moving from a mainframe to hybrid cloud, our questions became, ‘How do you protect user access and customer information when we move to a hybrid cloud environment?’” explains Farhan Abdullah, Director of Production Support.

They knew they needed to migrate to these services. The pandemic has made services like contactless billing even more vital at a time when those same services are also ripe for attacks. So securing third-party services has become paramount to maintaining DEP’s security.

With a small IT team supporting the agency’s digital transformation while also protecting critical infrastructure systems, the DEP needed a next-generation firewall partner that could provide visibility and intelligence across its cloud-based services and on-premises appliances. The team needed a security partner to help the DEP meet its strategic and evolving requirements with both present-day solutions and forward-looking planning.

“It’s all about the partnership: How do we continue to evolve and scale as a company, and continue to grow our partnership with our vendor so we can secure our environment today and in the future,” explains Michael Shum, IT chief of staff.

Importantly, Shum emphasizes that the security tools and services supporting the DEP’s operational infrastructure and business transformation need to provide full visibility into the security state of the DEP’s entire environment, including their workers’ system activities, along with strong reporting to support the BIT’s security benchmarking and compliance mandates.

When upgrading security to support more apps in the cloud, McMaster and Shum turned to Palo Alto Networks. This decision was made easier because Palo Alto Networks was already a trusted partner providing cybersecurity protection at key locations across the 19 business units within the DEP.

“We’ve been using Palo Alto Networks appliances the last two years. Time and time again, they’ve been instrumental in helping the DEP to achieve security both in terms of locking down the clouds and securing our ICS side,” says Vic Kayharee, cloud engineer. “They’ve delivered the global spectrum of security technologies we need to fulfill our new data centers.”

Below are some of the ways those Palo Alto Networks tools and services are implemented at the DEP to support its mission and challenges.

Securing Home Worker Connections. When the DEP suddenly sent 3,000 of its employees to work from home during the early days of COVID-19, it utilized Palo Alto Networks GlobalProtect Next-Generation VPN to enable always-on IPsec/SSL VPN access for home workers. The DEP uses these enhanced VPN services to scan the state of devices requesting access and to quarantine those devices that are compromised or that don’t meet patching and security criteria.

Palo Alto Networks virtual firewalls—Palo Alto Networks VM-Series—and on-premise next-generation firewall appliances monitor traffic from internal, external, and remote worker devices to detect, sandbox, and mitigate threats before they take hold in the organization.

Supporting Digital Transformation. To secure its growing catalog of cloud-based applications, including a new water billing system going live in 2021, the DEP also uses Palo Alto Networks VM-Series.

Immediate Threat Intelligence. The BIT relies on Palo Alto Networks Threat Intelligence to prevent new unknown threats across DEP’s cloud-based systems. When new threats are reported, the BIT team will check in with the Panorama centralized threat management dashboard to find that Palo Alto Networks already has the intelligence and is ready to detect and block them.

“We use multiple tools for threat intelligence. But Palo Alto Networks is my primary go-to for threat intel and sandboxing, while the others are more secondary,” says Vic Kayharee, cloud engineer. “It gives us the most comprehensive sandbox analysis and usable, actionable threat intel.”

Shum and McMaster appreciate that Palo Alto Networks uses compatible, integrated technologies and protocols across all its tools and services. Throughout the DEP’s digital evolution, the process of upgrading, integrating, and adding new Palo Alto Networks tools, services, and capabilities has been easy to accomplish and adapt for new uses.

The BIT team members use Palo Alto Networks Panorama management dashboards to drill down to key areas of need rather than overwhelming staff with too much information to act on.

“Looking at intelligence from Palo Alto Networks helps us proactively understand what’s happening throughout our world with real-time feedback,” explains McMaster. “Using the single-pane-of-glass view into our complex environment, along with added intelligence that aggregates with our logs from all over the world, we can distill down to the things we need to strengthen our security and make the DEP engineers more responsive to the risks.”

McMaster adds that Palo Alto Networks tools and services are among BIT’s most trusted means of taking the pulse of security health across technical systems—such as measuring uptime and responsiveness, counting outages, gathering user feedback, and preparing for audits related to the utility’s sale of bonds to taxpayers.

“Because of this intelligence and insight, we can tell our team to look at specific areas of risk and the things they need to do to secure it,” McMaster points out. He concludes, “That’s the goal: to make the DEP engineering staff smarter and to continuously monitor and improve our level of risk.”