One Step Forward, Two Steps Back: A History of NERC CIP
One Step Forward, Two Steps Back: A History of NERC CIP
Rewind back to the 1960s. The first communication satellites are being launched into space. DRAM is being used for the first time. The first LED was created. The first computer programming language began to take shape. DARPA started to think about how computers will connect to each other to share data. The Beatles are tearing up the music charts, and the Vietnam war became increasingly deadly. One specific event, a blackout in 1965, caused by a high demand for power on a cold night in Ontario coupled with a mis-programmed protective relay, started a chain of events causing a widespread blackout across New England, New York and New Jersey. The United States Federal Power Commission investigated and reported on the blackout and recommended:
“A council on power coordination made up of representatives from each of the nation’s Regional coordinating organizations to exchange and disseminate information on Regional coordinating practices to all of the Regional organizations, and to review, discuss, and assist in resolving matters affecting interregional coordination.”
⎯ Legislation proposed: Electric Power Reliability Act of 1967
Enter NERC (National Electric Reliability Council), quietly formed on June 1st, 1968 as a voluntary organization by the electric utility industry to promote the reliability and adequacy of bulk power transmission in the electric utility systems of North America.
As the years went by, new regions and members were added, and the organization analyzed and made recommendations to increase the overall reliability of the North American power grid. It also become very clear that utilities in many cases were operating as monopolies because they were not agreeing to a rate reduction for large industrial companies or allowing those customers to shop for better rates. The lack of flexibility within the electric utility monopoly created the drive for deregulation. NERC enters to help here, as well, regarding competition, fair pricing and how to work with independent power producers, marketers and brokers.
Who remembers Y2K? NERC was there to ensure that the grid was prepared and that there were no interruptions as NSYNC and Jessica Simpson rang us into the new Millennium. Right around this time, NERC stepped into the security arena and became the spearhead for issues related to national security and the power grid. To formalize these efforts, NERC became a founding member of the Partnership for Critical Infrastructure Security (PCIS). Deregulation is now running full steam, and California is lead into a power crisis due to a little company called Enron and their manipulation of the power market. Clearly this relationship is getting complicated.
Then in 2003, a butterfly flaps it wings in Ohio and creates one of the most widespread blackouts in history. People start asking questions. How can some foliage that affected transmission lines in Ohio impact power to the NYC subway system for 2 days? This blackout puts the NERC Urgent Action 1200 temporary standard on the map and fast tracks the development of NERC CIP (Critical Infrastructure Protection). Anyone who believes that NERC CIP was truly born from the need for cybersecurity was not a utility employee in the Northeast in August 2003. This is where the reliability vs. security dance really begins. Fast forward a few years to 2005, and the NERC CIP Version 3 is drafted and goes out for public review comments to over 61 separate entities. The standards are agreed to and published.
Utilities realize that to properly comply with these standards they need to make investments in time, training, processes and tools, none of which are planned or budgeted for, and rate hikes are not popular, so what happens? There is enough flexibility in the standard for utilities to do some interesting things. Entities were able to define their own risk-based methodologies. Weak contingency analyses are done. Infrastructure changes are made to minimize the footprint of the Electric Security Perimeter. Serial communications and non-routable protocols are frequently leveraged to take as many assets as possible out of scope. This also stunts the transition to IP connected devices and slows grid modernization. Data diodes become extremely popular as a method of limiting inbound communications and further reducing assets in scope.
The dance continues, and NERC CIP v5 is ratified in 2013. Modifications are created to remove some of the gray areas. It creates the concept of the BES (bulk electric system) and Cyber System Categorization. Bright-line criteria are published, and critical assets that do not have non-routable connectivity fall into scope. Utilities began budgeting and planning for the changes. Compliance and audits with the guidelines become more widespread. Even with this new clearer guidance utilities continue to employ strategies to keep as many assets as possible out of scope. Most commonly physically or logically breaking up High and Medium impact BES Cyber Systems to reduce their compliance requirements. Often this included large capital investments in new infrastructure purely to avoid having to comply with the new standards. Did this really make our nation’s infrastructure more secure?
2013 was also significant for another reason, but this time physical security was in the literal crosshairs with California in the spotlight once again. I’m talking about the sophisticated physical attack on a substation. A team of highly trained operatives cut the communications to the substation and systematically shot the cooling systems of several transformers with high powered rifles. The police arrived and were unable to enter the substation due to a locked gate. They surveyed the area to the best of their ability and ultimately left after finding nothing suspicious. All told, approximately $15M in damage was done to this utility’s equipment, and the substation was offline for approximately a month. Amazingly this attack did little to impact the stability of the grid as power was able to be re-routed from other parts of California. Now, I’m not one to normally don a tin foil hat, but something does not add up here. Like clockwork NERC introduces new standards (NERC CIP-014) around physical security to address the attack. I continue to maintain that these types of attacks are still the biggest threat vector to the grid, and it should not take another attack to decide we need more protections in place.
The standards continued to evolve, covering uncharted area like use of removable media, transient assets and supply chain. Again, more solid guidance, but the devil is in the implementation details.
In one of the latest developments, utilities are now required to report on “attempts to compromise” their infrastructure. At face value, this is a very noble goal. In practice, another regulation riddled with complexity and interpretation. Let’s back up for a second. First, would this have help anyone exploited by the SolarWinds hack? Likely not since it came through a compromised update server that appeared as a legitimate update to the casual observer. A new regulation related to better managing the cybersecurity requirements for your vendors would have been more appropriate. But wait, don’t we have NERC CIP-013? News flash: that would not have helped with the SolarWinds hack either.
Second, it’s completely unclear what NERC wants here. Is it a list of all the traffic that was denied from your edge firewall? What about a dictionary attack against something that has secure remote access turned on with multi-factor authentication? What about someone driving around your facility attempting to join a well-protected wireless network? Does having this information help the greater utility community? This could quickly turn into millions of events per day for a large utility with little to no value to NCCIC and the E-ISAC. The recently rescinded (third time’s a charm, right?) guidance on what constitutes a compromise was better but still leaves too much ambiguity.
I can only draw parallels to the daily struggle of parenting and being a surrogate teacher for my kids during this pandemic. Things started fine. We had a plan; kids were on a schedule, get up, eat do your school and commitments and then you can have some free time. School work was being completed and after a few weeks I figured the kids were in a groove, so I started to relax my oversight to actually get some work done during the day. Without the constant oversight things quickly devolved. Sleeping in, poor hygiene, kids joining their afternoon virtual meeting with their teacher shirtless in 3 day old, stained pajama pants eating Lucky Charms.
So, father FERC I understand where you are coming from – time to rein the entities in a bit. But do it collaboratively or this dance will keep occurring between security professionals, utilities and you, and the regulatory bodies. Until all three come to the table to understand their strengths and weaknesses, risk and probability, and operational impact, NERC will keep pushing new CIP regulations. Utilities will keep adopting a compliance-based security methodology. And security practitioners will continue the hyperbolic conversations of what could happen next.
Compliance Guide: The NERC CIP Standards