Protecting Data From the Bighorn Fire

This is the WAPA site on Mt. Lemmon that was threatened by the Bighorn Fire. As the fire approached, the site had to rely on its generators and was in danger of being shut down, so an alternate path for the data it processes had to be found quickly.

The Bighorn fire in the Catalina Mountains north of Tucson had already been burning for more than a week when Dallon Lines got the call from an official at the Western Area Power Administration (WAPA).The WAPA communications site on Mt. Lemmon, a critical junction where Arizona Generation and Transmission Cooperatives (AzGT) and WAPA SCADA data (Supervisory Control and Data Acquisition) is exchanged, was in the still-growing path of the fire, running on emergency generator power, and in danger of being shut down.Unless an alternate communication path could be found, and quickly,there was the possibility that the comms site (WAPA Mount Lemmon) could be lost and with it, the critical information needed both by WAPA and AzGT.

“It (the WAPA site) is essentially all our circuits that we hand off between each other. With WAPA being our BA (balancing authority) we have to supply them data from our RTUs (remote terminal unit); so it’s one of various sites where we have to give them data and hand it off to them back in Phoenix, to their control center,” said Dallon Lines, AzGT communications engineer II.

The data originates both at the Apache Generating Station and the Benson campus, and from those two locations it goes to an AzGT communications site in Vail. From there, much of the data is transmitted to the WAPA site on Mt. Lemmon before going to another tower west of Mt. Lemmon called Neuman Peak, where it is then transmitted to the WAPA control room in Phoenix. There were other things at stake besides just getting the data to WAPA.

This is the site at Vail where data is transmitted to the Mt. Lemmon site. When the Mt. Lemmon site was shut down because of the Bighorn Fire, an alternate path had to be found in order to get the data from this site to the WAPA control center in Phoenix.

“The second thing is how we pass data off; a good portion of our sites, for example in the Mohave area, we don’t have comm (SCADA and meter data) that reaches all the way up in to Bullhead City, so what we do is rely on WAPA – we basically take the comm (data) from Benson or Apache, our two hubs, to Vail, and then we hand it off to them and they take it for us all the way to the Mohave area,” Lines said.

Lines’s sudden challenge was to take the huge amount of data being transmitted on the Mt. Lemmon pathway and move it to what’s called a DS3 line that went directly from Vail to WAPA’s Phoenix control room. There wasn’t a lot of time and Lines didn’t know at the time if the DS3 line (alternate path) had the capacity to handle the increased amount of data – about twice the amount it was currently handling.

“It involved working with WAPA’s comms center – their hub in Phoenix – to log in to our equipment and re-route communication, re-route the data, from the Mt. Lemmon path over to this Level 3 path (DS3 line),”
Lines said.

The critical question: Could we re-route all of the data from the Mt.Lemmon site to the other pathway – already at 50 percent capacity – or could some data be lost?

“That’s a big thing with comm, making sure we have a path with high enough bandwidth for what we need so it can actually be a redundant circuit in case another one fails,” Lines said.

“There was just enough to be able to re-route everything onto this Level 3 circuit; (once the transfer was made) that essentially (ended up) at about 85 percent capacity,” Lines said.

It took about 24 hours just to plan on paper how the cutover would be made, and then the switch – figuratively – was thrown.

“It’s not a lot of physical stuff, although it used to be at one time, but once it’s built and the path is in place, it’s a matter of our great technology, being able to log in to the devices and cutting things over.

This is an excerpt from the letter from Mark Gabriel, CEO of WAPA, that he sent to Dallon Lines, thanking Lines for the work he did to re-route critical communications from the WAPA Mt. Lemmon site during the Bighorn Fire in June.

“So the actual cutover part maybe took an hour, at tops, but it took a lot of time before that just going through the circuits, looking through our documentation, what circuits go where,
those kinds of things,” Lines said.

Thanks to a lot of detailed engineering and preparation, the cutover went off without incident and, when the threat of the fire had passed about two weeks later, the data path to the Mt. Lemmon site was restored.

Lines said there were some good “lessons learned.”

“The capacity of our (and WAPA’s) existing systems, the need for expansion, and how to expand and cutover in the future,” Lines said.

Lines was the point man on this project, but he wanted to give a shout-out to the rest of the team.

“All of our technicians are totally capable of doing the work that I did; they’re a great group of people,” Lines said.

“It was a good experience.”