This incident reached near comical levels. My manager and I still laugh about it to this day. As this story involves SOC engineers and their manager, there will be a few sections on how to think like a manager for a given situation.
Thank you ahead of time for reading such a lengthy post. I hope it helps you in some way on your journey to the CISSP.
Working as a network security engineer out of a global security operations center managing hundreds of firewalls, multifactor authentication was a must. It was a must for the business. It was a must for high-value systems. It was a must for senior management to allocate the time and money to implement this added authentication measure. It was a must for our customers (and competitors) to see that we are serious about our security. Plus, we needed it to maintain our ISO 27001 certification.
We had a fairly large hedge fund customer with 35 Palo Alto firewalls throughout Singapore. Each firewall protected one of the branch locations of the client, with each branch averaging about a hundred users.
How To Think Like A Manager: There wasn't a lot of users per branch, so the client opted to use the lower cost PA-3020 model with about 250,000 maximum concurrent sessions, 2Gbps firewall throughput, and the ability to have around 3,000 VPN tunnel interfaces. In contrast, the PA-7080 has a firewall throughput of 700 Gbps and 8 million concurrent sessions.
All remote VPN users needed to access every firewall. All network security engineers (the ones who managed the firewalls) needed to access every firewall. All 35 firewalls needed to have the same security profiles for managing zero-day attacks, viruses, IPS signatures, or administrators. Given so many firewalls, and so few network security engineers, it was not cost-effective to login to each firewall to create these security policies.
How To Think Like A Manager: We had to get Panorama. Panorama is a Palo Alto manufactured device which acts like a "manager" for all the other firewalls. It's just like logging in and creating changes on any other Palo Alto firewall. Only on Panorama, you can create a security policy that can be pushed out to ALL the other 37 branch firewalls. Panorama makes it easier to deploy standardized policies, create role-based management for administrators, and have a central place to analyze, report, and investigate network traffic across multiple firewalls. It is centralized management. If we didn't have Panorama and had to manually login to all 37 firewalls to configure each one separately, that would be known as decentralized management. Below is a graphic that shows how Panorama works. You HTTPS to the device GUI (graphical user interface), then from there, SSL/TLS is used to communicate and send policy changes to all other branch firewalls located anywhere in the world.
Our manager wanted to make sure that multifactor authentication is to take place if any of security engineers HTTPS to the administrative console of Panorama (you know, where you can make changes to ALL firewalls, high-security risk). For our multifactor solution, we went with a company called DUO - they specialize in multifactor technology.
In true security fashion, we understood (and also there was a company policy) that the same people who are supposed to be getting administrative superuser access to a firewall, are NOT to also create the access for themselves - classic separation of duties. As in, it's not a good look for a security company when those who need access to a system are the ones who are creating the access. There had to be separation of duties, a four-eyes process, dual-approval, whatever you want to call it. For us, to make any sort of change on the firewall, a change request had to be created for another IT department to fulfill it. As we were based in the USA, and the firewalls were located in Singapore, our manager created a ticket for the Asia-Pacific team. Details of the ticket are below: Name: Manager Subject: Create Multifactor Authentication Access Short Description: In lieu of security management best practices and management requirements, please create a DUO-factor authentication process for access to the Panorama firewall as well as local logins for all branch firewalls for hedge fund client.
Long Description: For your convenience, SOC engineers have provided the technical procedures needed to be performed on the firewall below:
On DUO Dashboard
Create and configure Palo Altos on DUO Dashboard
Export certificate from DUO
On Palo Alto Firewall
Import certificate into Palo Alto
Create Certificate Profile for DUO
Add CA certificates
Create General Certificate
Create SSL/TLS Service Profile
Create Multifactor Authentication Server Profile
Create local user account
Add local user to Authentication Profile
Enable and configure Captive Portal
Go to VPN interface, allow "Response Pages"
Create Authentication Enforcement profile
Create Authentication Policy Rule
Commit Policy
Once thing we didn't separate however, was sharing knowledge. We knew our team could best configure the Panorama because we worked with it every day. But our team in Singapore may not have the full knowledge even though they were the ones who had to configure it, so we provided them with as much technical steps and information as possible. We all worked for the same company, we all wanted to see each other do our jobs to the best. It's part of being a security professional.
Our manager submitted the change request, alerted the team that we will soon (24 hours) be contacted by engineers in Singapore to test our multifactor access using the DUO app on our work phone, and then promptly went on a well-deserved vacation to Japan.
There's always a sense of nervousness when the manager takes a vacation. He is the one who can engage and tell other departments what to do in an official and authoritative capacity. He is the one who can talk to someone in management about speeding up a certain process. And most importantly, he is the one who has to listen to frustrated customers complaints and threats when things got real. And things were about to get very real very fast!
Two days after the manager went on vacation, we still had not heard from the Singapore team about our access to the firewalls. As a change was in the works, we also knew not to touch or login to the Palo Alto firewalls during that time and alerted our customers of a 24-hour change freeze. Our biggest fear is a customer that has some sort of giant DDOS, breach, or security policy request and needs to make immediate firewall changes. Then as with all things in security, hope for the best but prepare for the worst. So, of course the worst happened.
Our ticketing system logged a Severity 1 change from our hedge fund customer's network team at 10:30am Eastern Standard Time. 10 out of their 37 branch firewalls were completely offline. All network traffic inbound was being dropped. They wanted us to investigate what was going on and see if running this command would help on each firewall:
::clear session all filter destination (server IP)::
This command would clear all sessions initiated to the organization's server. There were incidents in the past where a vast amount of UDP traffic to an internal server protected by the firewall created stuck sessions on the firewall. Sometimes these stuck sessions were a result of the firewall unable to handle the maximum concurrent connections. It happened at least 4 times a year.
How To Think Like A Manager: If faced with the same problem over and over again, something has to be done. If the PA-3020s are not able to consistently handle the incoming traffic, then a risk analysis has to be done to see if money and time should be spent on creating a process to upgrade the firewall hardware to a stronger and more capable version.
I suddenly felt dehydrated. Palms sweaty. If I were eating mom's spaghetti, there would be vomit on my sweater already. I slowly, yet hastily, went to the URL of the Panorama firewall to login, fully knowing that we were currently undergoing a change being performed by another team which could very well undermine our ability to login. For one, we didn't receive instructions on our new multifactor process. And two, if they started their work and hadn't finished yet, we would be unable to login at this very crucial time. I pressed enter on the keyboard and was presented with the Panorama administrator login screen:
Source: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClS2CAK
So far so good. If I entered the administrator username and password and logged in successfully, that means the Singapore team had not begun changing the login process to one that is integrated with DUO. If after entering a password I was prompted to accept the DUO Push notification sent to my phone, we were F****D! I'll mention again, none of the network security engineers had received their multifactor confirmation via email, text, or phone call. We had the DUO app installed on our phones, but had not gone through the enrollment process yet.
Entered the username and password and clicked enter. Next message was:
"Secondary Authentication in progress please wait." - DAMMIT!
This meant the firewall was waiting for ME to press "Accept" on my DUO push app. Which I never got on my phone.
I immediately called the Singapore team, after fumbling through how to exactly dial-out with an international number. After 5 rings a member of the Singapore help desk picked up.
Singapore skyline.
"Singapore SOC how may I help you?"
"Hi yes, this is Luke calling from US SOC operations referring to change request K-29837. We are currently unable to login to the firewalls via multifactor authentication per the change request. Could I get an update? We have a customer who has an urgent change request for their firewall."
"Uhhh yes, could you please spell your name and the change request again?" The help desk engineer was nervous. It took five more minutes to overcome the language barrier and get all the preliminary information out of the way. Which is FINE, it was the proper thing to do.
"Okay, so can I get connected to the engineer who is implementing this multifactor change for us?"
"Um sir, it is currently after-hours Singapore time and the engineer has gone home."
My blood-pressure levels instantly elevate.
"Uhhh...is there anyone else who can do the change?" "I'm sorry no sir."
I decompressed my anxiety and sense of urgency onto the Singapore team. "Listen, the customer is undergoing critical issue where their firewalls are down. We need access to these firewalls right now.".
A calm and courteous demeanor always wins, but sometimes a little edge in the tone helps to convey the importance. Too many times we forget that everyone is just trying to do their job, and that our own problem isn't the single most important issue at all times. The thing is, with our manager on vacation IN JAPAN, we didn't have an immediate authority figure to escalate this issue internally. And I wasn't about to call the Director of IT on this kind of issue. It would be like someone trying to call Sundar Pichai, the CEO of Google, because their Gmail wasn't working.
Eventually we convinced the help desk engineer to call the Singapore engineer assigned to this ticket on an emergency basis. I had to put my senior security engineers on speakerphone so we all could <ahem> convince the engineer to give us the engineer's number. An issue we would apologize for later, but now was not the time for pleasantries.
After making it past that hurdle, we came to know an even more severe one: since the change request was raised by our manager, the Singapore engineer ONLY configured multifactor authentication access for the manager, not anyone else in the SOC. That is why none of us got enrollment, but our manager did, and who also happened to be on vacation IN JAPAN.
At this point the hedge fund customer was raging. It was in the middle of business hours, and 10 of their firewalls were down. We couldn't login to the firewalls because of our inability to get past the multifactor authentication process. Only our manager could. We were left with no choice. We had to call our manager on his vacation IN JAPAN, where it was currently 30 minutes after midnight! Somehow, I was voted to be the one who calls. GREAT.
Manager picks up. His first words "Wow, something must really be wrong Luke, what broke?" A man spoken from experience.
"Hey man, SO sorry to call you on your vacation, but we have a situation with our hedge fund customer..."
"Ugh..." The manager already knew how agitated this certain client got with even a small firewall issue, much less multiple firewall outages.
We explained the situation.
"So what do you want me to do?" asked the manager.
I was still a junior security engineer with about 2 years of experience, and not a CISSP yet. My response was "Well, I'm going to login with the administrator credentials, and then it asks for the DUO Push. Can you let me login with your credentials while screen sharing with me and then get the push on your phone and let me get access to the firewall so I can enter these commands and push out some policies?"
How To Think Like A Manager: I was asking the manager to open up his laptop, join my screen share application, login in with his credentials while on my computer and connected via HTTPS to the firewall, proceed past the DUO Push on his phone, and login to the firewall dashboard. This adds an extra layer of complexity, not to mention screen sharing and entering passwords wasn't a very secure process.
"Well there is a slight problem Luke. I didn't bring my work phone, because I'M ON VACATION." My manager wasn't yelling, but actually sounded humored. He was an easy-going guy even in the midst of turmoil. A sign of a good manager who can maintain composure. "My work phone is back home. The only person that can really get to it is my mother-in-law who is taking care of the kids while my wife and I are here...ON VACATION LUKE!" Again, he was kidding, but it stung just a little bit.
I stammered "Uhhh...soo...well, we could - "
Manager cut me off. "Let's do this." He knew it was his turn to step in. "Just get on the phone with the customer right now and conference me in. I have my work laptop so I'll login to each firewall and run the command and install the policies. It hasn't been that long since I worked on some Palo Altos. It's pretty much just like a video game anyway, not as complex as those Cisco ASAs."
"What about your DUO Push?"
"Well, my mother-in-law is about to get a lesson in multifactor authentication. I'm going to ask her to access my work phone and "Accept" the DUO Push after each time I enter my password on the Palo GUI. That's just how it's going to have to be. I'm going to tell her it's your fault Luke."
Again, a joke, but it still stung. But really, my manager should have said in the ticket to implement the change for all security engineers! But again, nobody was really at fault here. It was just an instance where everyone did their job to full effect, which somehow created multifactor confusion.
So in the end, my team, my manager, and the customer all joined a conference call. My manager logged into each firewall (after his mother-in-law successfully was instructed to "Accept" the DUO Push on his work phone), ran the command to clear stuck sessions, implement some policies to restrict traffic on each branch firewall, and then add one another policy on the Panorama for all the firewalls.
The customer was satisfied for the time being and had no idea about all the effort that went on in the background, as they shouldn't! We have to portray a constant image of professionalism and ability. The customer has their own problems, they don't need to hear about ours. Needless to say, my stress levels went down after it was all over. My fellow engineers took a long-deserved coffee break and laughed about what just happened.
CISSP Take-Away Concepts
Summary and Conclusion
We needed to implement two-factor authentication for our customer firewalls. A username and password for something you know, and a DUO Push notification for something you have.
Manager submitted a change request for another team in Singapore to implement the change to enable multifactor authentication within 24 hours. Manager then went on vacation.
None of the SOC engineers heard back from the Singapore team.
Customer started to have an outage issue within this time frame.
SOC engineers were unable to login to firewalls due to implementation of multifactor authentication.
After calling Singapore engineer after-hours, it was found only the manager had his multifactor authentication access configured since he was the one who opened the change request.
Called manager who was in vacation in Japan.
Manager winded up logging in with his work computer to the firewalls and troubleshooting to resolution.
Customer issue was resolved.
Internal process began of making sure change requests either specify to complete multifactor authentication process for the entire team, or each individual security engineer will have to submit their own change request to Singapore.
Domain 1: Security and Risk Management
ISO 27001 - Understanding the ISMS requirements of the ISO 27001 helps to understand a better picture of how the internal security of an organization can be managed.
Although not mentioned, and not in a malicious way, there was some social engineering which had to take place in order for the Singapore help desk engineer to escalate and call the Singapore SOC engineer after hours on his phone. We later notified all supervisors of what we did and made sure it was understood it was done only because it was an emergency situation
Domain 4: Network Security
Dealt with 35 firewalls and a centralized management system known as Panorama
Configuration of multifactor DUO authentication profile on firewalls
Command to clear stuck sessions on the firewall
Domain 5: Identity and Access Management
Single/Multifactor Authentication (MFA) - One of the strongest authentication measures that can be taken for access and identity management.
Domain 6: Security Assessment and Testing
While there was a signal that we were going to test our multifactor access to the firewalls, we never even received the enrollment email. But if we had, testing would have taken place immediately, instead of during a live customer issue.
Domain 7: Security Operations
Separation of duties - The engineers who needed multifactor authentication access to firewalls were also not the same engineers who would be configuring and implementing their own access
Change management - A proper change management cycle was implemented both from the hedge fund customer and within internal engineers. Unfortunately, the change request was followed with such strictness that it actually caused an access issue for the SOC engineers. That's part of being in the security industry.
Click here to read other Stories of a CISSP