8:45 a.m. was just too early to start off Monday morning with a call from the infrastructure project manager (PM). Not that it is always bad news, but a call from the PM only means my work for the rest of the week has already been pre-determined.
Please allow me to explain. I'd also like to thank you for reading this post ahead of time.
Everything was already off to a shaky start as I fumbled too long with my mic, my keyboard, the Skype application, and my bagel/egg/cheese breakfast sandwich to pick up the call in time. The voicemail played back:
"Hello, this message is for the Security Team. We have just found out that there is an IPS in our network that is nearing end of life. We need to upgrade this device right away! This will impact our online transactions! I am surprised the security team didn't send us any notice about this. We only heard about it because of a weekly newsletter we receive in our email from the vendor. Please provide us a plan to replace this device as soon as possible. I need to speak to an engineer who can explain next steps and why we were not alerted about this beforehand."
At times of panic, the client is always looking to the security professional for some peace of mind. For us, the only way to provide peace of mind to others, is with our knowledge. Customers, vendors, and business partners want peace of mind that their trust with us is solid because of the knowledge we hold in the security sector. That we have the ability to speak the language they want to hear.
Given this, it would not be fair of me to say that this project manager has no idea what they are talking about and everything they stated is not quite right or requires such panic. It isn't as big of an emergency as they are making it out to be.
But it is our job as a security person to provide them the knowledge to make informed decisions and to quell doubts. It may seem like everyone else knows about security in our lives, because that is because we are always surrounded by it. The truth is, security and IT in general are still a small sector in the global industry.
After playing back the voicemail several times, I took time to do some research about the customer's device, the current status of any licenses, and checked the vendor's website about an EOS date and the next recommended device in their upgrade path.
It was an IBM GX-5108 Intrusion Prevention System, and it was indeed facing end of support.
I also quickly checked to see all the routes on the IPS as well as the explicit whitelists so I can get a clearer picture of the work that will have to go into this upgrade if needed. Basically, for a device replacement or upgrade, we just copy the configuration on the old device, and then transfer to the new device. After that we make sure everything works by having the customer test their connections, and then close the ticket.
I listened once again to the voicemail, breaking it down into small digestible bites. I quickly calculated how I wanted to address each problem with the project manager and then called back the number.
The PM picked up on the first ring, showing their eagerness for a solution. I put on my most professional demeanor to address each of the inquiries from the project manager's voicemail:
Hello, this message is for the Security Team.
"Hello, this is Luke from the Security Team of Rymar Tech returning your call about Ticket Number X793847. Did you have a moment to discuss the next steps required to get this IPS replaced during a maintenance window?"
Everything just said is aimed to comfort and lower the stress levels of the client. Stating I'm from the Security Team assures the PM that the right person is calling back, and not a liaison or service desk representative.
A ticket number was given for tracking purposes and to get straight to the point and intention of this call. And then stating that this phone call will definitely be about getting the IPS replaced during a maintenance window without any excuses or blame games, brings it all into scope.
This is professional business language that keeps customers happy and renewing profitable contracts. You can never take an angry or complaining customer as a personal attack on yourself, it's all business. It's all about doing whatever it takes to pull someone else's money from their pockets and putting it into your own, that's business.
We have just found out that there is an IPS in our network that is nearing end of life.
"We have taken a look the IPS and can confirm from the vendor's website that it is nearing end of support."
The GX-5108 had an end of support date of March 2, 2015, which was in 5 days.
Some vendors use the term "end of life", but some also use "end of support". What's the difference?
End of life means the vendor considers the product no longer useful to be produced or effective for an environment, it has run its course. Perhaps it has been left behind by Moore's Law or better and more secure technology has come out. Either way, the vendor will no longer be taking the effort to sell or market it.
End of support (can also be known as end of service) is when the vendor will no longer provide RMAs, technical support, repairs, or upgrades.
The similarity is that both terms are encompassed in the data or product life cycle.
Software, hardware, operating systems, and most architectures in an organization are moving forward with the goal of keeping the organization functioning as it should. But at some point, some of these components outlive their usefulness - whether from the organization's point of view or a vendor. If systems are EOL/EOS, it doesn't mean they will suddenly stop working, it doesn't suddenly just turn off. If a firewall is EOL today, it doesn't mean it will automatically shut down at the stroke of midnight. If an IPS is due to be EOS in 5 days, it won't just stop working in 6 days - but it may stop downloading signature updates which are a critical part of maintaining an effective IPS.
How do you look at this from a CISSP perspective? Current updates, patches, and hotfixes are a critical control for the CISSP and is required for normal business operations with minimal downtime(s). It's the strict observation of a life cycle and a form of asset security. As a CISSP, both on the exam and in the real world, a solid management of the product life cycle or framework will ensure your organization and your customers don't approach EOL/EOS deadlines unexpectedly - this in itself can be thought of as a disaster.
Malicious actors are striving to perfect their software and tools to search the Internet for devices which have not been updated due to being EOS/EOL. Unpatched and out of support devices that are still connected and play an essential part of the network are a treasured vector of compromise.
As the security professionals in an organization, we want everything to be up-to-date and the best. But we just can't make it happen all the time because the business mission does not align with our desires - because the business mission always comes first. We want to keep our infrastructure current, but if the organization does not value the need to upgrade or change a drastic infrastructure device right away, the security professionals will have to wait.
We need to upgrade this device right away! This will impact our online transactions!
"Just wanted to note that because a device is EOL or EOS, it will not stop working. It just means that we will no longer obtain support. We assure you that you will not see an immediate impact to your production traffic. If anything, the IPS will stop downloading signatures. But if signatures are not downloaded, then the latest threats to the network cannot be prevented. So your concern is definitely valid, and we will work to get things upgraded before the EOL date."
Out of all of the PM's concerns, I knew this was the most important. The PM needed to hear that their device will not stop working just because it has reached a vendor-determined end of life/end of support cycle. The "end" doesn't actually mean the END. The vendor doesn't have a backdoor into the device to turn it off or manipulate it in any other way. The device does not have a built-in clock synchronized with the vendor's NTP server to shut down at an EOL/EOS date. Things like this are just not part of common security reality.
Change management, risk evaluation, production impact, value, maintenance windows, disaster recovery - these are the actual realities.
I am surprised the security team didn't send us any notice about this. We only heard about it because of a weekly newsletter we receive in our email from the vendor.
"We apologize for this inconvenience and understand the cause for concern as it affects your revenue stream, we completely understand this. Looking at your Statement of Work (SOW) and Service Level Agreement (SLA) with us, providing constant communication between vendor notices and your firewall is not one of the responsibilities of the Security Operations Center (SOC). This is something that is between you and our Sales team. Please see your sales manager for more information and how it can be resolved for next time."
We are "officially" not supposed to provide this per contract, and I mean...we get at least two emails per day about CVEs, vulnerabilities, or new features. Unfortunately, more often than not if its not a major release or hotfix, we don't really actively look for device EOL/EOS dates that often, there is just too much other work that takes priority, like setting up a customer's Checkpoint Mobile Access blade to use DUO 2-Factor Authentication as the global quarantine takes full effect.
In cases like this, we apologize and do what we've always done for damage control: send it to the sales team. They can calm down the customer with promises of making discounts in their contract or new perks when it comes time to renew their contract.
Upgrading the device and getting it stabilized throughout the network was our job. We considered this more important.
Please provide us a plan to replace this device as soon as possible. I need to speak to an engineer who can explain next steps.
"Absolutely, I can explain the high-level process now over the phone, and then send you and your team an email with a more detailed change plan. Here is what we plan to do from beginning to end:
Confirm to all parties the device upgrade will take place just after midnight on a weekend or holiday
Copy rules, routes, management IP address, default gateway IP, code version, and latest update file name
Confirm the arrival of the hardware at the customer's site
During maintenance window, ask customer's on-site engineer to disconnect the old hardware and plug in the new hardware
Get a console connection to the device via a remote screen share session
Configure the IPS for management connectivity
Configure or copy over the old configuration
Test outbound Internet connectivity (for the device to get its updates)
Download latest signatures and apply to the device
Observe IPS traffic and confirm with the customer that things are working as they should
Keep ticket open for 5 days and then close it out
The process to get the device replaced with a new model is within our scope as it is our job to deprovision the old device, stand up the new device in the client's network, and configure the rulebase or whitelist to match the policy on the previous device.
The SOC would be responsible for connecting to the new device via SSH, assigning IP addresses to appropriate interfaces, setting a default gateway, updating the device, and then either creating a new rulebase manually or SCP the one from the old device.
...and why we were not alerted about this beforehand.
"The SOC just doesn't have a means to provide updates on every CVE, but we always do take into account the major ones with the highest risk scores. The ones that have global impacts via zero-day attacks or SSH/RDP vulnerabilities, those are the risks we definitely alert our customers about and recommend an upgrade.
As stated previously, the sales team would be a better department to engage about vendor EOL/EOS announcements."
The Sales Team was assigned the responsibility of engaging the customer with the latest vendor updates. All risks associated from this responsibility is assigned to them.
"Risk assignment" is a term from the Official ISC2 CISSP Guide Fourth Edition. It is when the determination is made as to who is responsible for which risk. Though not common, it is put into the same list as risk acceptance, risk transfer, risk mitigation, and risk avoidance. Some quick definitions:
Risk acceptance is when management decides to take the hit of either the cost or loss for an asset, function, resource, or process. Management has calculated that the risk if fine to take. This is mostly done for low-level risks and not high or critical risk.
Risk transfer is like purchasing an insurance plan, in the form of a third-party vendor. If there is a DDOS on your organization, the vendor is contractually obligated to take care of it. Management has decided that it is cheaper and better for the long term to transfer the risk instead of taking on the burden themselves.
Risk mitigation involves taking proactive or reactive steps toward reducing risk to an acceptable level, dependent on the organization's risk appetite
Risk avoidance are techniques that try to circumvent a risk. For example, an online website decides not to collect user information such as addresses and phone numbers to avoid the risk of the information being exposed in a data breach.
Risk assignment ultimately belongs to senior management. But in reality, senior management assigns risks to directors, vice presidents, data owners, security officers, and deputy chief information security officers all the time.
The conclusion of the phone call with the project manager of their EOS IPS device ended a lot better than it started - that's one of the best outcomes you can hope for when dealing with irritated customers. Everyone is just looking for someone who knows what they are doing, and as security professionals, we carry this responsibility to ease the worries of senior management. They have a lot more to think about than a device that is end of life, they are too busy trying to make sure the business itself doesn't reach end of life.
Like sands through the hourglass, so are the Stories of a CISSP.
Thanks for reading.
CISSP Takeaway Concepts
Domain 1: Security and Risk Management
Risk assignment - An unmentioned component of the risk choices of accept, transfer, mitigate, reject, avoidance
Domain 2: Asset Security
EOL/EOS - Appropriate asset retention requires the maintenance of EOS/EOL
Domain 4: Network Security
IPS - The IBM GS-5102 is an Intrusion Prevention System and is a Layer 3 network device.
Domain 7: Security Operations
Change management - A proper change management is required for security operations. Change management exposes mistakes and minimizes unsolicited changes. This was done when mentioning Ticket Number X793847.
Click here to read other Stories of a CISSP