Deep Dive: 10 Major Lessons Learned From the Okta, Cloudflare, 1Password Hack

This post is too long for Gmail, to read online, click here or the link Read Online link above. Make sure to catch the end.

Introduction

The dust has FINALLY settled on the recent Okta hack, and we have a ton more information as Okta has updated their incident response report (see old version here).

This is a deep dive not just into all the things that went “wrong” but more importantly, all the things you and your organization can do to prevent such an attack. I think if we can’t learn from our and others' mistakes, then we’re just being lazy. 

I also use quotes for the word “wrong” as I acknowledge that information security is a tough job, however that being said there are definitely expectations out there regarding information security hygiene, upkeep, and preventative medicine. This becomes doubly so when you have been hit with a breach/compromise already.

The format of this article is divided into sections that cover what the “bad” was and include a remediation section at the end of each. They are not in any particular order of priority and all are different layers to the cake. Often attacks are successful because they opportunistically exploit multiple vulnerabilities that even sometimes seem unrelated.

Although I could have combined many of the below items into larger topics, I felt it was important to granularly break each part down as each item has its own preventative medicine.

I must commend Okta, Cloudflare, and 1Password for sharing detailed incident response reports publicly. The only way we can improve is by being transparent.

This post is over 3,000 words (not including this sentence 😅), please share with your network to get more Deep Dives like this.

Disclaimer: The below advice is meant for companies to implement in their own information security programs and is in no way an accusation of wrongdoing or neglect of Okta or any affiliated party.

Table of Contents

• Introduction

• 1. Static Usernames And Passwords Are Bad

  • Okta Incident Details

  • Lessons Learned - Avoid Static Passwords

• 2. Old Service Accounts Will Haunt You

  • What went wrong in the Okta hack

  • Lessons Learned - Audit & Delete Old Service Accou …

• 3. Storing Passwords In Chrome is Super BAD

  • Lessons Learned - Don’t Save Passwords, Insecurely

• 4. Beware Of Personal Employee Use On Corp Devices

  • What went wrong in the Okta hack - Shadow IT

  • Lessons Learned - Lockdown Company managed devices

• 5. Just Because You Have Logs, Doesn't Mean You ha …

  • Lessons Learned - Your Logs May Not Have The Full …

• 6. Admin Logins Need Protections Too!

  • Lessons Learned - Universally Apply Protections To …

• 7. Mistakes are a waste if you don’t learn from th …

• Lesson Learned - Learn From Your Mistakes

• 8. Threat Detection and Response Is Fundamental

  • Lessons Learned - Threat Detection and Response Mu …

• 9. Feedback From Customers Is Crucial (Security or …

  • Lessons Learned - Reduce Customer Friction To Secu …

• 10. Protect Your Customer’s Data As Your Own

  • Lesson Learned - Protect Customer Data

• Summary

1. Static Usernames And Passwords Are Bad

Yes, I’m willing to make a blanket statement like this. It’s 2023 and we’re still using usernames and passwords for service accounts?

Okta Incident Details

Okta had a service account with a username/password (shared?) for accessing customer information. Because it was a service account, it was likely shared among other employees. Not only that, but due to laziness, and seemingly not wanting to use the company’s password manager, the employee stored the password in their personal chrome browser. 

Lessons Learned - Avoid Static Passwords

• Do not use usernames/passwords whenever possible. Utilize OAuth or temporary access methods whenever possible.

• If a username/password cannot be avoided, issue separate logins that are stored in a password vault (where access can be tracked).

• Automatically change service account passwords daily/weekly/monthly based on your threat profile. This can easily be automated using 1Password API’s for example.

• Ensure service account passwords/keys/secrets are only manually accessible to a few individuals. This reduces your threat footprint.

2. Old Service Accounts Will Haunt You

You know that service that the founder used in their MVP years ago and is still in production use today? 

Or maybe that trial account an engineer used to test out a new feature, but then became part of production? 

The one with the same password in place from inception that numerous contractors had access to over the years?

Yeah that one. You know what I’m talking about. I see you.

What went wrong in the Okta hack

As mentioned above a service account was in use to provide some sort of functionality for the customer support team. Well, as part of their remediation, the service account was deleted. This indicates to me that either:

1. The service account was not necessary or 

2. That they found a more elegant solution to the service account like OAuth, etc

Lessons Learned - Audit & Delete Old Service Accounts

• Review service accounts in place regularly. Especially critical or high risk service accounts that either:

  • Have a lot of access / privileges to systems

  • Are widely shared

  • Have not had their credentials rotated in the last year

• Delete unnecessary/unused service accounts

• Transition service accounts to OAuth, Certificate, or non-static password model

• Break large service accounts with a lot of permissions into smaller ones with reduced privileges. This will also reduce your attack surface area.

3. Storing Passwords In Chrome is Super BAD

Did you know that passwords stored in Chrome are stored in cleartext? That means passwords are easily retrievable from the system if someone had full access to the system (physically or remotely).

Before anyone with access to your browser could automatically see the passwords stored there, however, when testing this out personally, I did notice Chrome asked me to authenticate to the system again before revealing the password. That’s a step in the right direction I guess.

In any case, if someone has access to your unencrypted drive, they can steal (borrow?) your password stored on disk.

However, YMMV as I’m not generally logged in as an administrator on my devices.

In this case, the employee had stored the username and password to the service account in the personal profile of their browser! Yikes!

Since the personal profile of the user was in use, Okta did not have access to govern that profile (more on that next). It also seems that no password manager was in use on this profile, otherwise it would the ability to save a password in Chrome would be disabled by default (see screenshot below).

Lessons Learned - Don’t Save Passwords, Insecurely

• Disable the offering and storage of passwords in the browser

  • TIP: Many password managers will automatically disable this setting upon installation (see below)

• Educate users often on the handling of sensitive information such as passwords, client data, financial information, and other typically restricted information

• Educate users on the use of private resources for work

4. Beware Of Personal Employee Use On Corp Devices

It’s almost guaranteed that users will be checking their personal email during work hours. If they can, they’ll be using whichever device they’re on during the day, and guess what, that’s probably their work machine. Why would they go through the hassle otherwise. People will always follow the path of least resistance.

That means a user can and will:

• Login to the personal email

• Create multiple chrome profiles

• Let their kids use their computers (yes, this happens)

• Use incognito mode if they can

• Install extensions

• Install random software downloaded from the internet

• Surf to porn or malicious websites (accidently or maliciously)

Each one of those can be a vector for an attacker to access your and associated systems including customer systems.

What went wrong in the Okta hack - Shadow IT

In the Okta hack, users were able to save passwords in chrome, but not the Okta managed chrome profile, their own chrome profiles. So they were able to create separate chrome profiles on their Okta managed devices, subverting any controls and password extensions that would have been enabled in their Okta profile.

This is part of a small segment called Shadow IT.

Shadow IT refers to the use of technology or software by employees within an organization without the knowledge or approval of the IT department. It often happens when employees find their own solutions to meet their needs, such as using personal devices or cloud services to store and share files. While it may seem convenient, Shadow IT can pose security risks and create challenges for the IT department in managing and supporting these unauthorized systems.

Lessons Learned - Lockdown Company managed devices

On a company managed device, you will want to lock down controls as much as possible based on your threat model/risk, and tools available to you. In this case, Device Management tools are not cheap nor easy to manage. However, if you already have them, you’re probably not taking full advantage of their capabilities.

• Restrict the use of personal accounts on highly sensitive devices and users

• Provide additional training on the use of work devices for personal use and browsing

• Limit the installation and use of apps, extensions, and other unapproved software

• Limit  which browsers can be used on the device and configured browsers accordingly

• Limit access to corporate systems from corporate devices only

• Deploy machine and user certificates on machines and utilize BOTH for authentication 

5. Just Because You Have Logs, Doesn't Mean You have Have The Right Data

According to Okta:

Context is everything. Filters and queries may not always be accurate the first time around, especially when first triaging an incident. Too narrow of a search, and you may miss some additional details. Too broad, and well you’re going to spend extra time searching for clues.

Of course if you don’t have logs at all, well it’s a non-starter! Get them logs in!

Lessons Learned - Your Logs May Not Have The Full Picture

• When capturing logs for analysis, ensure your filters are not too narrow where they may miss some IOC (Indicators of Compromise)

6. Admin Logins Need Protections Too!

Protections to a system need to apply all around, not just to consumer facing parts.

Well, it turns out that administrative logins were not protected, or had the option to be, in the same way as employee logins. I remember running into this limitation years ago, and was befuddled.

One of the most important features of using SSO (imho), context aware logins, was not a feature that applied to Okta Admin logins! 

Context Aware login is the review of additional contextual information (location, user-agent, IP, time, etc) when a login attempt is made. This information is compared to historical and reputational information made available to the system and a score is made. Too risky, then additional information is required for authentication/authorization or flat out blocked as well.

I love this feature and it's an important step towards modern authentication that I’ve written here and spoken about numerous times.

This is a fundamental aspect of Zero Trust Identity Management.

According to Okta, this feature has been rolled out and can be enabled in early access. (Not on by default AFAIK).

Lessons Learned - Universally Apply Protections To All Parts Of The System

• Apply security controls to all parts of an application ecosystem, including Administrative logins and portals

7. Mistakes are a waste if you don’t learn from them

Customers and the community can be a little forgiving at times, but they expect you to learn from your mistakes and grow. Just like when you have kids, you can let something go the first time or two, but don’t make it a habit.

In this case, this is the second time that Okta:

• Was notified of an issue by the customer first

• Had its customer support systems compromised

• Dealt with an incident that affected many, many customers

Lesson Learned - Learn From Your Mistakes

• Learn From Your Mistakes Quickly

• Update and Test Your IR Plans Frequently

8. Threat Detection and Response Is Fundamental

That is a bit of an understatement when it comes to Okta to tell you the truth.

A service provider with the size and importance of Okta has a lot going on. 

Here are some things to consider:

• Okta systems are the front door to millions of employee and customer logins worldwide

• Okta has a huge target on it for obvious reasons

• Okta has been attacked before and is being attacked nonstop

• Okta is a public company with investors trusting it

Having top notch incident response execution is paramount for any company with so much at stake.

If your company has or manages a wealth of highly sensitive information, whether it’s authentication tokens or customer SSNs and bank account numbers, ensure you are on top of things.

Lessons Learned - Threat Detection and Response Must Be Adequately Size To Fit Your Threat Model

• Ensure you have adequate detection and response capabilities and mechanisms in place analog to your threat model and incident history

• Have your IR plan updated often based on your threat profile

• Build excellent detection systems and conduct red team exercises often to keep your blue teams on their toes

• Never be complacent

9. Feedback From Customers Is Crucial (Security or Not)

Even without my cybersecurity hat (I’m wearing the product management hat right now) I know that customer feedback is crucial and important. 

• How much feedback was given in the past year from the last incident?

• Did that feedback make it to product management?

• What is the feedback loop between CS, Engineering, and Product?

These are basic questions every startup needs to consider.

Things happen. However, when a customer finds out about an incident before you do, that can be embarrassing. However, when that customer is also in the security industry as you, then you best listen closely.

At the end of the day, if an event cannot or does not reach your Incident Response team in a timely manner (or at all), then this is a failure in your Incident Response plan and awareness of it.

Anyone in the company should:

1. Know where to go to report incidents

2. Have an easy way to do so

The less friction, the better. 

So a helpdesk person or even a janitor needs to know where to report a security issue and that issue needs to have a reliable path to the CISO, otherwise, if it fails along the way, it’s dead in the water.

In addition, the person reporting the matter needs to feel SAFE in doing so. 

What does this mean?

Well, sometimes we’re not sure if a security issue is a real threat or not. In the event the matter is not a cause for concern, the person reporting it needs to have the assurance that they will not be reprimanded or face any negative consequences (emotional or otherwise) for doing so. 

This is another aspect of reducing friction.

Lessons Learned - Reduce Customer Friction To Security

• Ensure all employees and contractors know where to go to report a security incident and how to do so

• Ensure anyone reporting an incidents feels SAFE (emotionally, financially, or physically) to do so

• Update your IR plan frequently

• Reduce any friction for reporting security incidents

10. Protect Your Customer’s Data As Your Own

We often say in security, “Security is everyone’s responsibility”. 

Sure, however when you are the support person, you are “in charge” per se from a power perspective and the customer will generally listen to your guidance, and assumes you have their best interest in mind.

An uninformed customer may not know the implications of the actions you are asking them to take.

So in the case of HAR files, which literally contain live sessions cookies and authentication material, there is a bit of responsibility on the customer support person to:

1. Not handle/request data unless absolutely necessary

2. Inform the customers the full implications of their actions

3. Assist the customer in sanitizing any data as a result of their requested action

I’m talking about HAR files. These files contain a wealth of information and are often used for troubleshooting issues with SaaS based applications.

Take a look at Zendesk’s support article for example. They seem to have updated their page recently to include a warning regarding HAR files. Unfortunately I couldn’t find an earlier version in the Wayback machine.

Okta has updated their support page a few times regarding the wording/warning of HAR files:

Snapshot from Feb 7, 2023

Snapshot from Oct 23, 2023

Snapshot from today, Nov 19, 2023

Interesting discussion on the topic

Lesson Learned - Protect Customer Data

• Do not request sensitive information from a customer unless absolutely necessary and there is no other workaround

• Clearly inform the customer if there are any security considerations or implications of your requested actions and receive acknowledgement accordingly

• Guide the customer to reduce any risk to their data as a result

• Create technical mechanisms such, as an impersonation feature, to reduce the need for mechanisms such as the HAR file

• Train your CS teams on the implications of sensitive files and ensure processes are followed accordingly

• Ensure any file with highly sensitive information is handled and secured to the highest level:

  • Encrypted out of band file transfer (not email)

  • Extra encryption with either Customer Managed Keys or unique, temporary, and rotating keys to access files

    • Basically, these files should not be simply stored in a ticketing system or any file system with general access. Extra decryption mechanisms should be required to access these types of files. This will reduce your chances of illegitimate access.

  • Delete files within as short of a time frame as possible based on the sensitivity of information

  • Create mechanisms to extract the necessary information and discard the original information

• Tools for safe HAR file handling:

  • HARmor 

  • Introducing HAR Sanitizer: secure HAR sharing (Cloudflare)

  • Note: Kinda ironic to upload sensitive files with authentication credentials to the cloud to be sanitized 🤔

Summary

As you can see, security can be a very complex system. There is never one thing that went wrong, nor conversely never one thing that will fix your security.

Repeat after me:

Security is a process. Security is a journey.

Below is a link to just the summary of lessons with a downloadable PDF as well.