VMWare Infrastructure Actively Exploited to Compromise Organizations

Log4j vulnerability unpatched

CISA, the Cybersecurity and Infrastructure Security Agency, has issued an emergency directive highlighting an escalation of successful attacks against commonly deployed enterprise components of VMWare virtual infrastructure. The directive points to an escalation of successful attack against a series of VMWare vulnerabilities that are exploited independently, or in combination, to fully compromise VMWare infrastructure in these organizations. While VMWare has issued patches for these vulnerabilities, attackers have quickly reverse engineered them to develop and weaponize exploits now appearing in the wild.

The attacks highlighted require network access, but successful attackers have utilized 3rd party network access and web exposed servers to compromise vulnerable VMWare components and gain full access.

 Read the full bulletin

Millions of Log4j vulnerable systems still unpatched

Log4j vulnerability unpatched

A recent survey by Qualys and published in SC Magazine suggests that after over 3 months, roughly 1 in 3 devices and installations that were affected by the Log4j vulnerability are still unpatched. This number amounts to roughly 22 million vulnerable application installations — and it should be noted that these are just the devices that are readily accessible from the Internet.

Log4j reached critical status towards the end of 2021 when it was discovered that a feature its platform could allow an unauthenticated attacker to take complete control over a remote system. The vulnerability was classified in CVE-2021-44228, and has been extensively discussed in cyber security as well as in a published flash with guidance from the government’s cyber security agency, CISA, who published guidance.

 Read the full bulletin

Vulnerability Management: If you think it’s about missing patches, you’re missing the point

Unlike technology as a whole which continues to advance at an astoundingly rapid rate, vulnerability management is one area of cybersecurity that is harmfully stuck in the past.

In the 90s the primary sources for vulnerability information were mailings lists like Bugtag and FD. With the volume in these public lists security professionals struggled to scan, identify and patch their systems – but the long gaps between exploit releases gave security teams the luxury of time. The early 2000s brought remote vulnerability scanning engines that were managed, woefully, in spreadsheets. In that time we had learned two things: first that there was more to vulnerability management than missing patches, and second that identifying missing patches and unpatched systems was the easy part. But over a decade later in the 2010s spreadsheets were still the predominant vulnerability management tool. And now here we are in 2021, and what is the state of the art for vulnerability management? If you guessed vulnerability scanning and spreadsheets – you’re unfortunately correct. Nearly 30 years and countless technical advancements later, and we’re still basically on clay tablets and chisels.

Today’s Vulnerability Landscape

The time between a software or system vulnerability being identified and a readily available, weaponized, exploit in the wild has shrunk to a sliver of time. Security teams no longer have the luxury of time to try to scan, find systems, manually triage, prioritize and then push patches. Never mind that patching is only a part of overall vulnerability management.

In spite of all the latest technology now available to us, the lessons we’ve learned over the three decades, and the renewed sense of urgency brought on by the COVID19 pandemic, the whole vulnerability management process is a train wreck. IT and cybersecurity professionals are still primarily focusing on missed patches and relying on spreadsheets for reporting and prioritizing vulnerabilities – and no one is being held accountable. Combine that with the work-from-home environment, and you have an avalanche of new vulnerabilities daily, more tools than you can effectively manage, insufficient resources and skills to dig yourself out – and all of it is driving increased business risk.

So, let’s get back to fundamentals, and attempt to understand the problem we’re trying to collectively solve for. First, a vulnerability can be any weakness in your infrastructure that could compromise business operations. And they can come from anywhere. Yes, missing patches are a major factor, but so are misconfigurations, penetration test results and bug bounty programs (if you have them). Vulnerability scanners are highly ineffective in dealing with anything outside missing patches, so they’re not your solution. Finding vulnerabilities is relatively easy, but what do you do with them once they’re uncovered, and when they’re so out of date that patching isn’t an option (also known as technical debt)?

Many IT leaders think the answer lies in purchasing state-of-the-art technology. No matter what the buzzwords, technology doesn’t solve a human problem. Most companies have a volume problem, as well as a culture problem – but we’ll address that in another blog. When you have potentially 10,000 or more identified vulnerabilities, prioritization becomes a pretty tough obstacle to overcome. Your fancy tech is only as good as the analyst who operates it, no matter how much magical “AI” secret sauce is in their marketing fluff. Don’t get me wrong, a strong and capable technology platform is crucial – but that comes after you’ve addressed the human-sized problem in the equation.

What you need is a Programmatic Reduction of Risk

Unfortunately, a significant portion of technology owned by organizations is outdated or worse, no longer being supported by your vendors. This is partly due to tight budgets, partially feature dependence and partially an if-its-not-broken-don’t-fix-it mentality. What may seem like a minor issue today can result in a massive breach tomorrow. While the business is asking for agility from its technology, CIOs everywhere are facing huge technical debt. And the longer it goes on, the more expensive it becomes to fix or replace. How do you extinguish all of these burning fires?

The CIO essentially has to declare technical bankruptcy. Take inventory, acknowledge there are problems, identify them and create a plan to fix them. Communication and accountability among business leaders and IT professionals is the key to implementing an effective solution. In many cases, this includes admitting that the organization lacks the in-house expertise to solve the problems.

The most forward-looking CIOs turn to a trusted provider for help. Lightstream is one such partner with the expertise to move your organization out of this quagmire, and our Rapid Risk Profile is often the best place to start your risk management assessment. This approach helps us to understand your biggest systemic risk so we can work together to create an informed path forward that aligns to your business goals and financial situation. The first step is easy, no-friction, and involves virtually no invasive technology. What we assess and identify are hallmark people, process, and program categories to understand your organizational and program maturity. Whatever stage your organization is at in its journey and program development, we can provide understanding and high-level guidance.

The immediate next step is to take a consultative and technical deep-dive, to understand not only what your organization does in terms of vulnerability management, but how it does it. We create your baseline, and provide a gap-assessment against industry-driven baselines. Lastly, we develop a bespoke roadmap that involves both short-term tactical remediation strategy to prevent catastrophic business disruption, and long-term program development to aid your business into effectively managing technical debt and vulnerabilities across the business. Lightstream’s suite of packaged services provides peace of mind, technical as well as program capabilities, and continuous evolution in your vulnerability management program. The key to effectively managing vulnerabilities is to go beyond patching and implement a lifecycle approach for identification, triage, mitigation and reporting.
Stop buying into the misconception that vulnerability management is about scanning and patching. It’s time to acknowledge the magnitude of the problem and the risks it’s creating for your business. Contact Lightstream today to find out how we can help you establish an effective vulnerability management program – protecting and future-proofing your organization while creating a culture of accountability.

Overcoming Security Challenges at Mid-Market Organizations

When it comes to cybersecurity, mid-market organizations are uniquely challenged. They encounter many of the same issues that large enterprises do, but in most cases are forced to try to remedy them without the substantial budgets and IT departments that major corporations have.

According to first quarter 2021 Middle Market Business Index data from advisory firm RSM, 28% of middle market executives reported a data breach at their company in the last year, the highest level since RSM began tracking data in 2015 and a significant increase from 18% in 2019. Larger middle market organizations were most at risk, with 42% of executives at those companies reporting a breach, compared to 16% at smaller counterparts.

As attacks and adversaries ramp up, mid-market businesses find themselves in a battle to sustain their growth and security budget. Facing a shortage of security talent without the deep pockets to pay the high salaries demanded by top-level security professionals, these positions often go unfilled and the responsibilities associated with them are assigned to other staff members who are already overworked and wearing too many hats. In other cases, these positions get filled by less-qualified candidates. Either way, the organization is at risk of increased vulnerability to security breaches.

Another challenge is the siloed nature of security tools. The effectiveness of an organization’s security program on the operational side is often inversely proportionate to how many times IT professionals have to switch screens to figure out what’s going on. If you do not have a highly scalable and optimized security infrastructure backed by a qualified team to analyze threats in near real-time and respond, it puts you at significant risk and disadvantage – no matter how much technology you buy.

The pandemic and resulting economic downturn of 2020 exacerbated these issues. When countless companies throughout the world were forced to shut down, mid-market organizations with outdated infrastructure were unable to scale down in response. They overpaid for technology, licenses and features they no longer needed at a time when they couldn’t afford to waste valuable budget dollars. Moreover, with the majority of staff forced to work from home, users were accessing the network and data from everywhere, making the challenges of protecting assets even greater and requiring an increasing number of resources.

Outsourcing and the evolution of security service providers

What is a security threatened, budget constrained, short-staffed mid-market company to do? Many organizations are led to the decision to fully or partially outsource the day-to-day operations of their security program. For the past 20 years, this meant hiring a Managed Security Service Provider (MSSP). While doing so took some of the security management responsibilities off of the in-house IT team, it was not without its flaws. In order to remain profitable, most MSSPs rigidly deliver “cookie cutter” solutions. They are paid to monitor a dashboard and if there are any concerns, they simply alert the company to the issue. It is a best-effort model wherein the customer still remains very much engaged – thus delivering little real value.

Today, the MSSP is slowly being replaced by Security as a Service (SECaaS). With this new and improved “all-inclusive” delivery model, the service provider integrates their security services into the corporate infrastructure on a subscription basis. In most cases this is more cost effective than it would be for the midsized organization to manage its own security, particularly when total cost of ownership is considered. With SECaaS, the organization no longer has to worry about purchasing and managing infrastructure, tools and licenses. Instead, it is protected by a provider that partners directly to provide everything needed – tools, expertise, staffing – to deliver a next generation security services in a pay-as-you-go model.

This new model delivers immediate value by completely lifting the day-to-day security management off the IT department. And when the business needs to scale up or down quickly in response to seasonal or other shifts in demand, they can simply add to or remove services and scale up or down as business requires while delivering desired security outcomes.

In today’s world where the unpredictability of life comes at you fast, organizations must have the mindset that “IT comes at you faster.” IT must have the ability to immediately respond, support and thrive under whatever circumstances the organization is experiencing. It is imperative that organizations adopt a zero trust model, which trusts no one and requires continuous verification and multi-factor authentication for anyone trying to access the network. Mid-market companies that partner with the right SECaaS provider can gain peace of mind by relying on their vendor to update settings based on current threats.

For advice on how your midsized business can overcome its security challenges and adopt a zero trust model, contact the experts at Lightstream. As a networking and cloud company that integrates security into everything we build, we specialize in 24x7x365 network monitoring, detecting, protecting, analyzing and remediating security issues. We’ll help you manage costs effectively, reduce complexity and improve the efficiency and efficacy of your data center, network and cloud security.

The Maturity of Vulnerability Management Matters

If you work in cybersecurity at a typical mid-market company, you probably cringe when you hear the term “vulnerability management.” Let me see if I can guess how the workflow goes for you:

  1. Someone uses a scanning tool to scan as many assets across your network as you know about.
  2. The output gets exported to a spreadsheet.
  3. The spreadsheet is sorted by “criticality.”
  4. Various department or organization heads receive line-items they are responsible for patching with little context on why.
  5. You wait a week or so, then repeat the process.

How close did I get?

The various pieces of your cybersecurity strategy are programs in themselves, which means we can measure them for maturity. The problem is that mid-market companies seldom have the time or resources, not to mention the capital, to execute a full-scale program. That approach ultimately leads to a “just-do-something” execution of cybersecurity, and things get complicated.

Some hallmarks help me determine what level of maturity an organization has attained. Here are just a few:

  1. Strategy: How an organization thinks about vulnerability management and whether it’s truly managing vulnerabilities or simply trying to play whack-a-mole is telling. Managing vulnerabilities means a lifecycle approach and understanding that once they’re discovered, vulnerabilities can have one of three fates:
    1. Remediate – simply put, apply the fix or patch.
    2. Defer – push the fix until a later point in time such as when the system is retired shortly.
    3. Accept – accept that the vulnerability will not be fixed, and alternative accommodation needs to be made.
  2. Execution Discipline: Understanding the discipline with which an organization executes the tasks of a vulnerability-management program says a lot. Are there change controls? Is the process well-documented and universally accepted across the organization? Does the program include all of the organization’s assets? These and more are important questions to consider. As an organization matures, execution will be more repeatable and predictable.
  3. Follow-through: Few things are more important than following through. It makes no sense to scan, notify, but then do nothing but wait for someone else to do something. As an organization matures it will learn to not only notify but report, and follow-through on impacting positive change.

All this said the important thing is to figure out how your organization ranks, and what your real level of maturity is. There is no universal answer to what maturity level your particular organization should be at. But knowing is a critical first step.

The Red Herrings of Cybersecurity Blog Series 3 of 4

Welcome to 2021.

I felt like I needed to write that we survived 2020 and are now well on our way to whatever things this year holds. In this series, I’m addressing the things that your vendors do or say that are “red herrings” – that is, they sound good but aren’t quite right.

In this installment, I’m going to address complexity. Having been involved in selling cybersecurity solutions since roughly 2007, I believe I know a few things about this.

I believe with all my heart the following statement to be true.

“The value of any security solution is inversely proportional to its complexity.”

Give that a think for a second.

The more pieces of a solution your vendor has to virtually duct-tape together for you, the less real security value the solution holds overall. I do not doubt in my mind this is true. The reason for that – I’ve seen it with my very own eyes. I’ve witnessed 100+ page solution specifications that were so complex I don’t think anyone truly understood what was happening. Forget about actually explaining it.

I think customers sometimes believe that because a solution they’re being presented is exceptionally complex that it is better. That has something to do with the level of knowledge of the buyer. I’ve seen opportunistic sales teams take advantage of this, and it’s unfortunate.

The truth of the matter is simplicity always wins. It is difficult to debate that rationally. The more steps there are in a process; the higher the chance that there will be a failure along that chain of events. As a buyer, you should be looking for the simplicity of the overall solution. Additionally, look for simplicity in the various technology components, processes, and outcomes.

Rejecting complexity and insisting on simplicity is critical in security. It is particularly critical when you’re dealing with managed services. Here are 3 of the most important pieces, when it comes to keeping it simple:

  1. Engagement process – the process by which a customer engages with the vendor for specific tasks, workflows, or requests; for example, requesting changes or working incidents
  2. Integrations – connecting technologies together, to maximize their effectiveness, must be simplified to keep the system from becoming brittle and incurring unexpected outages
  3. Technical solution – the various technical pieces of the solution should minimize complexity by limiting the number of specialized components, and the number of times that a workflow passes from one technical system to another

There you go, part 3 on complexity. In a nutshell – if you don’t understand the solution someone is trying to sell you because it’s uber-complex … it’s probably not right for you.

What the SolarWinds Compromise Means to You

A Summary Analysis of the SolarWinds Breach

What happened?

In the simplest of terms, SolarWinds – a company synonymous with Network Management Systems (NMS) that is used almost universally across ~300,000 customers worldwide – was compromised through what is being labeled a “supply-chain” attack. This means that attackers from what appears to be a nation state-sponsored APT (Advanced Persistent Threat) group executed an attack against the software company that allowed them to insert code into SolarWinds’ most popular platform called Orion.

Between March and June 2020, the attackers were able insert code into the build system of Solarwinds’ Orion tool and push out updates which contained what is effectively a trojan horse. This means that the attackers were then able to use the compromised update of the Orion platform to then pull down malware (Sunburst) onto systems that were compromised with this update. From there, attackers had what is effectively free-range on the victim’s network. Since they were operating from a tool that is meant to reach out and monitor/manage network and system infrastructure, their compromised allowed them virtually limitless capabilities on most networks they infected.

It should be noted that while the attack appears to be targeted to the government sector and it’s providers, such as FireEye in one documented case, it is being relayed that any customer who had the relevant software installed should assume compromise.

*Please keep in mind that this situation is actively evolving with significant global effort to provide more information as it becomes available. The information contained in this advisory is subject to change at any time, and we encourage you to do additional research.

 

Why is the situation critical?

SolarWinds’ Orion is one of the most popular NMS (Network Management System) platforms out there. As a result, it’s confirmed install base is some 18,000+ networks worldwide. If you have the tool installed you are advised to assume breach and immediately enact your breach protocols and procedures. Work your incident response processes through, and in the event that you have evidence of no compromise you will have peace of mind and certainty. Even though your organization may not have the tool installed, it’s highly likely that one of your partners or suppliers may, leading to a third-party risk management nightmare that requires urgent attention. Now is the time to reach out to your close partners, particularly those that have assets connected into your physical or virtual network, and obtain certainty on their current state relevant to this compromise.

Organizations that find themselves compromised with this attack should assume that the attacker has had full access to all NMS and connected systems, assets, and data, and could move around the network undetected and exfiltrate sensitive data at will. There have been no detection capabilities prior to this breach going public, and new indicators of compromise (IOCs) are being published as researchers around the world work to uncover them. It should be noted that attackers will adapt and change their signatures to avoid detection. It is highly advised that companies review their logs for signs of long-term compromise based on the IOCs known at this point.

 

What should you do now?

If your organization does have SolarWinds’ Orion installed, you can take immediate steps to mitigate while you investigate. At minimum we urge all customers to review their logging, network access, and security strategies at this time to minimize potential impact and mitigate risk. Additionally, we provide the following suggestions:

  1. If you have Orion installed on your network and rely on it for monitoring/management you must immediately disable it’s access to the Internet. If you are unable to do so, access should be limited to absolutely only those IP addresses that are required to operate,
    1. Additionally, perform in-depth log analysis going back to March for the IOCs being published including domains that are using in the attack. Keep in mind now that the attack has been uncovered, these will likely change as the attackers pivot their attack to avoid discovery.
    2. Monitor closely all Orion NMS network activity, and perform packet-capture logging for evidentiary purposes, is possible.
  2. If you do not have Orion installed you should not necessarily assume your organization is safe. Consider your 3rd party suppliers and connected partners and perform due-diligence to understand whether these have the tool installed and could have a potential compromise.

 

What can Lightstream do for you?

 

Right now

  • Lightstream’s security team can assist in assessment or analysis of the situation to understand potential impact to your organization
  • Lightstream’s teams should be alerted immediately via ticket if your organization has SolarWinds’ Orion installed so that we can take additional measures for investigation
  • If your organization has minimal, none, or insufficiently operationalized endpoint or network security monitoring and response capabilities, Lightstream can help by deploying, managing, and detecting and responding to threats such as this both today and in the future

Near and Long-term

  • Lightstream’s Edge Defense and Endpoint Defense services are optimized to identify, protect against, detect and respond to, and recover from threats to your organization’s IT infrastructure, systems, and applications. Enterprises large and small can use our fully managed platform to supplement their own security operations (SOC) or fully outsource the management, detection and response 24x7x365
  • Lightstream’s expertise in Zero Trust architecture can be used to evolve your physical and virtual network to minimize the damage and business impact from even sophisticated attackers. We offer this service to our managed and new customers
  • Lightstream’s Security Advisory Services can perform a Security Strategy Program Framework (SSPF) assessment to understand how your existing security strategy would be impacted in cases such as this. This is offered to both new and existing customers.

 

Additional Links and Resources

The Red Herrings of Cybersecurity – Blog 2 of 4

Hello again.

In the previous blog in this series, I set things up for you. I explained the three things that I believe are “red herrings” in our industry – and now we’re going to dive into the first. Let’s go for a short, pointed, and honest ride.

There has been a consistency about managed services providers in the years I’ve worked for them. While not particularly comforting, the consistency of failings at least meant that we were all doing it wrong together. There is cold comfort in that.

One of those things that killed me for years is the speed of implementation. Or should I say, the complete lack thereof? In my years with HP, one of the managed services accounts that I worked with directly was grumpy because it had taken over 9 months to get an IDS successfully implemented. Yes, you read that right. Nine months. It’s not like security is a real-time battle of good and evil, and losing seconds is cause for concern, right?

I swore that I’d work to improve this, but ultimately I was unsuccessful. Then I left the company. But this stayed in my mind for a while. In my next role, I was too far removed from this situation to be able to affect it. That said, it never left my mind as my team and I advised CISOs on strategy and program development. The goal was always to decrease the time that elapsed between signing a contract and getting “security value.”

Fast-forward a bit to when I joined my previous role at Armor. The company was touting “2 minutes to deploy” and given my previous experience I thought I hit the jackpot. I’d learn over the next two years why I had been chasing a false dream. I’d recognize that faster is not necessarily better, although rapid time to value is desirable.

So what changed that swayed my thinking? Experience.

You see, I had the opportunity to witness a few “2-minute” deployments. They were categorically a disaster. Why? The answer lies in another question.

“How much protection can you expect from a security tool that does near-zero customization?”

If you answered the above with “about that, near-zero” you are now in my headspace. One of the reasons; and this is personal opinion now, there were so many install failures and missed issues downstream was that we were going for speed versus security. Sure we had it installed in two minutes. But did it serve any value? That was debatable, at best.

The lesson is this – to provide a valuable outcome to your customers, you need to do the work. There is a multi-step process that needs to be followed that I’ll readily share with you, here.

1. Understand your customer, their environment, and their challenges. Without this, you’re applying peanut butter. There are no two customers that share the same strategy, architecture, network topology, and security response needs. This I can guarantee. So why would you pretend that a single stock configuration would do anything but provide for the most basic of controls? I would argue that without this step you’ll be doing more harm than good.

2. Prototype and test your configurations. Once you think you know your customer, develop the defensive model, policies, and response actions. Work hard to identify not just the 80% case but those 20% outliers that are going to cause trouble once you deploy. Here’s a hint – one of the most difficult things to get right is the disruptive cases. The situations where something happens to upset the customer’s ecosystem due to a configuration you’ve made are irreversible – especially during initial deployment. If you can’t get it right from the start, you’ll lose your customer’s trust before you ever get to protect them. Minimize your unknowns; that’s the best advice I can give.

3. Expertly guided deployment is essential. Far too many times I heard customers say, “We got this” and then proceed to bungle everything because of either ego or something else. But I promise if your provider is offering you assistance to deploy – take it. If they’re not, ask why they’re not helping you be successful.

Expect this effort to take you north of forty hours for a mid-size implementation. That’s my estimation. You, the provider, should spend a week of solid work to get to a deployment stage. That’s a far cry from 2 minutes but provides infinitely more security value.

While I still believe that deploying as quickly as possible to get security value is critical, I no longer believe that doing so at the expense of customization and testing is viable. Everything comes at a price, and in cybersecurity, the price for protection is time. And effort. It takes effort, planning, patience, and expertise on your part and your customers. I don’t care how you present it – those are things you can’t rush.

Next up, removing complexity. I welcome your comments in the meantime.