In this part of the blog series, and before we dive into the mechanics of the product, I want you to pause for a second, step back and think of the big picture.
Never try to implement security solutions just because they are cool or you have to do it because everyone else is doing it. Whether you have a Microsoft security product for endpoint or one from different vendor, I urge you to consider where and how this fits in the big picture.
That’s what I usually do. I take a look at the whole security strategy and build my defenses to protect my valuable assets. Let’s start!
Read other parts here:
P1: Microsoft Defender for Endpoint – Architecture
P2: MS Defender for Security Strategy & Role of AI
P3: MS Defender for Endpoint – Threat and Vulnerability Management (TVM)
P4: MS Defender for Endpoint – Attack Surface Reduction ASR
P5: Microsoft Defender Antivirus Internal Mechanics
P6: Microsoft Defender Endpoint Detection & Response (EDR)
DISCLAIMER: This content was written for the “Microsoft 365 Security for IT PRO 2020/2021” Edition which talks in great details about the entire security stack for Microsoft 365. Newer version of the book is now released and can be accessed here. I encourage you to download the book to get updated content of defender for endpoint and many other M365 security products.
Microsoft Defender for Endpoint and your Endpoint Defense Strategy
Now that you know the different components and capabilities of Microsoft Defender for Endpoint (Check part 1), and before diving into how each component really works, I want to spend some time to help you position Microsoft Defender for Endpoint’s components within your endpoint defense strategy. Only by doing that, you can really understand the power of the product and how relevant each component is to your overall endpoint security strategy.
The reason why I am introducing this blog section is because I see a lot of customers focusing on some of the main product capabilities without proper attention to others. Doing so will not help you get the full potential of the product and might leave you at risk. You should have a clear context on how each component of this product fits within your endpoint defense strategy, so you can configure it properly to maximize your return of investment (ROI).
Each capability in Microsoft Defender for Endpoint serves a purpose, a definite goal, and represents a piece of a bigger puzzle. You must understand how each piece fits in the puzzle to get the complete picture. As already depicted in part 1, I don’t want you to dive in blindly and configure product features without an end goal in mind.
To do that, I want you to forget about Microsoft Defender for Endpoint for a moment and focus on the bigger picture: your endpoint security strategy. What are you trying to accomplish? What is the end game here? Are you looking at stopping all types of attacks? Do you think that by deploying a security product that all your problems will go away? Moreover, how does Microsoft Defender for Endpoint fit in your strategy and what is the homework you should do to mitigate the risk of compromised endpoints?
Well, there is good and bad news for you. The bad news is that most companies and organizations are super easy to hack and compromise. The good news is it doesn’t have to be this way!
Building your defenses in the real world
I want to give you an example of how security is designed in the real-world. Let’s take the Pentagon building for example (the headquarters building of the United States Department of Defense), and let’s say we want to plan the security for the Pentagon building. Sounds interesting, right?
The first thing that comes to mind is to implement basic security controls and adopting best practices on how to secure buildings. Our goal, in this phase, is to harden the building and minimize the chance of being compromised. We can start by deploying a security gate with mantraps to inspect people going into the building. This could perhaps be combined with additional access control systems with biometrics.
These are called preventative controls as we are trying to prevent bad things from happening at this stage. The more we deploy these security best practices, the more we increase our security posture. In the endpoint security world, this mainly translates to patching our endpoints, configuring the Windows Firewall, and following the security recommendations from Microsoft Defender for Endpoint Threat and Vulnerability Management. This will also help us increase our Microsoft Defender for Endpoint configuration score (see the TVM section in coming blog posts for more details).
We also can go one step further in our preventative controls, that is, trying to reduce our attack surface. Instead of having ten gates in the building, it is better to have one main gate with heavy security controls. This will reduce our attack surface since our focus will be in protecting one gate instead of ten. We can also implement additional checks for people leaving the building to prevent people from exiting the building with, for example, sensitive data.
Another idea that comes to mind is to put in place a policy that restricts employees to access only floors in the building where their own offices are located. Mind you that this is not about trusting or not trusting people, it is rather reducing the chances that bad things might happen, accidentally or on purpose. By restricting which floors each person can access, we are reducing the chances of unauthorized employees entering a room with sensitive information or systems. These (type of) controls that reduce attack surface are mapped to Attack Surface Reduction component in Microsoft Defender for Endpoint (see the Attack Surface Reduction section in coming blog posts for more details).
Following the security recommendations in Microsoft Defender for Endpoint Threat and Vulnerability Management, configuring Attack Surface Reduction feature, configuring the Windows Firewall, and applying security patches are all preventive measures. It is us doing our homework to our best knowledge to secure the endpoint and to reduce the probability of an attack (lowering the risk).
But we are not done yet. We should also inspect everyone entering the building to stop unwanted visitors (stopping malware from running). This can be a security officer at the gate asking employees for their IDs, and then running their names against the internal database to check if they are authorized to enter the building (signature-based).
For visitors who are not listed in the internal database, the security officer might run their names against the FBI database (the Microsoft Intelligent Security Graph in our world); just to make sure visitors don’t have criminal records. We might have security officers walking inside the building and detecting unusual behavior. This could be someone (who is not allowed to be near the server room) who is found lingering near or even inside the server room with his laptop and a console cable. This maps to Microsoft Defender for Endpoint Next Generation Protection which uses Microsoft Defender Antivirus. The objective here is to inspect employees (malicious code), run their names against a local database (signature-based) and for visitors (who we don’t know about) we inspect their names against the FBI database (Microsoft Intelligent Security Graph and cloud-based protection).
Is that it? Not even close. We still need to assume that bad things will happen (assume breach mindset) and create a response and containment strategy. We will install security cameras everywhere to monitor what’s happening inside and outside the building. This helps us to detect an attack and to plan for our response.
There will be a security room with people monitoring all security cameras who will shout out when something out of the norm happens. Perhaps we have a response team that we notify when an event unfolds so they can investigate, collect evidence, and contain the attack.
Without a response mechanism, we may be good in detecting attacks. But what good is it if you can’t take any action to stop the attack or prevent it from doing (more) damage? Perhaps we involve our forensic team to understand how this attack happened so we can prevent similar ones from happening again in the future. But that doesn’t solve the ongoing issue. This is what the Detect and Response capability in Microsoft Defender for Endpoint is all about. It is assuming that bad things will happen, putting a plan to respond, containing, and remediating the attack (see Endpoint Detection and Response section in coming blog posts for more details).
But with all those security cameras inside the building, it is hard to keep an eye on everything that happens. We need some sort of automation to help. Perhaps we have a code that reads faces of people from the security camera feeds, uses cognitive services to identify their faces, runs them against an FBI database to accelerate detecting and investigating suspicious behaviors. This is what Automated Investigation and Remediation in Microsoft Defender for Endpoint is built to do (see Automated Investigation and Remediation section in coming blog posts for more details).
With that in mind, let’s not forget that the FBI maintains a database for (most wanted) criminals. If we see one of those criminals inside the building, we need to immediately respond as their existence is an indicator of an attack.
Think of those pictures of criminals placed on the walls of a police station. You might also be familiar with the phrase “if you see something, say something”. This is what the Microsoft Defender for Endpoint Indicators of Compromise is all about. You get a list of suspicious files hashes that are well-known to be malicious, and you ask the product to perform an action once it detects one of those files.
You can also go one step further: Hire security professionals who will routinely go through the access control logs and try to find if there is something suspicious. We are talking about proactively hunting for malicious acts; this is what the Microsoft Defender for Endpoint Advanced Hunting is all about (see Advanced Hunting section in coming blog posts).
As you can see, security in the real world is not that different than security for your endpoints because security concepts and theory are the same. In both worlds, you need to plan your defenses carefully using a defense-in-depth and multi-layered approach. Start with preventative controls to keep the bad guys out. Then move to detection controls and plan your response strategy accordingly. You should always have the assume breach mindset to better improve your security.
Now that you have some context of how endpoint security maps to the real world security, let’s talk about the elephant in the room, Machine Learning and AI. The goal here is to use automation and large set of data collected from various locations to maximize our detection capability. If a zero day attack malware is detected in another organization, it would be great if your endpoints are aware of it in case it hits yours. This is the power of big data and sharing signals.
The Assume Breach Mindset
There is a story of a criminal who was asked why he robbed banks, and he answered, “because that’s where the money is”. His answer was simple and makes sense. Cybercriminals today are no different, they go where they can profit from. You’d have to be living under a rock to not be aware of the scourge of security breaches that occur every day.
But the reality is that most security professionals put all their chips towards preventing breaches from happening in the first place. They think that by putting all these fancy and expensive defences, they could keep any attacker from compromising their defences and get to their most valuable assets. This didn’t work out very well.
Patient and resourceful attackers nowadays can get into any organization using well-known techniques such as research, reconnaissance, stealthy intrusion, and quite exfiltration. This led to a mind-set shift on how we think of security incidents and defences.
In fact, there is a philosophy of information security – assumption of breach, which simply means that you should accept the very possibility that attackers are already inside your network, regardless of your defences and your ability (or inability) to detect them. To think otherwise is foolish, just because your defences can’t see intruders or your systems could not alert you on their existence, doesn’t mean they aren’t there.
It is not that your security defences are not good, but they might not be good enough to address determined and persistent adversaries. None of the security prevention technologies can guarantee complete protection against advanced threats.
| Note: When the former director of the CIA and National Security Agency Retired Gen. Michael Hayden said in 2012 “Fundamentally, if somebody wants to get in, they’re getting in. Alright, good. Accept that.” At that time, no one from the audience understood what he really meant, but his words are the core of the assume breach approach we know today. |
Operating with this assumption reshapes detection and response strategies in a way that pushes the limits of any organization.
With such mindset, security focus changes to identifying and addressing gaps in:
- Detecting an attack that might be happening in my network.
- Responding to an attack (investigation, collecting evidence and remediation).
- Recover from an attack (stopping the attack, isolating machines, recovery data and systems).
- Lesson learned (prevent future similar attacks, indicator of compromise, hardening systems).
When we adopt the assume breach methodology in the endpoint security story, we want to start thinking of an effective way to gather evidences left by the adversary and to detect the evidence as an indicators of compromise. A lot of alerts will be generated in the process, so we need to triage the alerts to find if they warrant further investigation.
To do that, there is a need for automation to help in the investigation (see AIR blog post). We need to notify relevant people when detection happen and act quickly. This means, we need a way to contain or evict the adversary and then gather more evidence to understand how the attack happened in the first place. Perhaps more systems are involved, and they should be included in our investigation.
This might not be enough; we need to actively hunt of attacks and look for signs of suspicious behaviors (see the Advanced Hunting blog post). We might need help from threat experts (see the Threat Experts blog post). We also don’t live alone in this planet, so we need to keep an eye on major attacks happening in the wild and investigate how they are relevant to our organization (see Threat Analytics blog post).
Microsoft Defender for Endpoint (MDE) components and capabilities are positioned to help you build a good endpoint security story. From prevention controls, to stopping malicious code from running, to containment and remediation threats across your endpoints.
Whenever you learn about one of the product capabilities, I want you to clearly position it within the overall security story and understand how this capability is helping to secure your endpoints. One component is not enough by its own. You need to use a combination of pre-breach and post-breach capabilities for better visibility and protection.
As you can see from the figure below, we have prevention controls such as following security recommendation from the Threat and Vulnerability engine in Microsoft Defender for Endpoint, and configuring Attack Surface Reduction capabilities.
Next, we have the pre-execution real-time protection engine which is Microsoft Defender Antivirus. It tries to allow/deny malicious code from running empowered by the cloud-delivered protection. If the code is allowed to run, then the post-execution sensors of the Microsoft Defender for Endpoint EDR alert on suspicious activities (post-breach) and automated investigation and remediation helps in automation.
With that in mind, SecOps are empowered with tools to help them actively hunt for threats, consult a threat experts, or understanding how global cyber attacks are relevant to their organization.
The Cyber Kill Chain
Another topic I want to cover in this section is the famous and well-known cyber kill chain. It helps you connect the dots and enforce what you’ve learned so far.
As security professionals, we love to talk about risk and risk analysis, and from there, we tend to evaluate threats and vulnerabilities so we can decide on which security controls to implement. For that, I want to talk about the cyber kill chain.
What is the cyber kill chain? You might know what it means even if you don’t’ recognize the name. A cyber kill chain reveals the phases of a cyber-attack from early reconnaissance to the goal of data exfiltration. It can be used however by security professional to improve network defences on each stage of the cyber kill chain.
Let me explain more how sophisticated attacks happen. Usually an attacker selects a target and performs some research to learn more about vulnerabilities. This is usually called the reconnaissance phase. After doing all the research, the attacker is ready to move to the weaponization phase as he creates a malware trailed to one of the vulnerabilities discovered.
Guess what is the next step? Of course, the attacker delivers the malware to the target via an email attachment, USB drives or any other possible way. Now that the malware lives in the target machine and network, the malware starts a privilege escalation on the local machine to elevate its right and install an access point or a backdoor and then connects to the Command and Control (C&C) server. The intruder now have remote access to the network.
Most of the time, patient zero or the first machine being hacked, is not an interesting target by itself. It just happened that it is the weakest entry point to attack the network. The attacker now starts discovering nearby machines and resources and move from one machine to another (this is called lateral movement) until he gets to the intended resource or credential. This can be the domain admin credentials or perhaps a database with high value information, which is the data exfiltration phase. The objective can also be data corruption or data destruction.
Usually it takes long time until someone discovers that an attack happened and then having forensics teams involved trying to understand how the attack happened in the first place, what targets are compromised, and what was the damage.
These phases together are called the cyber kill chain. However, we (as the protectors) can design and plan our security strategy around the same phases of this kill chain. We can either focus all our security controls trying to prevent the attacker from installing a malware inside our network, which is the pre-breach security approach, or we can focus our security controls on detecting lateral movement of an attacker after the attacker compromises a machine, which is the post-breach approach.
Most security controls nowadays focus on the pre-breach approach, that is how to prevent a malware from getting delivered in the first place. Here you have signatures and packet filters that are good in recognizing known threats and then injecting the results in the form of antivirus signatures or intrusion detection-based signature systems.
However, attacks become more sophisticated and they start to adapt to evade detection using technologies like polymorphism, and with that, the defences themselves start to evolve and we start seeing heuristics and behavioural rules being introduced into the security space, including sandboxes where pieces of the content would be executed in a safe isolated environment and then monitored for signs of malicious behaviour.
But the problem with this approach is that it really based on having identified threats and then constructing these rules and behaviours that look for intruder to identify similar threat, even if their signatures had changed.
The next wave is machine learning, promise of being able to get a head of a threat, and not being reliant on having to have found something before in order to be able to detect it for the first time. This is driven by the introduction of zero-day malware that are coming out and the sophistication of the adversary was growing and therefore there was a desire to get more sophisticated defense. The promise is being able to build super intelligent machine that would be able to reason its way through the high volume and velocity of data that is prevalent in the cyber environment.
Such machine learning power can be used in the pre-breach approach and at the post-breach approach. When used in the post-breach approach, the machine learning model is trying to detect anomalies in the network that might be caused by a lateral movement of an attacker. Azure Advanced Threat Protection (known known as Microsoft Defender for Identity) is a good example of the machine learning model that tries to detect anomalies in user’s identities and lateral movement happening inside your network by analyzing authentication traffic hitting your on-premises domain controllers.
Pre-breach detection on the other hand is focused on the prevention side of the story and in identifying threats early in the cyber kill chain and preventing the malware from installing in the target machine or quickly remediate such zero-day malware after execution (post-breach). Endpoint antivirus solution is the first and oldest technology here. But with time and as the sophistication of attacks increase, we start seeing machine learning playing a big role and this is where Microsoft Defender ATP comes to the picture.
Microsoft Security and the Cyber Kill Chain
Now that you know more about the cyber kill chain, let’s see how Microsoft Defender for Endpoint (MDE) and other Microsoft security products fits into the picture.
The famous defence in depth strategy still apply here but requires re-thinking of the approach. Nowadays, network perimeters are not defined by the corporate network that is protected by top the art edge. Remote working motion is demanding a change of such boundary. Business applications are no longer hosted in our data centres, but rather consumed from the cloud as a SaaS offering. Employees are bringing their own devices and millennials are demanding a new culture of working.
We need, as security professionals, to re-imagine how we are building our new defenses to meet these new challenges. Microsoft offers a lot of security solutions that work together in a layered approach, defining by that a new philosophy for defense in depth.
Let’s take the cyber kill chain and see how different Microsoft solutions helps you build your new defences. In this new approach, we can re-define our perimeter in four key areas, the application, the devices, the identity, and the data perimeters as you can see in the next figure.
An attack might start with a user receiving an email, opening a suspicious attachment, or clicking on a malicious URL. At this point, Microsoft Office 365 Advanced Threat Protection or Defender for Office 365 (with the use of Safe Links and Safe Attachment features) can detect and block the attack at this stage. Microsoft Defender SmartScreen might also help in protecting against phishing or malware websites and applications and prevent downloading of potentially malicious files.
If the attack was clever to evade detection at this layer, then the malware is already delivered to the device and Microsoft Defender for Endpoint can detect the malware after it infiltrates the machine. If the attack was so sophisticated and evade detection, then the attacker for sure will try to perform privilege escalation and move inside the network to find more interesting target (domain controller or the domain admin credentials).
Many techniques are used at this stage by the attacker such as pass-the-hash or pass-the-ticket, and a connection to the attacker’s command and control center is usually established at this stage. Defender for Identity (Azure ATP) helps detecting lateral movement inside your network and detecting unusual behaviors and privilege escalations.
On the other hand, Microsoft Cloud App Security and Azure AD Identity Protection play a role in detecting identity anomalies across your cloud environment (impossible travel, risky IP addresses, risky users and more).
The attacker then tries to perform data exfiltration, and this is where both Microsoft Cloud App Security and Microsoft Information Protection helps by protecting your data itself.
Together, these products help you build a new defense in depth strategy by preventing attacks from happening, detecting the existence of an attack inside your network, and protecting your data from being exposed.
Microsoft Defender ATP and other ATP products
As you learned so far, Microsoft Defender ATP is not the only ATP product out there. We have other defender products in Microsoft including the Defender for Identity (previously Azure ATP) and Defender for Office 365 (previously known as Office 365 ATP. The good news is that these products work together to give you defines in depth and an integrated layered approach as shown in the below figure.
Layer one is Office 365 Advanced Threat Protection (Defender for Office 365 at the email level where most attacks begins. Layer two is Microsoft Defender for Endpoint at the endpoint, where you can see what the attack is doing to the host. Layer 3 is Defender for Identity which addresses what identities have been compromised and what’s being done with them.
These three layers all work together seamlessly. With a layered integrated approach, you have a better chance of catching an attack early. If a clever attacker gets through the first layer or the attack didn’t come from an email message, then Microsoft Defender for Endpoint may catch the attack after it infiltrates the machine. It will then warn the Defender for Office 365 service of this new threat so that it can block additional email infections across the network and for all Office 365 customers. Even if this happens at a different Defender for Office 365 tenant, you get the protection due to the signal sharing capability between Microsoft security products. In such integrated protection approach, if the attack bypassed layer 1 and layer 2 defenses, then Defender for Identity helps detecting the existence of attacks by detecting unusual behaviors and privilege escalations.
The Role of Machine Learning
Machine learning plays a big role in the endpoint security protection story that I decided to dedicate a section to it. Every day, new stories emerge about one or the other attack. In such an attack, and in a normal average day, more than two million people in 232 countries around the world were attacked with 1.7 million new malware attacks that never been seen before. What makes things worse is that 60% of those first seen attacks were over within the hour.
In the security industry, we like to think a lot about incident security response (how we are going to respond to such attacks). Well, when all what you have is one hour or less and more than 1.7 million new malware attacks to deal with in a single day, there’s no response time that matters. The other aspect of this problem is having your security teams deal with a lot of security alerts and trying to prioritize which ones to deal with first. This is why a new way of security defenses is needed, powered by machine learning and automated responses!
The endpoint protection tool that you are going to deploy across all your machines needs to do more than detecting well-known malware based on signatures, those were the old days. It must be an intelligent endpoint with machine learning and intelligent response capabilities to deal with today’s security threats. It also should not consume all your machine’s resources (low performance) and be highly effective in protecting your endpoints. This can be in the form of consulting the cloud and leveraging the scale of the cloud as part of its protection capabilities.
Leveraging the power of the cloud
The cloud is not only used to offload the processing of malware analysis from the endpoint itself. It also opens the door for many opportunities that would be difficult to have on-premises otherwise. Machine learning and AI are good examples. Not only is it expensive for organizations to invest in machine learning on-premises, but also the visibility that cloud providers have across their entire network, and by extension larger parts of the world, gives them a clear advantage.
Machine learning, machine learning, machine learning…. We are security professionals, what do we know about machine learning! well, let me explain and believe me it is fun.
Machine learning models typically consist of an algorithm and training data. The quality of the model depends on the data it is trained with. Cloud providers like Google, Amazon and Microsoft have visibility across their entire network, which gives them a better opportunity to train their machine learning models on what is normal and what might be malicious, as they just have a lot more data (telemetry) to work with.
There are many cloud security services that use AI and machine learning to provide unique services that organizations can leverage. A good example is the Identity Protection service, and that is part of Microsoft Azure Active Directory. This service analyses authentication trends happening across the whole organization and detects anomalies and risky sign-ins. For example, if John usually logs in from the office and then suddenly logs on from a location he’s never been before, the service can raise a flag. Depending on the configuration of the feature, it could even challenge John to perform a multi-factor authentication.
What about using the power of the cloud with the endpoint security model? The way that Microsoft Defender for Endpoint defends against attacks is by leveraging the same principle and using telemetry and data, like signals from the Microsoft Intelligent Security Graph. Microsoft Intelligent Security Graph provides rich signals from vast security intelligence, machine learning and behavioral analytics that Microsoft allows you to consume and use to enhance your protection and detection speeds.
In fact, millions of unique threat indicators are generated everyday by Microsoft and its partners and shared across Microsoft products and services as shown in the figure below. This provides an unparalleled view into the evolving threat landscape and enables rapid innovation to detect and respond to threats. This is known as Threat Intelligence, and it is one of the main elements used to defend against sophisticated attacks.
Why Machine Learning and AI?
Let’s go deeper and try to understand how machine learning and AI play a big role in securing the endpoint and then how Microsoft Defender for Endpoint fits in the picture. I believe it is important to talk about this topic to distinguish Microsoft Defender for Endpoint from other anti-malware solutions out there.
It is a fact that human analysts are extremely capable of reasoning over data to and alert on breach activities. However, we have a lot of signals that come from different sources which makes it very difficult for an analyst to investigate. After all, an analyst can only correlate a limited set of signals to identify a breach or an attack. Humans just lack the processing power to deal with vast amounts of data. This is where machine learning comes into the picture. It can cut through all the noise and data more precisely and more efficiently (quickly). In a recent study conducted by Microsoft, it turned out their Machine Learning-systems are at least 20% more precise than manually crafted heuristics. This may not look like a big deal, but it is!
To understand how Microsoft Defender for Endpoint uses machine learning, let’s step back and understand how machine learning works. There are two basic types of machine learning: supervised learning, and unsupervised learning. A label is a name that you give to something. In turn, you feed that information (label) to your machine learning model and it tells you what that item is. In the context of this blog post, let’s assume that a label refers to something (piece of code) that is either malicious or benign (not malicious).
Supervised learning uses the labels that are fed into the machine learning algorithm. The algorithm learns from those labels, learns from the properties of files (entities) on the machine, and then predicts whether something is malicious or clean, given a brand-new sample that was not seen before. Imagine you are telling the machine learning algorithm that this ‘thing’ is a car (you label it as a car). You then start feeding the machine learning algorithm pictures of different cars and the fact that those pictures show images of a car (learning phase). Now, if you give the same machine learning a picture of a car it never saw before, the machine learning algorithm can predict that the new picture also depicts a car. In a way, it is very similar to how humans learn things.
Unsupervised learning, on the other hand, doesn’t need labels and therefore they don’t have bias that humans have. For example, what if a car doesn’t match the image of any other car we’ve seen so far? Is it still a car? This makes unsupervised learning a great complement to supervised learning. Take, for example, clustering algorithms (a type of unsupervised learning method) that groups things together without having any kind of predisposed labels. It makes them very good in detecting unknown elements (such as identifying new type of malware that have not been seen before). It’s like showing an unsupervised system pictures of cars and trains, which results in it sorting the cars and trains in separate groups. The system was never told one is a car and the other a train. It just looks objectively, without any bias, at the pictures and (tries to) determine common elements which may link one object to another.
Another good example of unsupervised learning is anomaly detection, which is simply the machine that learns what is normal and what is outside of ‘normal’. It doesn’t label things as good or bad, it just points out elements that don’t match a specific pattern. It is a great complement to supervised learning.
Microsoft Defender for Endpoint and Machine Learning
When it comes to Microsoft Defender for Endpoint, supervised learning is the primary method used for malware detection. The power of using machine learning is to overcome human limitations. On average, a human can keep seven variables in mind while, with machines, the capacity is infinite. Let’s look at a real-world example. An analyst can look at a piece of malware and perhaps identify five or seven attributes that makes it a malicious piece of code. These attributes could be the size, network connectivity or the location of the file, and then label it as malware. However, that malware or file actually has hundreds of other attributes for which an analyst just simply doesn’t have the time to go through entirely. But a machine can.
This is relevant because when a researcher labels a piece of code as malicious because he identifies five or seven attributes or indicators that he believes are suspicious, he can then feed the file (which is labelled as malicious by the analyst) to supervised learning algorithm, and the algorithm can go through hundreds of other attributes and learn of other new (and potentially unique) attributes that make the file malicious. This is how we can take one single thing that a malware researcher has labelled as malicious and turn that into a prediction for thousands of other malicious files.
The way Microsoft built machine learning for Microsoft Defender for Endpoint is by having a significant set of labelled samples. These samples include both clean and malicious files. This yields better results as the machine isn’t only trained to detect malicious code, but also learns what a clean file looks like.
Microsoft Defender for Endpoint looks at a lot of attributes and elements when inspecting a file or a piece of code. Things like static attributes of the file, partial hashes, who signed the file, but also behavioral and relational elements: Is this file related to another file? Was the file injected from another file? Was it downloaded from somewhere? What are the contextual elements related to this file? Is it communicating with a remote IP and is that IP part of a well-known command and control system? As you can imagine, the more data points you take into consideration to create a verdict, the more accurate the result will be.
There are many learning models that Microsoft Defender for Endpoint uses, and these models might run locally on the machine or in the cloud. When big data analysis is needed, it happens in the cloud because they might run for a long time and need a lot of processing (computer) power whereas while client-based models are used at the endpoint level, are super-fast, and usually don’t require a lot of compute power. For more information about Microsoft Defender for Endpoint and machine learning, see Microsoft documentation here.
About this Microsoft Defender for Endpoint Blog Series
During the years, I have worked with many security and Infrastructure services, and I usually don’t find good information in the web on how a product or service works. For me to master a service, I need to learn how it thinks, the internal mechanics, and even how the product group who designed it really thought about different features.
So, I started blogging years back to reflect my understanding and help others find useful information that is not found elsewhere on the internet (at least in one place) and direct from my experience.
This blog series is written after careful consideration and will help you imagine how Defender for Endpoint works from the bottom up. I rarely have time to blog these days, so I might not update the blog on new features. However, the content here will give the information you need to build on top.
CREDITS Big thanks to my friend and fellow Microsoft MVP and RD: Ahmad Nabil who helped me put such content and the Microsoft 365 Security for IT PRO book family who helped in reviewing and editing this chapter. Newer version of the book is available here with updated content and valuable content about other Microsoft 365 security services. Download the new book here.
