How can we prevent malware from communicating with a C&C server? You may think of using a CTI (Cyber Threat Intelligence) feed with a network blacklisting appliance. You may also think about blocking certain protocols or even using a Next-Generation Firewall to perform traffic inspection. But malware creators can be quite clever. They wrote malware that communicates through the HTTP(S) protocol, hiding among all the other legitimate web requests in your organization. After all, the best place to hide a tree is in a forest. Now, this still doesn’t get around a CTI blacklist, but there’s more to the story.

Instead of using a single domain that can easily be marked as malicious, malware operators began registering hundreds of domains daily as rendezvous points with their malware. But they didn’t simply hard code these domains into their malware, as these could be easily dumped from the malware and blacklisted. Instead, they began using domain generating algorithms to create a potentially unlimited number of domains in a way that the domains generated by the C&C server would be mostly identical to the domains generated by the malware. As such, the malware would simply communicate through a new domain each time and the previous domains would be discarded. This makes it infeasible to create a CTI feed, as each malware family or even instance would be generating up to tens of thousands of domains each day, that you need to collect, store and blacklist.

So, do we just give up and accept that there’s nothing we can do after a malware infection has occurred? Of course not! Let’s take a look at one of these domains to see if we can get any clues about how to detect them. Take a look at the below domain. Looks like gibberish, right?

That’s because it is. It is simply a random sequence of characters. Now, we don’t need to go into too much detail, but there is a mathematical way we can quantify the randomness of a string. So, we could simply flag domains that seem to be completely random, right? Well, this is exactly what we tried to do in the early stages of developing our detection capabilities. But when testing with a real-world dataset, we couldn’t get accuracies above 65%. This wasn’t a problem with how we were quantifying randomness, but rather that malware creators had thought of this as well. We could have gotten away with it if it weren’t for these meddling malware developers.

Below you can see some domains also generated by a DGA. They seem completely normal, right? This is because malware developers began using dictionaries to generate their domains. They simply combine random words together, often resulting in completely inconspicuous domains.

Therefore, we need to come up with something more advanced. There are many approaches you could take, but the one we chose was the use of neural networks. Neural networks are made up of a number of interconnected artificial neurons, modeled on biological neural networks such as those found in animal brains. This sounds quite esoteric, but at its core, it’s really quite simple. Think of them as a system that can infer what aspects of a domain name are important when determining whether or not they were algorithmically generated. It can do all this without us having to dictate exactly what sort of details it should be looking for. These details can become highly intricate and nearly impossible for a human to program. All we have to do is set up a good neural network architecture, collect a good dataset, and mark each domain in the dataset as either algorithmically generated or legitimate. The neural network will do most of the heavy lifting when it is trained. Sounds easy, but there are quite a few quirks when deciding what architecture to use and the subsequent optimizations required can be quite complex. Depending on who you ask, creating a good dataset is the most difficult, time-consuming, and also the most important part.

When setting up our dataset, we collected and obtained many gigantic datasets containing tens of gigabytes of domain names, both algorithmically generated and legitimate. We then crafted a neural network that we trained using these datasets to achieve up to 98% accuracy. Below you can see the accuracy we were able to achieve for 92 malware families in our validation dataset

For comparison, you can find the accuracies produced by a method that merely looks at the randomness of the domain name. As you can see it performs a lot worse, especially with DGA’s that use dictionaries. It also struggles with very short domains, where there is not enough information to make a good prediction and the accuracy begins to devolve to a random guess or even worse.

We often associate machine learning with many graphics cards or even tensor processing units, and you may assume that our detection method would consume a load of resources to make predictions. However, this is not really the case. We tested the throughput of our implementation and summarized the results below. As you can see no special hardware is required to run these detection methods. Keep in mind that the throughput refers to unique domains. In a real-world scenario, with deduplication and a whitelist, you will struggle to saturate even a single vCPU.

We use the previously discussed neural network and many other tools in our Fusion Center to help protect our clients’ infrastructures. If you would like to learn more about how neural networks work and how we can use them to detect DGAs, read our new whitepaper on the topic.

We go into details about how DGAs work, what neural network architectures we can employ, and how these architectures perform when detecting these domains. If you’re in the mood for something less technical, learn more about the tools, techniques, and philosophies that set our Fusion Center aside from a regular SOC.