How Much Do You Really Know About Good and Bad Bots?

Believe it or not, bots (short for ‘robots’) have been around for quite a while—the first one, ELIZA, was created in 1966 by Joseph Weizenbaum—but it’s in the last couple of years that they’ve gained a reputation so bad that nowadays we see them as one of the worst things to happen to the online world.

The word alone makes most people grab their website and run in the other direction because the first thing that pops into someone’s head after hearing ‘bots’ in a conversation is spam, ad fraud, bad traffic, and so on.

And with “bad bots” accounting for 20.4% of web traffic in 2018 (according to the Bad Bot Report 2019, can we be surprised that this is the case?

However, in this bad bot frenzy, we tend to forget one important piece of information: that bots are also our friendly neighborhood computer assistants who, among other things, are there to help us improve the visibility of our websites and their online authority.

So, how are we supposed to distinguish the good from the bad ones? Is there a way to limit the fake traffic coming from the bad bots? What awaits us in the future when it comes to these seemingly hostile programs?

Today, we’ll do our best to answer these questions and give you information that could help you fight bad bot traffic in the future.

Good Bots Always Come in Peace

Believe it or not, bots appear in a number of your favorite messaging apps, such as Skype or Facebook Messenger, but also e.g. Skyscanner (an app for finding and booking flights). Virtual assistants in your phone, such as Siri and Alexa, are all bots, too—the so-called chatbots—there to help you with your day-to-day tasks.

Google, Yahoo, and Bing all have their bots, as well, which crawl websites and their content to produce search engine results for certain queries. You might know them as spider bots, and they are extremely important when it comes to a website’s SEO and overall performance.

Copyright bots, for example, search for content that might be plagiarized and stop other people from taking the credit for your original work. Shopbots help you find the lowest prices for the products you love and wish to buy. We could go on and on, but you probably get the point by now.

Like most things in the world of technology, bots were created to help people and their businesses thrive. They weren’t designed to be the main culprits in ad frauds, DDoS attacks, spam relay, and so on.

However, as is the case with most things created for the sake of technology, people eventually found a way to use bots for the above-listed malicious activities, which is what has led us to this point and the rise of bad bots in almost every industry imaginable.

Bad Bots Have Bad Intentions

If you are a marketer or an advertiser—or even a business owner—you’ve probably stumbled upon issues with bot traffic by now. It’s almost inevitable, as they make up that damned 20.4% of all web traffic.

If you’ve had the luck of avoiding them so far, then know this: bad bots don’t play by any rules, whatsoever. They are sent by a third party (or your competitor) to your website, to harm it and, ultimately, harm your business. They can steal content, account logins, and identities with ease, and spam forums with invasive messages or ads—and that’s not all.

Last year, the OWASP (Open Web Application Security Project) published the Automated Threats Handbook for Web Applications, in which they list top 20 automated threats and categorize them into four distinctive groups: Account Credentials, Payment Cardholder Data, Vulnerability Identification, and Other (this includes everything from ad fraud and scraping, to token cracking and spamming). This should give you a pretty clear picture of what bad bots are capable of. In short: they are not here to help you out.

So, how did they become so widespread in this modern-day age? The reason for this is really simple: if you have basic programming skills, you can build them yourself, or if you don’t, you can buy them for as little as $2. You even have websites whose sole purpose is to direct bots to a certain website.

However, if the attackers want to create damage of devastating proportions, they will turn to botnets—a huge number of “infected” devices (called zombies) connected via the Internet and run by a “bot herder” (who can either be a spammer, a hacker, or both). A bot herder can easily perform a large-scale attack by using a C&C (command and control) software, and “telling” all the bots where to go and what to do.

More often than not, you won’t even notice that your device has been infected by a zombie until it’s too late, i.e. until your traffic starts dropping (search engines WILL detect the botnet and blacklist your site, which will lead to you losing your position in the search results) and you start spamming other devices with emails and messages.

To give you an idea of just how much power botnets have, we’ll remind you of a denial-of-service attack from 2000, which rendered Yahoo, eBay, Amazon, and CNN completely useless for a week. At the time, it was one of the largest DoS attacks ever performed. And it was all done by a 16-year-old Canadian boy, working under the username Mafiaboy.

Once you put things into perspective and realize just how much damage bots can do to your business, you begin to wonder whether it’s at all possible to protect yourself against this digital menace.

Fighting Bad Bots: An “Incessant Game of Cat and Mouse”

“Bot operators and bot defenders are playing an incessant game of cat and mouse, and techniques used today, such as mimicking mouse movements, are more human-like than ever before,” said Tiffany Olson Kleemann, CEO of Distil Networks.

Sadly, there is no magic, one-size-fits-all solution that will help you get rid of unwanted bots completely, but there are ways to fight them and keep them at bay. Knowing the difference between good and bad bots is a start when it comes to protecting your website because it allows you to take the right course of action.

Here are some things you can do to prevent bots from swarming your site:

• Use CAPTCHAs or 2-factor verification—Bots may be getting smarter and finding ways to fool CAPTCHA (although Google has updated its reCAPTCHA to better protect your website from spam and abuse), but this can still help you block some of the bad traffic coming your way. Using advanced CAPTCHA methods, e.g. apps that stop bots by having your users watch videos and answer related questions are also a great solution to this problem. Two-factor verification functions similarly, so you can give that a go, too.

• Be wary of malicious extensions—We all have several extensions installed within our browsers, so it might be difficult at first to spot a malicious one. However, it’s important to remove any suspicious browser extensions, because they can be a sign of—you guessed it—bots. If you notice anything suspicious going on in your browser, react immediately, and always use verified extensions.

• Secure your backdoor paths—Your exposed APIs and mobile apps, too, can be your vulnerable spots. Protecting solely your website will have little to no effect if bots can use backdoor paths to attack you.

• Block suspicious IP addresses and links—To do this, you’ll need to pay special attention to your inbound and outbound traffic in Google Analytics. If you notice a spike, higher bounce rates, and lower conversion rates, this might be a sign that bots are lurking somewhere around your site. Find the IP addresses and links they are using and block them as soon as possible.

• Use a Javascript challenge—JavaScript Challenges are generally used to discern attackers from legitimate clients during DDoS mitigations. The way it works is that you send every possible client a JavaScript code that comes with a certain challenge. The best part of it all? Most bots cannot respond to these challenges, which means that you’ll know for sure if the traffic you’re getting is legitimate.

If you want to get the best possible results when trying to fend off bad bots, then you need to try and test different methods, and see which ones are effective and which ones are simply not. As we said, the solution doesn’t come in a one-size-fits-all package, so you’ll need to figure out the best possible direction for your website and business.

Conclusion

No amount of wishful thinking will help us get rid of bad bots forever, as the most experienced attackers will keep on coming up with more and more dangerous bots as time passes. And as long as people have such easy access to bots—and at cheap prices, too—no website will be completely safe.

However, what you can do is take precautionary measures and try to protect yourself as much as you can. Awareness and being able to recognize what you’re up against is a great first step in your fight against bad bots while experimenting with different protection methods will help you be better prepared for stopping these nuisances.

But the most important tip we can give you is: don’t panic. No matter how scary they may seem, bots do not equal the end of the world.

Plus, in your panic, you might forget about all the good virtual assistants out there, doing their best to make your favorite apps a bit better every day or helping your website rank higher in Google’s SERPs.

And let’s all agree—good bots deserve more than that.