Dreamhost Slammed with Brute Force

Filed Under

There are 17 comments

May 14, 2013 #

Mika Epstein

I wish we were alone, but a lot of hosts are getting hit again. Pretty much anything running less than 2G of memory got tanked on another host I use. From what I can guess, these attacks are only just starting. Last time, we got hit, then someone else, then everyone. It’s probably going ’round the cycle again, and will keep doing so until we find some way of stopping them.
- May 14, 2013 #
  
  John Saddington
  
  You “busy” then I suppose? 😉
  - May 14, 2013 #
    
    Mika Epstein
    
    Just a little bit 😉
    
    The DDoS problem is only getting worse. When you have a hundred thousand IPs hammering a domain, you have to start improving how our firewalls are handling the traffic. The old way was to check “Is this IP hitting the server X times? Okay, it’s bad.” And that works, to an extent. Already, hosts have improved to checking “Are you hitting THIS file X times in X span?” but we’re clearly going to have to improve our detection of what’s bad.
    
    You can throw all the plugins you want at this, but it doesn’t stop the effect of the DDoS, it just protects your site if they get to it. A DDoS is stopping anyone from getting to your site by overloading it (which kind of makes this a stupid bot-net when you think about it, since they’re killing the boxes without getting in…).
    
    I’m also not a fan of using a proxy for ‘protection’ because it’s not fixing the underlying issue: How do we make our firewalls smarter to detect the pattern of behavior of these attacks, and dynamically block, without putting strain on the web servers?
    
    (FWIW, I know many other hosts today having the same problem, but it’s not nice to name names or point fingers. They know.)
    - May 14, 2013 #
      
      Mark Root-Wiley
      
      Thanks for sharing these details, Mika! It’s interesting to hear this stuff from an insider such as yourself. Just in general, I really appreciate Dreamhost’s transparency on this one. Good luck with it!
      - May 14, 2013 #
        
        John Saddington
        
        That’s what I’m talking about! She’s a model community manager for sure.
    - May 15, 2013 #
      
      Chip Bennett
      
      Ultimately, doesn’t the real solution have to come from the ISPs through which the bot boxes are accessing the internet?
      
      ISPs need to have their feet held to the fire. Log the offending boxes, and start blocking their access. If/when users complain, tell them to clean up their hacked boxes.
      
      (I think this idea breaks down, though, the more bots that come from outside the US or similar countries.)
May 14, 2013 #

Adam Shields

Site5 has had some downtime lately from similar attacks. Seems to be happening with a lot of hosts that focus on WordPress.
May 14, 2013 #

Yongwoo Cho

thanks for the info. that explains why i couldn’t access our community blog last night that’s hosted with dreamhost. but from the other comments, it seems like other hosts will get affected too. can wordpress be modified so these types of attacks can’t happen?
- May 14, 2013 #
  
  Amy Hendrix
  
  Unfortunately, this kind of attack works by choking the server’s resources before you even get as far as WordPress — it doesn’t matter if your site is WP, Drupal, Joomla, or plain ol’ static HTML; if all the available bandwidth and memory are being used up by the bots, legitimate traffic simply can’t get through the traffic jam far enough to even access WP (or Drupal, or Joomla, or plain’ ol… you get the idea).
  
  So there’s really nothing we can do as far as modifying WP is concerned; having hosts who keep actively working on solutions, and being as transparent as they can be in the mean time, is about the best we can do.
May 14, 2013 #

Ray Gulick

Suggest BruteProtect plugin to deal with current multiple-IP brute force attacks: http://wordpress.org/extend/plugins/bruteprotect/

The more sites that install it, the better it works, because it draws on info furnished by each site to ban IP addresses from login screen.
May 14, 2013 #

Claverhouse

As far as I can see, this happens randomly to every cheap host — and no doubt to the more expensive solutions — and a user just has to wait it out; however many Dreamhost customers, not most, immediately start leaping up and down in their seats and scream abuse at the hosts, vow million-dollar deals are being wrecked by the outage, and swear to leave, never ever to return.

The lights go on, and apathetic calm reigns with lofty serenity until the next incident happens — as it will happen, no matter what anyone does, or wants — and the whole status comments circus starts up again as if it had never stopped.

I can live with a day or so of my websites being unavailable — and 99.9% of sites are not essential — unthrilled, but worse things happen at sea. If on the other hand my hosts lost my files then I would be aggrieved; site off-line now and then, not so much.

Dreamhost are pretty damn good at what they do: if people have gold-plated sites then perhaps they should look to pay considerably more than $100 a year.

On the other hand, over-estimation of the worth of things is common, and for most personal websites it would be ridiculous to pay more than that for hosting.
May 15, 2013 #

Primary Image

I agree with Mika above that WordPress plugins are not the solution to this. If a plugin is taking the workload, it’s already running a lot of server resources, especially database queries. For those on shared hosts, hopefully your hosting company will take steps to protect your sites. Two things I have seen work well though: (1) Putting htaccess file protection on the wp-login.php file and wp-admin directory. Whilst it’s not going to fend off a large brute force attack, it has successfully blocked the ovcasional bot trying to gain access to my client’s websites. Blocking bad bots here uses much fewer server resources than letting them get through to the WordPress login page. (2) One host – Heart Internet – redirects users to a Recaptcha check before they get to the WordPress login page. They’ve done this for all WordPress sites on their servers. Whilst I’m not a great fan of captcha checks (and they can be a nuisance!), it certainly has restored stability back to normal and seems to be successful in coping with the attack. Mike
May 15, 2013 #

John Lalor

Hi I am a novice user and have a few wp sites hosted on Dreamhost. I am having trouble still trying to access the back end of the sites – pages timing out etc.

Is this the problem everyone is talking about above….

John
- May 15, 2013 #
  
  keith
  
  They had other issues today. I know that because I was impacted too. I’ve been with Dreamhost for a long time and they rarely have issues but when they do they are massive and one right after the other. And then back to normal for a year.
May 15, 2013 #

Mike Dunham

I have a WordPress network (WPMU) install on Dreamhost. I’ve used other hosting systems and rolled my own for a while on Amazon (AWS).

IMHO – Dreamhost is doing as well as host could in this situation. They are not a managed WordPress provider. They provide a wide range of applications for quick install on their virtual private servers and also have shared WordPress available. They reported the problem in their panel header and on their status blog. The service has been hit several times over the past week. They proactively installed fixes that improved the stability of WordPress installs during the attacks. Because of the random nature of the attacks – the IPs are from all over – there is less they can do directly than with some other types of attacks.

I use WordFence and have my settings to lock out login attempts from any IP that uses a username that doesn’t exist. I have removed all admin users and variants. That means a bot can hit once, but the rest of the scripts from at IP will be blocked.

I also signed up for CloudFlare just before the worst of the attack and initialized my busiest domain. It has cut down a great deal on the incoming spam and attacks from what I can see in logs and stats.

There is no perfect defense against this sort of thing, but we can’t just sit back and point fingers at hosting providers – managed or unmanaged. We need to be as proactive as we can – locking down our sites, using security plugins and services in smart ways (they are not all created equal…) and generally taking responsibility. It takes some time to test and learn what works for you. For me, it has been worth the time.
May 16, 2013 #

Michael

Novice at these sorts of things here — how long does this usually go on before it gets resolved? I’ve been having substantial problems for several days, and it doesn’t really seem to be getting better? I can be patient as long as there is a light at the end of the tunnel. Thanks for the info.
May 17, 2013 #

Strebel

We added some additional measures today as well. Nearly impossible to block based on IP’s alone since the origins are so diverse: 10’s of thousands of IP’s.

After the fix today server load dropped by 40% across all pools. A combination of tweaked Varnish rules, increased IP and URL rate limiting, and some cookie management helped. Dropping more bad requests at the network edge before the hit the web stack.

This shit is nasty, no two ways about it.

How do you stay on top of your WordPress game?