When Blackholes backfire…
According to our current scientific folklore nothing will ever come out of a black hole, no matter or particles, no light, no information. But black holes in networking can backfire from time to time. Of course I’m talking about “black-holing” Internet traffic, a strategy often used on backbones to defend against attacks, specifically flooding, DDoS and the like.
Here is a little story about black hole routing that actually happened, the involved ISP and the victim will not be disclosed for hopefully obvious reasons:
Black Hole Routing
The specific case I want to talk about is not the common black hole routing explained nicely by Jeremy Stretch on Packetlife which drops traffic to a victim of a DDoS attack. Instead I focus on the “advanced” version of this:
RFC 5635: Remote Triggered Black Hole Filtering with Unicast Reverse Path Forwarding (uRPF)
which drops traffic from the supposed attacker. In my eyes a much more attractive strategy than blocking traffic to your own resources or servers. As Jeremy Stretch states:
Of course, the victim is now unreachable, and we’ve effectively assisted the DDoS in accomplishing its goal
But did it ever cross your mind that “black-holing” the attacker with unicast reverse path forwarding can also assist the bad guy? Here is how:
Lending a Black Hole to the Attacker
Let’s see what happened recently on the network of a large Pan-European ISP (several Million Internet users were affected):
A DNS flooding attack was observed against the ISP’s public DNS servers, which has exhausted the resources and put the DNS servers in a critical state. DNS replies from legitimate users where heavily delayed or timed out and of course the ISP’s customers started to complain about the issue. The culprit was easily found, a single IP address was the source of the DNS-request flood.
As a quick fix the attacker was black holed and the situation stabilized. Obviously you don’t want to black-hole the victim (your DNS server) in such a case.
So far so good… but then something happened which was a little surprise for everyone:
The Black Hole Reports in
Soon after the incident the operator of a large online dating service contacted the ISP and asked why the dating site was not reachable from within the ISP’s network. Many users of the dating service complained about timeouts, page not loading and so on. And all the complaints came from the ISP’s customers.
I’m sure that you can guess it by now:
The server’s IP address turned out to be the supposed attacker.
And I’m sure that you are wrong with your second guess:
NO, the server was not compromised.
Not only that, the server never sent out any DNS requests to the ISP’s DNS servers. What actually happened was a custom tailored attack against the online dating service. Someone (presumably a black-mailer or the competition of the dating service) ordered an attack and the black-hat who did this used a spoofed DNS-flood as the weapon of choice. Of course the spoofed source-IP was the victim’s IP address and the attack turned out be “super-effective”: several Million Internet users all over Europe could not reach the dating service because of the black hole routing.
A similar incident was identified that happened a few months earlier, that time it was a server, which, politely described, offered “adult entertainment”.
I was asking around some friends who are quite knowledgeable about the scene and got confirmation that this kind of attack is becoming “standard repertoire” of the black hats: exploiting defense mechanisms like shunning, black-hole routing or black-listing as a weapon.
Lessons Learned:
- Never trust a source-IP
- Be careful with aggressive responses (like shunning, black hole routing etc.)
- Look twice before you shoot, it could be an innocent bystander.
Regards,
MiKa