“May 19th Network Breakdown”triggers the warning on network security once again.At 9:00 p.m., May 19th, 2009, network disruption overwhelmed a wide range of China. Netizens from many cities and provinces including Jiangsu, Hebei, Shanxi, Guangxi, Zhejiang, Tianjin, Inner Mongolia, Heilongjiang, and Guangdong said that they could not open webpage. According to the latest report of the Telecom Assurance Bureau of the Ministry of Industry and Information Technology of the PRC, this issue is caused by an attack on the Domain Name System (DNS) of baofeng.com, which results in severe congestion on the recursive domain name servers of main telecom operators by sending a large amount of anomaly requests to them.
As a leading enterprise in the network security industry, NSFOCUS accompanied its customers in the exciting two hours immediately after this issue happened and fought against DNS DDOS attacks with great energies and tactics.
Receive Orders
At about 9:30 p.m., May 19th, Xiaolin, security expert of NSFOCUS, was on the way home when he discovered a weird thing: cyber bars that used to be loaded with people at that time of the day were then empty, while some of the bars were crowded with noisy and indignant people. Xiaolin wondered what the fuss was about.
Suddenly, a call came in, interrupting Xiaolin’s thought. The call was from a telecom engineer, who told Xiaolin anxiously that a significant network exception occurs in his company’s network that needs Xiaolin in the IDC room immediately. Xiaolin rushed to the operator’s IDC room and followed their staff to its Operation and Maintenance Center.
Internet in Danger
The Customer Service Center of the telecom operator was next to the Operation and Maintenance Center. Numerous calls ring now and again, to which the customer service personnel can only repeat the phrases like “excessively slow speed in network access”, “failure to get emails” and “network access denied”. The Operation and Maintenance Center was like daylight, with experts in network, system and service fields present. Dean of the Operation and Maintenance Center was very serious, and convened an emergent meeting soon after everyone was there to summarize the problem. Technicians from network team responded that no anomaly occurs in CPU utilization or data traffic of network devices; technicians from system team also found no abnormity in RADIUS server or customer bandwidth authentication and authorization; but technicians from service team found that access to the Internet became extremely slow or totally denied for a large number of users of MAN bandwidth.
Xiaolin took a note about on-site data and kept in touch with engineers of NSFOCUS Technique Support Center. A moment later, two NSFOCUS engineers arrived at the Operation and Maintenance Center.
Emergency Response
At this time, Technique Support Center of NSFOCUS headquarter were also very busy. Right about 9 p.m. of that night, NSFOCUS headquarter received reports from branches in south, north, and east China, explaining that vendors’ networks were troubled by malfunctions and that services for some of Internet users were disrupted. The Technique Support Center reported this situation to the manager of the emergency response team, and after analysis of the existing data, determined that this was a severe Internet security issue. The manager notified technical experts of all branch offices nationwide at once to form an emergency response team, and made preparations for potential security problems in each area.
Important Turn
Upon analysis of the local MAN malfunctions, the on-site technical engineers communicated with their company’s Operation and Maintenance Department about the problems met in the local network and asked about the situation of the backbone network. Xiaolin checked relevant security systems and found an alert from the Collapsar security prevention device deployed at the DNS network egress warning against excessive increase in DNS network traffic. Xiaolin enabled the packet capturing function bundled with Collapsar device, analyzed captured packets, and found that over 50% of the DNS requests are targeted at an Internet service provider. Later, staff of the Operation and Maintenance Center assessed DNS and discovered the overload state of the DNS server group and a huge delay in DNS response.
After Xiaolin told NSFOCUS headquarter about his discovery, they discussed on it and made an immediate solution, which was introduced to the operator. Bureau staff and the group company of the operator held an emergent meeting and then asked the Internet service provider if its system faults. The provider answered yes.
Problem was located. NSFOCUS technical experts discussed it with the bureau staff and formulated an emergency solution: first, enable the pattern matching policy on Collapsar to filter DNS requests that target at the Internet service provider, so as to reduce pressure on the DNS server; second, on the local DNS server, set strong resolution policy about relevant domain names of the Internet service provider to ensure the best operation of most applications on the basis of minimum cost.
Eliminate Risk
At 11 p.m. that night, after the emergent policy was distributed, DNS network traffic reduced from 150M to 10M, and DNS query requests decreased by 70%. DNS recovered from the disaster rapidly, and the Internet access service resumed to normal.
In addition to Xiaolin, emergency response engineers of north China were also occupied with network exception analysis, packet capturing, data analysis, etc. in the IDC room of a telecom operator. Through data analysis, NSFOCUS headquarter technicians found that this telecom operator was harassed by the same massive DNS traffic attack as the southern telecom operator. However, the DNS traffic was still growing rapidly. To avoid a possible crash in DNS system, NSFOCUS headquarter communicated with the on-site emergency response engineer, made an emergency handling solution, and launched the corresponding prevention policy soon after the operator’s confirmation. A severe Internet service interruption was avoided after the policy was distributed.
Telecom engineers in east, south, north and north west China contact their group companies, and adopted emergency solutions in each province to prevent against DNS attacks. DNS service and Internet services resumed to normal after that. Until the morning of May 20, Internet operation basically recovered all over China.
Postscript
It appears that this issue is caused by massive DNS query requests. But in fact DDoS attacks are common in Internet, especially DNS server-targeted DDoS attacks. Attack vectors increase dramatically, mainly including buffer overflow, massive traffic flood, IP address spoofing, UDP flood on port 53, and recursive query caused by massive query of random domain names.
NSFOCUS experts point out that deploying traffic scrubbing systems at backbone networks of an operator can help scrub DDoS traffic, prevent against various DDoS attacks by using Anti-DDoS products’ unique prevention algorithm like pattern matching and IP address reputation mechanism, and provide efficient and timely security assurance for the operator’s DNS server under attack. |
