My collectors’s request rate the week of PHPShell’s disclosure against the events of that week.
I’ve been running a country-scale collector on China’s government web servers, gathering the documents they publish that traditional search engines never index. I saw my collection rate fall after the PHP2Shell vulnerability disclosure, and wondered if that might be indicative of the Chinese government enabling a wartime reserve cyber-defense mode on the Great Firewall. With active telemetry on 147,000 Chinese government servers, I can actually measure how they respond to a doomsday vulnerability.
To cut to the chase, I don’t think I saw any wartime reserve cyber-defense mode deployed (but I’m not throwing any exploits). I think it was indicative mostly of a slower link on my Hong Kong boxes, my collection backing off politely during the Beijing evening, and some of my collectors finishing some big jobs. So this is more a study of how the PRC responds to doomsday vulns.
What China did, and when
One of my favorite things about running active collectors is that you find the site-specific search interfaces. Those let you hunt for needles in haystacks that the regular search engines never index. A surprising amount of cross-organizational coordination shows up there. So we start looking there. Is there any evidence that the Chinese government or companies got an early heads-up or started to respond to the vulnerability before the leak?
The first Chinese warning I can time came from a company. Tencent Cloud posted about ten hours before the fix was released. The same Beijing day, the two national vulnerability databases logged the flaws and disagreed about them: CNVD (run by the Chinese CERT) rated it high; CNNVD rated it critical. To be clear, I didn’t see any indication that they had an early warning let alone started implementing defenses early.
Most of the state spoke the next day, 15 to 20 hours after the fix was published. The national CERT (CNCERT) posted a risk notice and then a full bulletin; these were its first public vulnerability bulletins since Log4j in December 2021. The industry ministry’s vulnerability platform posted a critical notice. The Ministry of Public Security’s notification centre warned on WeChat and recommended upgrading. CNNVD’s own bulletin came two and a half days after the fix. Turtle races.
All of them said upgrade. For systems that could not be patched, CNCERT and the industry ministry advised isolating them, restricting access, or taking non-essential systems offline for a while; CNCERT also suggested firewall rules against the exploit’s request pattern. I didn’t find any compulsory orders, and I found nothing from the Cyberspace Administration of China, the internet regulator.
There were sixty-three PRC government hosts running a vulnerable PHP version that I collected from in the week before the vulnerability was announced, and again in the ~90 hours between the patch being released and me writing this. Only seven were patched. Call it 11%. That’s far better than the 4% of Chinese non-governmental servers fixed. Most of the servers ran very old versions of PHP, so most of these “patches” were jumps to a new PHP line rather than a point release with the fix. For instance, one city health commission went from PHP 5.6 to 8.2. PHP 5.6 was released in August 2014.
I only saw two PRC sites being taken offline; and that is in the normal range for routine maintenance. On 21 October the Netherlands, Finland, France, Germany, Belgium and the Czech Republic took government sites offline. China kept serving.
Under all of it sits one fact. Nearly nine in ten Chinese government hosts that announce a PHP version run an end-of-life line, and so do Chinese commercial sites. Those lines get no new releases, so the fix arrives as a migration or as a backported patch that I can’t see without exercising the vulnerability directly, which I am not going to do.
What I think they should have done
The Chinese government web estate is truly horrendous. China already has the Multi-Level Protection Scheme (MLPS) (网络安全等级保护制度), roughly the equivalent of the American FISMA plus NIST 800-53 plus FedRAMP-style accredited assessment. In brief, it requires yearly audits on important systems and less-frequent audits on less important systems. One of the main purposes of an audit is to act as a forcing function to periodically pay down technical (and paperwork) debt. Something is wrong in the system if people are running ten-year-old software. It’s not that the PRC needs to update their software, it’s that they need to fix what broke in their accountability processes.
Speak once, early, with one severity. Seven national items over three days, two severity ratings, one database two and a half days late. They have a bureaucracy problem.
Disseminate the information. No local government relayed the warning that I could see, and CNVD’s own pages sit behind a captcha. The Chinese Communist Party (CCP) is otherwise very good at flowing information down throughout the organization. Caveat lector: I haven’t measured how quickly the CCP can flow down information, just that I have seen instructions issued by the Politburo captured in even podunk provincial record-keeping.
If you advise taking unpatchable systems offline, count whether anyone did. From outside, nothing happened at scale. Only the government can measure compliance, and only from inside. They already have records on all of the government web servers. It would take just several minutes of vibe coding to scan every government web server and make a list of those running vulnerable software. It would take longer to WeChat all of the organizations to take their vulnerable web servers offline.
I went into this looking for the Great Firewall’s wartime reserve cyber-defense mode. I found a country’s webmasters who still aren’t patching. That does not bode well for Chinese cybersecurity as the pace of vulnerability discovery and exploitation continues to accelerate.