This situation is written from the perspective of a project manager.
Recently while executing a routine task, one of our developers noticed something out of place and brought it to the attention of the team. A process began that lead us to discover something deeper and more sinister. The site were were working on had been compromised by hackers. It seems we hear about such things so often that we easily dismiss without understanding the fallout.
My role as the project manager required me to immediately understand what this meant for our client and for Mediacurrent, then to formulate a strategy to rectify the issue. This was the beginning of what was a very complicated, puzzling, frustrating and eventually amazingly enlightening few days.
Reaction and Planning
All that was known at the time was there was a compromise, it was intentional and malicious. The next steps involved quickly assembling a team, contacting the client and rapidly developing a plan of attack.
The client was informed and the site was taken offline (Something no project manager wants to have to inform a client). I organized together my dream team; the Sr Architect assigned to the project, our top security experts, Solutions Architect and the VP of Development. Working with the lead architect I assigned each person an area of analysis and investigation and set the immediate expectations.
We divided the team into analysis areas; determining attack vectors, investigate files and folder structure, analyze for database compromises, and document known vulnerabilities.
All about the “W”s
One of our earliest findings was frustrating to say the least, the attacker was able to eliminate or mask most traces of the intrusion, this was not some random attack, it was professionally developed and executed. To resolve this issue we needed to know when it occurred, where it came from, what was done, how it was accomplished and if possible why it was done.
What Happened when and from where?
It didn’t take long to have the what solved. We found what appeared to be image files with embedded php code, the entity table of the database had been altered and users had been created in the system. We traced the attack to China, uncovered when it occurred and documented all of the alterations made. Nice to have the first 3 “W”’s done, but we had more needles in this haystack to find as we hadn’t solved for how this happened.
Why do this?
As we continued we uncovered something that was a tipping point in eventually solving the issue. We found traces of suspicious URLs, and with help from an analysis of Google Webmaster Tools we uncovered these URLs were dynamically inserted to the page delivery mechanism without leaving a trace in the database, One step closer to another “W”.
The malicious links lead us to why. They were being delivered to google search results and promoted in relevance by other traffic from presumably compromised sites in the attacker’s network. We were able to conclude this scheme generated some form of financial gain and was the main purpose of the hacking.
How was it done?
The how of course is the most important and interesting aspect. In order to prevent this from happening again, we needed every open vulnerability explored and a solution implemented. We discovered the exploit that was used to initially gain access used a SQL injection vulnerability. This allowed the hackers to load the malicious fake image files, compromise the database and create admin users, lay the groundwork for hijacking traffic and then cover their tracks.
The Project Manager’s love
Getting to the point of understanding all 5 of the “W”s is what makes a project manager like me giddy. This was clearly a terrible thing, it was not planned for, I was not anticipating something of this level, but leveraging my experience and Mediacurrent expertise I was able to treat the issue as a mini-project. We organized a team, held a kickoff meeting, determined roles and responsibilities, created and assigned specific tasks, determined solutions and delivered a conclusion. At the end of that day, we were only half-way done, but we again had a plan.
Additional Resources
How to create a more findable Drupal issue | Mediacurrent Blog Post
Locking Down Drupal and Managing Security Breaches | Video
How Much Documentation is Enough? | Mediacurrent Blog Post