Remind me to never Tweet something like this again:
I fired that off around 7:30am on Wednesday, you know, trying to be all positive and everything. “Setting the tone for the day” as those bright and cheery people say you should do.
Three hours later, my email, phone, text messages and Twitter DM’s starting lighting up with things like, “Dude, your site is throwing malware warnings” and “Why am I getting redirected to some Russian site when I go to your blog?”
Hackers!
Bastards.
And here we are, a full 2.5 days later”¦
As you can see (assuming you are actually on the site and not looking at your feed reader right now), things are quite different here. We’re now running on a plain Jane Twenty Eleven theme. That’s because once I found out the site was hacked, I started purging any file that might hold rogue code.
Nothing I did got rid of the redirection problem.
I tried:
A clean install of the latest version of WordPress.
Nuking every plugin and reinstalling them.
Peeking into the MySQL database looking for oddities. Noting the bloat from running a database for over six years harking all the way back to the WordPress 1.5 era, I decided to obliterate the whole thing and do a restore from a recent backup (you DO back up your database regularly, don’t you?).
That was fun.
NOT.
I had to split the backup file up because phpMyAdmin chokes on big files. And 1600 some-odd posts with 22,000+ comments and a boat load of photos makes for a choke-worthy file. (Thank goodness I stumbled across BigDump ”“ a Staggered MySQL Dump Importer. It worked like a charm).
Trust me, dinking around in the database makes me nervous as hell. It’s terrifying to delete a database that contains the last 6.5 years of your blogging life.
Several hours into this madness, I started seeing this in Google:
Isn’t that special? There’s my friend The Google telling the world that my site may harm their computers. That lovely little warning was on every search result returned for PhoenixRealEstateGuy.com”¦
My first thought was”¦ well, I can’t print that here, kids stop by some times. My second thought was, “This is not good.” It was time to call in the professionals.
Enter Sucuri.net. They remove malware and monitor your site for future issues. These folks are good. Just a few hours after dropping $90 (the best $90 I’ve spent in a long time) they had located, isolated and fixed multiple compromised files.
Or at least so we thought.
Fast forward several hours, I forget now exactly how many. The nasty warning in Google results were gone, the site wasn’t redirecting to Siberia. All seemed well in Phoenix Real Estate Guy land. And then, it all came rushing back. Redirections, malware warnings, flags in the search results…
Enter Day 2 of the nightmare.
What Went Wrong
I got back with the wizards at Sucuri and they immediately asked if I had other sites running on the same server. With their direction and guidance, I purged a bunch of old stuff from the server and updated what was left. They then scrubbed everything again.
If you are a regular reader you know I try a lot of different “blogsperiments” ”“ testing new ideas, new sites, themes, plugins, you name it. Most of these experiments fail, some miserably. Some just fade away. A lucky few actually work and get incorporated on this site or others.
The problem is, I am woefully poor at cleaning out these failed and faded experiments. I just leave them laying around on the server. After all, they can’t hurt anything, right?
Wrong!
When the warnings and such returned with a vengeance, the message was clear. The hackers still had a way into my site and were hell-bent on destruction.
By leaving old, and neglected, WordPress installs around, I was leaving the back door to the server wide open for these”¦ sub-humans”¦ to waltz right back in and do their nasty deeds. Some of these sites were running very old versions of WordPress. Versions that had no business being on a live site/server. You see, hackers love to attack WordPress installs, because WordPress powers 14.7% of the top million websites in the world, and 22 out of every 100 new active domains in the US run WordPress (source ”“ State of the Word 2011). Douchebags Hackers like to find vulnerabilities in software that has a large user base. WordPress qualifies. That’s why the WordPress development team is so diligent about releasing new versions ”“ not only to add cool new features, but to thwart hackers.
It is not WordPress’ fault I had live sites that were FIFTY FOUR versions (I kid you not) behind the latest release. Right now, some WordPress geeks are saying, “For the love of God man, 54 versions? You deserve having this happen to you!”
And they’re right. But honestly, I had no idea a hacker could get in through one site and use that to infiltrate another one on the same server. Hey, I’m neither a hacker or a developer, I’m just some dude trying to sell real estate.
Lesson learned: Update EVERYTHING you run. Or just delete the old sites (don’t forget to delete the database too!) I used to think it was OK to be a couple of versions behind. (Clearly being 54 versions behind is just stupid.) Never again. I will forever more update ALL my sites within 24 hours of WordPress releasing a new version. Especially versions that are released as “security updates”. Thinking, “My site is inconsequential, no hacker would waste their time with it” is also stupid. Hackers are stupid, we don’t need to be practically opening the door and saying, “Come on in fellas!”
And at this moment in time, we are clean as a whistle. Google has just removed that ugly “This site may harm your computer” warning from the search results (it may take a few more hours for those to clear completely) and we are back in business.
What I Learned
The afore mentioned keeping the server clean of ancient sites is the primary lesson. Do that. Stop reading right now and see what all you are running. If you have an old site that’s been abandoned, kill it off. If you’re like me and you also try things like adding a forum, putting in a server based chat, putting separate WordPress installs on subdomains, get in there and update the software or purge the files.
You should scan your site for malware regularly. Pay Sucuri Security $90/year and they will run scans every 6 hours automatically. And they perform malware cleanup for you. They are outstanding to work with. Good, fast, and cheap with superior customer service. Trust me on this.
Recommended: Host your main site by itself. Host your toys, tests and experiments somewhere else. Keep the money site separate from the test sites.
There are other things you can do to help keep the nasty people out. I’ve read a TON about WordPress security in the past couple of days. You can also get cyber incident response management so you have the ability to respond to cyber security incidents immediately, click here for info.
I’m going to lock this thing down tighter than the vault at Fort Knox. Here’s a great post on WordPress security ”“ Top 5 WordPress Security Tips You Most Likely Don’t Follow. Read it and heed it.
What’s Next
Things are still a little goofy on the back end. Most notably I can’t get the Facebook Comments plugin to activate. I likely deleted something I shouldn’t have. So for now, Facebook comments are missing. I may or may not bring them back. But that’s another blog post.
You’ll have to deal with this rather drab theme for a couple of days. In a fortunate stroke of coincidental timing, I’m really excited to announce that for the past several weeks I’ve been working with the world class team at Copyblogger Media to do a wholesale redesign of the theme TPREG runs on.
If you’re not familiar with Copyblogger Media and names like Brian Clark, Sonia Simone, and Brian Gardner, well, you should be. These folks are internet studs. Their lead designer I’ve been working with, Rafal Tomal, is a freaking genius and is the man who has served up designs for a few little sites like Problogger.net, Copyblogger.com, ChrisBrogan.com, Jay Baer’s Convince & Convert, and real estate’s own Better Homes & Gardens blog Clean Slate blog (and others).
Just to be in the same design portfolio with these folks is a ridiculous honor, and I can’t wait to roll out the site. It’ll be a BIG change, but it will set the stage for some really cool stuff to come.
Any day now, we’ll be a full-blown custom Genesis Framework site. Due to this hacker mess, we’re bringing it online a little sooner than expected, so it may be missing some minor functionality here and there, but it’s going to be awesomeness. This will cause a little downtime as I’ll have to switch Domain Name Servers over to the new host, but that’s downtime I have no problem with. This may happen very soon.
Thanks for Bearing With Me
This has been an excruciating couple of days. (Having to take an online traffic school course in the middle of all this crap sure didn’t help!) I appreciate all the folks that let me know about the issues and things they were seeing. Special thanks to Jon and Inna Hardison at HA Media for a call and offer of their considerable experience and skills! Sucuri Security along with Brian, Rafal and Derick at Copyblogger have been a godsend. And the support of my wife and kids, our agents and lots of friends as I lost my mind over the past few days helped save me from finding a bridge to take a header from.
I’m sure there are other broken things. But most importantly, the scary career-limiting warnings are gone. The site is clean. We’re back online with bigger and better things to come!
The Phoenix Real Estate Guy was started by Jay Thompson, and I have a little blogging problem... he liked to say...
Hope you flushed it all. Hackers suck. Been there.
Thanks for the security info!!! I’d also suggest looking into BackupBuddy as another layer of protection. You can set it to back up your whole WP install and then offload it to a different server. I’ve used it to restore borked sites but also to easily move one from one server to another.
I got BackupBuddy — after this happened…
Doh! Congrats on the new site design – looks perfect!
Hi Jay,
Welcome to the malware club, it sucks. I experienced this 2 years ago X many sites. Even had a detective drill down to the hacker being someone I know. Brute force attack is what they called it. The detective thought it was of interest that the hack happened during NAR here in San Diego, when I had planned to attend. Sites down for a week.
So all the sites were moved to a new server, stronger firewalls. About 100 clients on server now. Starting to see weird activity, thought we nailed the bastard’s code. Nope, learned the hard way that we brought the injected javascript with us. Likes sites that are not updated, poor user/password combinations and yes, all those fun plugins just to try.
Well, it all happened at once this time, snaking it’s way through the plugins and sites were now infecting with malware, who knows how many. Google hates me.
We learned our lesson, passwords are like Fort Knox, we only use plugins that we know to be safe and keeping sites updated. We quickly fixed everything. Google’s good graces again.
This is still not enough protection to be honest. I am asking all clients to move to their own server space. We are in the age of technology and there is no excuse to host your WP site with an outside company. We have clients who still pay for technical support, but hosting with GoDaddy, MediaTemple or a variety of companies is inexpensive.
If you want Word Press, which is the best, take care of your site. If you are reading this, then I feel confident that whoever you are working with will help you move your site to a new server. This is not a sales pitch, if you want help to go to a new server go to my site and contact me.
So no
Thanks Cherie. I’ve got some people working on the server move.
Sadly, I don’t think there is anything one can do to make a site truly bullet-proof. The hackers can always find a way. All we can do is make it as difficult as possible. If nothing else, they will move on to a less hardened site…
Idiots.
Wow, what an ordeal. Thanks for sharing your experience, I hope I never have to go through anything like this. I am off to update and/or remove some old WP sites.
Sorry to hear about the ordeal Jay. Glad to see you are up and running again.
Ouch.
I went through this a while back and I know it’s no fun. Like you, I’ve tried to make my blog harder to get into than a loan modification program. One thing I did was install the WordPress Firewall program. Since hardening the blog and adding that plug-in, I haven’t had a single issue. Now watch, just because I said that I’ll get a bajillion SQL injection attacks…
One thing I’d like to see you do, when you have the time, is a blog post explaining how to use that BigDump importer. Someone (cough, not me) MIGHT get some use from it.
The instructions on the BigDump site are pretty good Wayne, but I’m sure I’ll have some follow up posts….
Really sorry to hear you went through this PITA. I’m in the process of procrastinating moving all of my sites to amazon cloud. The nice thing about cloud is that you can revert to older disc images very easily – you’d lose a few posts/comments, but wouldn’t have to spend 3 days fixing this bs. Here’s an Austin based company that takes care of it for you (I have no affiliation, btw.) http://wpengine.com/
Amazon EC2 is the service, btw. Rackspace offers a similar, more expensive/supported service. The wpengine guys are the most expensive/supported (they run on EC2, I believe.)
Wow. I’ve been hacked in the past, but apparently not by hackers as smart as yours! Thanks for sharing so many tips for how to lock down our sites.
About a year ago, I implemented a sandbox site. That’s where I do all of my experiments, because experiments are important. That’s also where I test upgrades so I don’t accidentally crash the money site.
Bummer you had to go through this, but great that you won’t ever go through it again. And your new theme announcement is very cool, indeed. I know the work of Rafal Tomal and can’t wait to see the brilliance he has created for you.
I have a sandbox site too Charlene, but I put it on the same server as the primary site. In hindsight, that was a bad idea.
Rafal is good. Really good!
In all seriousness, your site is popular enough to be hacked. Thanks for sharing some of the lessons learned…even if they came about through your blood, sweat, and tears (so to speak).
Have a brewski for me.
I had several brewski’s Bubba… 😉
Sheese Jay sounds like the making of a movie script!
Or the making of a living nightmare…
Jay, Working in web hosting I see this daily, Sometimes hourly. I just cleaned a site that had an infected .htaccess file. I read the whole article and just nodded my head. Great advice. If I can add one minor thing… Strong Passwords. Use a site like http://www.strongpasswordgenerator.com to get an idea of what a real strong password looks like, Then make yours the same way. Keep up the good work.
Agreed on strong passwords. And while it may seem stunningly obvious, different passwords for each site as well…
What a week you have had, Jay! 🙁 Sorry you had to go through all that! Thanks for posting that, though. We are getting ready to delve into WordPress for the first time and this is great information to have.
Wow, I am certainly going to heed the tips on securing a website. Your experience as troublesome as it may have been will motivate a lot of DIY agents to take extra precaution.
I agreed with comments above and taking the advice to delegate the money site from the experiments.
Bummmerrr! Thanks for the full explanation for us all to learn from. I have had a DNS attack but nothing like yours. I immediately installed a full-site, daily automatic back-up off site.
Maybe I missed it but I am curious if you know or have a good guess about the motive for the hack. Traffic? Pranksters? Your competition?
Seriously, another good wake up call to call of us. Thank-you!
I have no idea what the motivations were Larry. Who knows what drives these idiots to do what they do. The redirects sent them traffic, but it was traffic they wouldn’t want. And my site could have infected others, starting a “snowball effect”.
I think they are just evil people that enjoy wrecking havoc…
Super post!! I really like this site, and hope you will write more.
Guess what? Since I posted this, I was reinfected, DO NOT GO TO MY SITE PLEASE. Working with Sucuri right now. This is a bold reminder to take charge of your website and be on your own server!
I’ve never understood what motivates hackers. I know what I would do if I ever got my hands on a hacker who screwed up my site. This is how I feed my family. No mercy.
@Cherie: I do hope that you prosecuted the hacker/competitor. That is absolutely ridiculous–have they nothing better to do?
@Jay: WOW, what a saga–I know you’re glad to have this almost behind you and to have learned a good deal from it too. At least it didn’t happen during our busiest season! (Hey, there’s a silver lining in every cloud, right?!) Thanks for the tips–off to check out a couple of these now.
Hey Jay –
Firstly, nighmare!
Next – thanks for the detailed article – as someone new to and tinkering with WP it was interesting to read how you dealt with this.
Finally – This design looks great.
cheers,
phil
Amen, brother. I believer I was a victim of these same hackers, though perhaps there are many that redirect to Russia. Who knows?
After everything I tried just yielded re-hacks hours later, I quit. I migrated the entire thing over to Squarespace, deleted everything on the original host, and I haven’t looked back. Less fancy, sure, but at least someone else is in charge of my security.
My site is clean now, Google reconsidered quickly.
I’m just a newb but when my site shot up like a bullet, short time later my hotmail account got hacked…wonder if there’s a connection?
It’s stories like this that force me to maintain several sites. I had a crippling issue with one of my websites a few years back. Ever since then I’ve maintained multiple income producing sites. If one goes down, there’s still an income stream coming in from the others.
Thank goodness you are smart enough to back up your site and work through the issue with a little help.
Sorry to hear about your grief, we are moving servers right now and while we were not hacked I can appreciate your experiences as I am running into file issues and added issues from a previous IT guy who added security to my site that we are trying to undo to be able to run as a separate “normal” WP site.
Unbelievable. I am finally coming up for air. Jay, thank you for calling me back quickly. It was a brief conversation, but the words that you said to me “And I bet they think it’s your fault” carried me through these last 2 weeks.
I am not the end all, no web designer, hosting company, etc. knows everything. I was called horrible names, a thief, ripoff, idiot, you name it. This is the 3rd go round with malware. The final decision was to be sure that everyone understands that this can happen. It happens. Emails can get hacked, credit cards, identity stolen.
Lessons learned were hard, but valuable. It was not just Word Press sites. Please read this last line if you read nothing else.
Be on your own server. Isolate yourself. Grid servers are cheap, there is no excuse. Run malware protetion at all times. Thanks to this post Jay, sucuri.net saved close to 100 sites that went down. Within hours they scrubbed and monitored for injected scripts. We still see activity of malware trying to get in, we alert sucuri.net and they remove within hours.
It’s not personal with the hackers. They don’t even have to be particularly smart, just persistent. Find an open doorway, easy password. NEVER have your computer remember passwords. Use very strong passwords, 15 letters usually. Google strong passwords and you will find sites to generate for you. Change your passwords often, especially if you bank online, use Paypal, anything that can be compromised and steal information.
Sony was recently hacked, we are all vulnerable. Hospital records get hacked, governments get hacked.
Protect yourself and don’t wait until a catastrophic meltdown happens. If you are reading this, you have the upper edge.
It’s not personal for hackers, it’s business. They can easily write scripts from Russia that inject redirect links in your code. These redirects can link to pharmaceutical companies, and worse.
If your site is compromised, sucuri.net will help you. You will need your ftp login.
If your site is blacklisted, you can request consideration from Google http://www.google.com/support/webmasters/bin/answer.py?answer=35843.
It’s the wild west out here and only the brave survive. Strap on those spurs and hang on for the ride.
I can start to work on my site in about a week and do what I love!
For interest’s sake look for the book “Net Crimes & Misdemeanors” by J.A. Hitchcock. This book helped me enormously! It may be in its 10th printing by now or obsolete… not sure.
In any event, I was stalked for 8 years from 2000 till 2008 and after 4 cyber police reports (“duh, what is cyberstalking??”), 8 more police reports, letters to the State Attorney’s Office, and being forced to leave email completely for 8 years, go through 11 computers, including a Mac which was hacked in a record 13 days, have entire photo libraries sucked out of my computer and end up in over 14 nations all over the world (found the locations on a file in my computer), have an entire user account simply disappear with hundreds of files and documents, and lose all business contacts, my entire real estate business, and practically lived at the library (for anonymity)… to read your blog makes me feel less alone. In the early 2000’s NO ONE believed me. I was hobbled, crippled and alienated for over 8 years, and just as suddenly as it started it stopped, like an earthquake… but ever ready for the latest attack and anxious at every boot… and even now I am just learning how to use the internet and a computer and ready to jump back into a website for real estate.
My question is this: I have the option to go to a locally known website creation company called “ALaMode” with all the formatting and linking and real estate specific IT options. I do not know most of the vernacular you all just used, so I am uncertain as to where to go to build a website, what server, desiring to have all the social media capability, mapping, interaction and live feeds and video etc… Do you have a recommendation for someone or company that would develop a website (budget conscious right now as everyone who is just getting started is) that I would feel confident about?
And 2, what does “having your own server” mean?
I know you have much more important things to do than respond.. but this is my first foray into a blog and request. I am so safety conscious but do not know what is out there…
Thank you for your time..
God bless you.
Kate
ps: the email “Hushmail.com” has total encryption (and choice of non encryption), and is a great email alternative to Gmail, where I have been hacked by a Korean person who added their email address to be a link to my main email account… just discovered that last week… very scary.
thanks!!!!!!!!!!!!!!!
Hey Jay,
Sorry to hear about the hack, but finding Securi.net was clearly the silver lining.
Dre Armada, who co-founded Securi.net did a great presentation on WP security at WordCamp Chicago. Her is the link to the WordCamp TV video of his presentation. It should be a must view for anyone using WP. http://wordpress.tv/2011/11/19/dre-armeda-wordpress-end-user-security/
His #1 mantra is update, update, update. He also advises to NEVER screw with the core. Unfortunately there are a large number of agents on sites where the WP core has been modded and these agents are completely unaware.
Sorry again for your experience, but thanks for sharing and educating so many others out there.
Hi Bob,
I was literally dealing with around 100 sites that were infected. Sucuri.net and Dre are my heroes. It took several rounds of scrubbing and hardening the sites, but this is now under control. Everyone is on their own server, which is the way it should be if you have Word Press. More control over your site means more responsibility.
This has actually been a great experience. I would have told you different 30 days ago, but it brought light to never use plugins that you don’t know, keep your plugins up to date and make sure your Word Press site is current. If you are running multiple WP sites, you may want to use http://www.managewp.com which will alert you when you need to update items.
I love Word Press and will continue to look for the best ways to design, code and keep them safe.
Merry Christmas,
Cherie Young
I havent tried Managewp yet, but Vladimir knows his stuff, so I have no doubt its a cool tool.
Unfortunately there are still a few out there who host sites for others on modified versions of WPMU with old plugins and they dont get how much at risk they are putting their clients.
Yeah, it’s ok to work with a company that uses the multi-user platform, but you are limited in what is offered to you. Many of these companies will allow you to take your site with you after a certain time or buyout.
I think most of our clients were happy to have their sites moved to their own servers and even cut down costs of hosting on my server, which is completely ok. I will never take that responsibility again.
Being on your own server is so easy and cheap. Don’t try to run 20 sites on one grid server either. Work with one hosting plan for one site. You open yourself up for all sites to be infected on one server, versus isolating problems to one site, one server.
Happy Word Pressing, I LOVE WORD PRESS 🙂
Cherie
I hate when I come across posts like this, bashing hackers.
Hackers don’t suck. Crackers suck.
There is a really well-written Wikipedia article on the history of hacker culture, and why people should refer to malicious computer hoodlums as “crackers” and not “hackers”.
http://en.wikipedia.org/wiki/Hacker_(programmer_subculture)
Steven Levy’s “Hackers: Heroes of the Computer Revolution” also provides excellent insight into the hacker culture.
I suggest you read the article and the book before throwing the term “hacker” around without so much as a second thought. It tarnishes the name of all the good hackers out there.
Michael –
I certainly didn’t intend to offend the “good hackers” with this post. I think it’s pretty clear I was speaking of the malicious variety.
Thanks for the link to the Wikipedia article. I did read it. The very first paragraph mentions that not all “hackers” differentiate between “hacker” and “cracker”.
Also, like it or not, the term “hacker” is quite commonly used to indicate maliciousness. (Reference definition 3a of “hacker” at Dictionary.com for example. Or this page. And of course throughout the mainstream media.) I’m certainly not the first person to use the term in relation to what happened (not that that makes it right, or wrong, to use the term in that way).
There are dozens (hundreds?) of other such references to malicious “hacking”. Probably dozens or hundreds of references as well to “cracking”.
Again, no ill will was intended to those that are good hackers. I guess I’m guilty of adopting the common media-tainted usage of the word “hacker”. I’ll keep that in mind if a malicious attack happens again to my site (hopefully it won’t!)