Roughly two weeks ago, Raven Tools surprised many in the SEO industry when they announced that they would no longer be providing ranking data. This came as quite a shock to me, as I’ve used Raven specifically for their search ranking reports in the past. They do provide other tools as well, but I never really spent much time using them.
Raven obviously didn’t make a major decision like this out of the blue. Search ranking data is dependent upon services that scrape Google’s index, and this is against Google’s Terms Of Service. Raven decided to remove this functionality per Google’s request, as they were threatened with having their access to the Google AdWords API revoked. If they’re no longer able to query the AdWords API, they’ll lose access to accurate traffic and CPC data. Beyond that, Raven obviously wants to maintain a positive relationship with Google moving forward, as the success of their company depends on it – they don’t want to “bite the hand that feeds”.
Raven’s decision had a lot of people in the industry talking… And then ahrefs.com made a similar announcement. In order to maintain their access to the Google AdWords API, they were forced to remove their keyword analysis tool, which was also dependent upon scraped data. In my opinion, the keyword analysis tool was not ahrefs.com’s biggest strength. I use them for their backlink data, however, that doesn’t change the fact that this is a disturbing trend.
And It doesn’t stop there. SEOMoz, who provide what is arguably the most popular/recognizable set of SEO tools available, have had their access to the AdWords API revoked as well. Because of this, AdWords data (search volume) is no longer available in their Keyword Difficulty Tool, which I use quite frequently. SEOMoz did not elaborate on why Google revoked access, but they do have their own rank tracking software, so it’s easy to speculate what Google’s issue might be.
Google has strong-armed SEOMoz in the past – they made them remove Google PageRank reporting from the MozBar (as detailed here). However, SEOMoz might be taking a stand this time. They’ve stated that they don’t plan on changing the data provided by their tools (here). They’ve also said (in the comments here) that they are currently looking for other providers of keyword search volume.
A War Against SEOs?
Some will say that Google is declaring war against SEOs. It wasn’t that long ago that Google began encrypting search queries from logged in users, taking a big chunk (roughly 40%!) out of the referral keywords that we saw in our analytics (the “not provided” debacle) in the process. They told us that this would only affect “a minority of [our] traffic”, but that didn’t end up being the case at all.
So the question remains: What is Google’s motivation here? As far as I know, scraping the index has always been a violation of their TOS, so this just seems like a classic case of selective enforcement. They have not consistently pursued violators of this rule, so when they fire off a warning shot like this (crippling the functionality of some of the most popular SEO tools out there) people take notice. Are they trying to make a statement?
I would speculate that Google wants SEOs to stop focusing on things like rankings and focus on creating an overall better experience for users. While they have denied any sort of crackdown, something tells me that this most recent strike won’t be the last. Who is next? SEMRush?
Google’s Failure To Provide Access To Data
I think that this quote from Brian LaFrance (Authority Labs) says it best: “Google’s position to block access to their API for those who use scraped data isn’t unreasonable. What is unreasonable is that they know full well how important SERP monitoring is for users but still fail to offer any kind of accurate API for that. They have no problem leeching off of the content of others but do have a problem with others using their content.”
SEO In 2013
At the end of the day, traffic is (arguably) the biggest metric of success in an SEO campaign. As we see more personalization/localization in the search results, rankings are beginning to lose their significance as an indicator of success. But as any SEO will tell you, clients still want to see rankings, and they’re still an important signal.
In this post, the CEO of Raven Tools goes more in-depth on his decision to stop using scraped data, and speculates on what’s in store for SEO software companies in 2013.
Raven will continue to offer their rank tracker functionality until Jan 2nd, and ahrefs.com has made their keyword research tool free until January 11th, when they’ll be closing it down. If you use either of these tools regularly, hopefully you’ve already started looking for a replacement.
It may be worth noting that Raven was using a third party for their rankings data – Authority Labs, who still offer this data directly, and are working on a solution for importing historical Raven tracking data. I have no firsthand experience with Authority Labs, though I’ve always been happy with the ranking data provided by Raven in the past.
Looking For A Replacement?
If you’re looking for a new rank tracking solution, I’d recommend AWR (Advanced Web Ranking). I have used both Raven Tools and AWR extensively, and I prefer AWR. It’s not a web-based app; you install it locally and run the reports when you choose to. You can do a lot more with the data from the get-go, and it’s traditionally more customizable than Raven Tools. If you’re not looking for a local solution, I’ve also heard of people having success with Microsite Masters as well.
A number of other rank checking and competitive research tools are offering discounts/free trials – you can read up on them here.
Regardless of the message that Google is trying to send with these actions, as long as there’s a demand for rankings data, people will come up with ways of reporting that data. Scrapers will prevail. It will be interesting to see what happens over the next few months. I know I’ll be keeping an eye on Google and the major SEO tools.
I think you bring up some good points here although the biggest loss for Raven tools as you touch upon is that historical data will be lost and that’s going to screw agencies up seriously. Also if I think of the crack down as an Algo update and but less severe. Google has been trying to get more people to use Webmaster tools for years and they keep expanding what they offer within WMT so I think of it more as an opportunity to go to the source and see what is happening. Granted I still use SEOmoz for tracking keywords but there is a wealth of info within WMT that gives me more up-to-date info than i can get from weekly crawls.Don’t get me wrong it’s annoying that Google does this shit but Google is also the reason we have jobs so it makes sense that they want us to play by their rules, unlike FB who just announced that they own everyone’s photos. http://seekingalpha.com/currents/post/725821
They’re going to have to offer better ranking data in WMT if they want people to use them as a primary source for SERP rank data. WMT currently provides “average position” on broad match. Way too vague to be useful IMO.
John, enjoyed reading this.
Dan, we’ve made it super easy for customers to export all of their ranking history: http://raventools.com/blog/easy-export-historical-serp-tracker-rankings-data/
I’ve found another tool that is also quite good for developing this info. It’s server based, open source, and can be scheduled to run reports daily for keyword position.
It’s a bit more of a pita to install that some options, but its free and open source. Between seopanel and piwik, I get rather detailed data for free off my own server.
Great post, it’s nice that you actually took a look at the whole picture. Definitely a frustrating trend coming from Google…
Like you stated, at SEOmoz, we don’t have any plans to change our data service. And when our AdWords API was revoked, we weren’t given a reason for the revocation or the choice that Raven was to stop delivering SERPs or lose the API.
Thanks Jon, I heard that something was on the way. I’m sure this will make many people and agencies very happy!
AWR had its AdWords API access revoked almost a year ago. They gave us no reason or time to react. All we had to do was to submit a sort of “reinclusion request” which we did, but with no success.
Wow, I didn’t realize how common this was. It’s tough to speculate on what their motivation may be if they don’t even provide a reason.
Wonderful article. Rank Tracker Tool is another useful tool that tracks keyword ranks across top five search engines amongst other things. The link is http://www.ranktrackertool.com