Desperately Seeking Search

Internet search is broken.

Battered by spammers, content farms and black hat optimizers, the typical Internet search is a horrible mess. Try basic search terms like "washing machine". Or try a term relevant to this site like "Finance Blog". What you get is a bunch of tricksters who have gamed the search engines to bring their clearly less relevant sites to the top of the pile. You as the user have to wade through these sites, hoping to find something remotely relevant to your search. Many give up in frustration. And while this situation is most acute at Google because their large search market share makes them a target for these shenanigans, the situation is better at Bing/Yahoo, but pretty much the same.

How do we solve this problem?

Pressure Google to get religion about spam.

Complaining helps. It is ironic that many Internet users are now searching high and low for ways to improve search. But, these frustrations have spurred the search guys into action and we are now starting to see results. It first started in late November with a big New York Times story on the allegedly fraudulent and intimidating behaviour of one online merchant using Google to help him receive traffic and customers despite his poor online reputation. I first mentioned my own problems with site scrapers copying content purely for Google rank and advertising dollars back in December. Yves Smith at Naked Capitalism had similar problems with scrapers. And just as we were publicizing our issues, other bloggers in the tech space were as well. Here are a few posts from right after the New York Times story showing both the concerns and response from Google:

• Google’s “Gold Standard” Search Results Take Big Hit In New York Times Story
• Google Changes Its Rank Algorithm In Response To DecorMyEyes Story
• Google Takes Steps To Combat Copyright-Infringing “Bad Apples”
• Google Improves Copyright Protection – Search Engine Land
• Google News Dropping Sites, Reviewing Inclusion Standards – Search Engine Land

Look elsewhere

All of this was good, but I saw no noticeable difference in my search results at Google and other users felt the same way. On New Year’s day, Vivek Wadhwa wrote a widely-read piece at Tech Crunch on how Google search had become almost unusable. Wadhwa said this about Google searches he wanted to do:

it turns out that you can’t easily do such searches in Google any more. Google has become a jungle: a tropical paradise for spammers and marketers. Almost every search takes you to websites that want you to click on links that make them money, or to sponsored sites that make Google money. There’s no way to do a meaningful chronological search.

He ended up using Blekko, a new search engine that had been getting some buzz in 2010. Search traffic for Blekko has since skyrocketed. I have used Blekko and I can tell you I don’t get as many spammy results. For a basic targeted search, Blekko is better than Google. The same goes for Bing, Microsoft’s search engine. Blekko has upped the ante by starting a spam clock showing the number of spam pages created since the beginning of 2011 – now already at 1.1 billion and counting. Blekko has also banned content farms, networks of sites that produce low quality content for search ranking and advertising dollars.

I now use Bing as my default search and Blekko for more specific and granular general searches. Google is still better for those esoteric one in a million searches because their database is so big. So that means I am no longer a big Google search customer. I suspect this is the case for tens of thousands like me since Bing has steadily been gaining ground on Google.

If you are having problems with your searches, some other choices that might work for you are Quora and SearchReviews. Quora is like crowdsourcing your search question. You ask a question like: "What is the Paleo Diet and what are its benefits?" and Quora members answer, with yet more members voting on the best answer, the best ranked answers rising to the top. Think of it like the "Who Wants To Be A Millionaire" ask the audience lifeline except you get to ask a lot more people. Yahoo! used to own this kind of search with Yahoo! Answers. But, as with everything at Yahoo!, this property has seen better days. When I checked the site this morning, they were promoting questions like "how do you cover up a fart in public?" on their front page. Thanks for the tip, but I’ll look elsewhere for serious questions.

SearchReviews is a site dedicated to becoming the biggest database of review content online. They take reviews from Amazon, TripAdvisor, and a bunch of other sites and put them in one place. So if you are looking to go to a hotel, buy a book or dine at a restaurant, you can find a review at SearchReviews. TechCrunch has a good article on SearchReviews.

Tweet it or Facebook it

I found Facebook a good way to get answers to questions. When I was looking to buy an e-reader, I just put up a question on my status asking what e-reader I should get and I got a ton of different responses from people I know and respect all in one place. That’s the power of Facebook. (We got a Kindle, by the way). You can get the same sort of Feedback from Twitter if you have a lot of twitter followers. But Facebook is a more intimate and personal environment and I suspect Google sees more of certain types of search moving to Facebook in this way. When you hear the rumours about Google or Apple buying Facebook, the compelling part of an acquisition comes from these kinds of networking opportunities that in Google’s case it would want to sell advertising against. Apple might benefit for altogether different reasons.

Google is feeling the heat

Now, Google is feeling it in all kinds of directions as I outlined above. But, it is Microsoft which is really Google’s main competitor. And Google knows that; Then Google CEO Eric Schmidt revealed this in a Wall Street Journal Interview back in September. I’m not just talking about search, although that’s where Google gets the lions share of revenue. But Google is up against Microsoft in a huge number of its ancillary businesses: Google Search versus Bing, Google Apps and Google Docs versus Microsoft Office, Android versus Windows Mobile 7, etc. The Nokia – Microsoft tie-up is certainly a threat to Google in mobile search. We have learned that Google asked Nokia to adopt Android instead of Windows Mobile. But since the Nokia CEO had just left Microsoft, it went with them instead. Bottom line: Microsoft is Google’s biggest competitive threat, not Apple and not Facebook.

The Bing sting was all about the pressure Google is feeling from Microsoft. When I wrote about What’s really happening with Bing ‘Cheating’ and Copying Google Search Results, I prefaced my understanding of the specific issue saying:

There are three main background issues.

1. Google is now seen as the biggest target for anti-trust regulators because of its dominance in search and advertising
2. At the same time, Google’s search results have deteriorated significantly of late.
3. As a result, decent but distant competitors to Google have re-emerged in search.

And this is significant because these are the issues which led to the confrontation. Do you think Google would have run the sting if its search results weren’t deteriorating or if Microsoft had 2% search share of if Google wasn’t getting stick from anti-trust regulators on both sides of the Atlantic? Of course not. Google is all about search and they are getting it from all sides right now. That is the why they did the sting (Update: And their arguments in that debate are now deemed considerably weaker by search experts).

Aftermath of the Bing Sting

The reality is Google still has a lot of work to do. Just this past week, TechCrunch founder Michael Arrington complained that "search still sucks." The very same day, the New York Times ran yet another big story this weekend on how JCPenney had gamed Google’s search ranking algorithm in order to get to the top of the search engine page ranking on literally hundreds of different keywords. A company doing search engine optimization on Penney’s behalf set up a ‘black hat’ operation with scores of websites to insert keywords linking back to the appropriate page on JCPenney’s website. The company clearly benefitted from this effort as it was number one across all of these keywords through the huge Internet retail holiday season.

Me, I just want relevant search results. And that’s all other Internet users want too. If Google can give them to me, fine; I’ll use Google. They have come out with a great way to get back at the spammers with a browser extension that personalizes your search results by blocking spam sites. I suspect Google will use this tool to down-rank those sites most blocked by users, which, of course, is something competitor sites can use to their advantage. As Google search Guru Matt Cutts says it is an endless cat and mouse game between the search engines and the spammers. I appreciate that. But I want results nonetheless. If Google can’t deliver, I will continue to look elsewhere.