I’ve read some of the reviews on Bing and the comparisons to the other engines too. It seems that Bing is actually being well recieved by the general public and the SEO people as well. I like Bing, I like Google as well, and I like Yahoo! too but all for different reasons. My reasons are however subjective just as everybody’s are so I decided to do a blind test on Bing, Google and Yahoo.
Michael Kordahi released BlindSearch
which queries all 3 engines and then asks you to choose the set of results you like best. Simple but great invention.
Now I can put my beliefs to the test:
I’ve just entered in a series of different types of query and picked my favourite results. Here is what happened.
surfboard -> Bing
ashtanga yoga -> Yahoo!
green tea effects -> Google
sleep deprivation experiment -> Google
skateboard trick ->Bing
time in Boston ->Bing
fairy bower manly -> Bing
batman comic -> Bing
dalmatian characteristics -> Bing
sony EOS manual -> Bing
construction grammar -> Bing
sentiment analysis -> Yahoo
5 day forecast -> Bing
how to build pyramids -> Google
how to peel a mango -> Google
when to take vitamins -> Google
pipeline masters -> Google
safe biking -> Yahoo
buying a surfboard -> Bing
getting over jetlag -> Bing
marylin monroe’s birthday -> Bing
repair fridge -> Google
asics trail shoes -> Yahoo
buy submarine -> Google
sell racehorse -> Google
visit orangutan borneo -> Bing
chemical symbol Fe -> Yahoo
caffeine chemicals -> Bing
statistics tutorial -> Bing
rainbow colours -> Bing
The results:
Bing = 16
Google = 9
Yahoo = 5
I am overwhelmed by how much Bing has scored compared to Google. I was hoping to be able to tell you in short that one of them did better on informational or navigational queries. It did look like Google was better at informational ones for a bit there, but actually Bing is just as good. I continued testing and actually it may even be better than Google.
Bing is actually pretty accurate in the sense that it really focuses in on your query and doesn’t approximate it so much. Bing is the only engine to have delivered links for a “5 day forecast”. The other engines gave me 7 day forecast links. It’s not what I asked for. The “Sony EOS Manual” was particularly successful in Bing. It’s basically asking about a camera manual. The other results gave me instructions for the Canon for example but Bing was the only one to deliver.
“marylin monroe’s birthday” was specifically chosen to confuse because Marylin also famously sang “Happy birthday” to JFK. Bing is the only engine to have given me biographical information about Marylin and her date of birth.
The problem:
It’s all well and good evaluating the engines like this, but it’s not the right way to establish which ione is actually the better one. At the end of the day, I’m a user and so picking which one works best is going to be established through the type of excerise carried out above, and that’s good. It means that users can decide which engine is the best for them.
To decide on which the best engine is overall in a scientific way we would need to do a whole lot more work than that. The following papers give some idea of what is involved:
“Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks” by B Carterette, R Jones
“A Largescale Evaluation and Analysis of Personalized Search Strategies” by Dou, Song, Wen
“Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search” Joachims,Granka, Pan, Hembrooke, Radlinski, Gay
“Results and challenges in Web search evaluation” by Hawking, Craswell, Thistlewaite and Harman
“Measuring Search Engine Quality” by David Hawking, Nick Craswell1, Peter Bailey and Kathleen Griffihs
“Extracting accurate and complete results from search engines: Case study Windows Live” by M. Thelwall
In some instances I think that more testing was required from some of the above experiments. There is a point where testing should stop because it stops telling you anything significant, but if we were to qualify one engine as beingm better than the others, we’d need to spend a lot of time evaluating.
Overkill?
For some of the queries I chose in my own test, I don’t need 10 results telling me exactly the same thing. The “most relevant to least revelant” format isn’t always intuitive. I might get the answer directly from the 1st result but for further information #5 is better. I am also spoilt for choice a lot of the time as sometimes the results are rather close so Bing and Google results are hardly better or worse than each other.
Information retrieval is not a delicate science as such: a ranking algorithm is a hammer and every query is a nail.
did a check on some keywords for one of my sites with the blindsearch tool you link to above and found that Bing returned much better results for my site than Google. Yahoo was also quite good.
Teh bing results (mine and those from competitors)were highly accurate and related to the travel theme my site and its keywords relate to.
It would be a cool experiment to de brand all 3 engines and do a bunch of blind testing, like the pepsi blind taste test and see what comes of it.
My guess would be that the brand influence of google V blind is massive on how users perceive the relevant of a search engine….
Just tried out the blindsearch. I like Bling and Google results better. Both scored well compared to Yahoo.
Yes that’s right, I reckon that Bing and Google would so neck and neck. Yahoo is a good engine for different things. Google and Bing seem to be most similar. I do think that we are also used to using Google (most of us), so we actually change the way we query to match the engine.
I had the same results when comparing serps. Picture search in Bing is also better. Can we now have an engine that looks like Google but gives the results from Bing please? Bingle (!?)
Ive just been so surprised at the lack of media hype over bing! am i living in a cave or something or why so little media attention to the new search facilities?
This is great. Thanks so much for putting in the time to research this – its helpful!
SAP I reckon this is because when something is good and works well nobody covers it much. If it’s something that is rubbish it seems that everybody enjoys having a dig at it. Micro were relatively quiet about it though.
Lets hope Bing can start to hack away at Google’s dominance. I hate the way Google loves old pages. If you search for ‘advanced SEO’, for example, you just get old pages ranking well. These will be totally outdated (usually), and no good to the user.
It’s a good point Gareth. I find myself increasingly having to use “advanced search” for the types of queries you suggested there.
I am not a big microsoft user, but I must admint I like what they did with Bing! I also prefer using bing for images research, i feel it’s more precise and has more options for the user!
Nice post! Keep up the good work!
Cheers
In my oppinion BING there is just a poor copy of google. Because bing design is poor and search results are not qualify.
I don’t enjoy Bing all that much, and I must say, Google has become a reflex, the industry leader. It will be hard to get me to change!