I’ve read some of the reviews on Bing and the comparisons to the other engines too. It seems that Bing is actually being well recieved by the general public and the SEO people as well. I like Bing, I like Google as well, and I like Yahoo! too but all for different reasons. My reasons are however subjective just as everybody’s are so I decided to do a blind test on Bing, Google and Yahoo.
Michael Kordahi released BlindSearch which queries all 3 engines and then asks you to choose the set of results you like best. Simple but great invention.
Now I can put my beliefs to the test:
I’ve just entered in a series of different types of query and picked my favourite results. Here is what happened.
surfboard -> Bing
ashtanga yoga -> Yahoo!
green tea effects -> Google
sleep deprivation experiment -> Google
skateboard trick ->Bing
time in Boston ->Bing
fairy bower manly -> Bing
batman comic -> Bing
dalmatian characteristics -> Bing
sony EOS manual -> Bing
construction grammar -> Bing
sentiment analysis -> Yahoo
5 day forecast -> Bing
how to build pyramids -> Google
how to peel a mango -> Google
when to take vitamins -> Google
pipeline masters -> Google
safe biking -> Yahoo
buying a surfboard -> Bing
getting over jetlag -> Bing
marylin monroe’s birthday -> Bing
repair fridge -> Google
asics trail shoes -> Yahoo
buy submarine -> Google
sell racehorse -> Google
visit orangutan borneo -> Bing
chemical symbol Fe -> Yahoo
caffeine chemicals -> Bing
statistics tutorial -> Bing
rainbow colours -> Bing
Bing = 16
Google = 9
Yahoo = 5
I am overwhelmed by how much Bing has scored compared to Google. I was hoping to be able to tell you in short that one of them did better on informational or navigational queries. It did look like Google was better at informational ones for a bit there, but actually Bing is just as good. I continued testing and actually it may even be better than Google.
Bing is actually pretty accurate in the sense that it really focuses in on your query and doesn’t approximate it so much. Bing is the only engine to have delivered links for a “5 day forecast”. The other engines gave me 7 day forecast links. It’s not what I asked for. The “Sony EOS Manual” was particularly successful in Bing. It’s basically asking about a camera manual. The other results gave me instructions for the Canon for example but Bing was the only one to deliver.
“marylin monroe’s birthday” was specifically chosen to confuse because Marylin also famously sang “Happy birthday” to JFK. Bing is the only engine to have given me biographical information about Marylin and her date of birth.
It’s all well and good evaluating the engines like this, but it’s not the right way to establish which ione is actually the better one. At the end of the day, I’m a user and so picking which one works best is going to be established through the type of excerise carried out above, and that’s good. It means that users can decide which engine is the best for them.
To decide on which the best engine is overall in a scientific way we would need to do a whole lot more work than that. The following papers give some idea of what is involved:
“Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks” by B Carterette, R Jones
“A Largescale Evaluation and Analysis of Personalized Search Strategies” by Dou, Song, Wen
“Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search” Joachims,Granka, Pan, Hembrooke, Radlinski, Gay
“Results and challenges in Web search evaluation” by Hawking, Craswell, Thistlewaite and Harman
“Measuring Search Engine Quality” by David Hawking, Nick Craswell1, Peter Bailey and Kathleen Griffihs
In some instances I think that more testing was required from some of the above experiments. There is a point where testing should stop because it stops telling you anything significant, but if we were to qualify one engine as beingm better than the others, we’d need to spend a lot of time evaluating.
For some of the queries I chose in my own test, I don’t need 10 results telling me exactly the same thing. The “most relevant to least revelant” format isn’t always intuitive. I might get the answer directly from the 1st result but for further information #5 is better. I am also spoilt for choice a lot of the time as sometimes the results are rather close so Bing and Google results are hardly better or worse than each other.
Information retrieval is not a delicate science as such: a ranking algorithm is a hammer and every query is a nail.