Science for SEO

Duck Duck Go is a new search engine that has quietly come along and raised eyebrows.  In a good way.  You know how we sigh and say “NASE” (not another search engine)?  Well that’s exactly what I was thinking until I actually tried it.  It’s not a Google killer, nor does it claim to be.  It’s a very useful tool for finding a lot of information related by topic quickly.  I tested it on a random term and came up with all sorts of goodies, information I wasn’t even looking for.

What it does that’s cool:

It allows you to find a lot of information related to your topic fast. Let’s take it for a ride…

(This is my conversation with the search engine):

Query: Killers

DDG: Do you mean the book by Hemingway, the film adaptation or that other one, or do you mean the band from Nevada or or the Iron Maiden or KISS album?

CJ: Um..the band from Nevada

DDG: Here’s a link to the official site, MTV, their MySpace page, oh and did you know about the tribute band? You could look at other bands from Nevada or find specific information about the band, like what awards they’ve won…

CJ: Yea, who else is from Nevada?

….and so it continued….

I like that you can intuitively link topics and information together.  Duck Duck Go does not have any special interface or swanky visualisation of semantically related items, and that’s quite a breath of fresh air!  The results are good, in fact the information sources are good I should say.  Because you can manoeuvre through the results by scrolling down the short list, but also through using the drop down to go to an associated topic or to drill down further, it moves “sideways” if you see what I mean. This is a bloggers playground.  You can find a wealth of connected information super fast.

I interviewed Gabriel Weinberg, the president and founder of Duck Duck Go. He graduated from MIT in Physics, and then in technology, his thesis was “A Systems Analysis of the Spam Problem”…we like spam here, it’s a fun problem.  Also he likes animals and soccer.

CJ: Gabriel, who are you?

GW: I’m a self-taught startup hacker and serial entrepreneur out of MIT. My last startup, namesdatabase.com, was acquired by classmates.com in 2006.  I’ve always been interested in search, but got more interested after the acquisition.  Then I got serious about it at the end of 2007 and started working on what eventually became Duck Duck Go.

CJ: Why is Duck Duck Go different to any other old search engine?

GW: I keep working on the answer to this important question, and my most succinct version is here.

At a higher level then what you will find there, we’re out to do two things.  First, we want to simplify the search experience to get you the information you’re looking for faster and with less mental effort. Our zero-click info, simpler links, less clutter, removed parked/spam pages, etc. all fall into this category.

When you see a page of Google results there are usually a bunch of titles and snippets that don’t really make sense, and so you find yourself clicking forward and back a lot, and spending mental effort (and your time) figuring out what is what.  At Duck Duck Go, we want to get rid of all that extra effort.  So if you just want to know what something is, we tell you in the zero-click into; if you just want the official site, we tell you by labeling the link Official site, etc.

Second, we want to help you find related information you might not otherwise have thought of, but nevertheless wanted.  Our explore box, category pages, ambiguous keyword detection, etc. fall into that category.

CJ: How does it work?

GW: We use structured data from human powered sources like Wikipedia and Crunchbase (over 50 of them at the moment) to drive our zero-click info, official site detection, ambiguous keyword detection, related topics, and category pages.  We do not have our own human editors–instead, we rely on data from great sources that often have human editors at their core.  Our rule is we won’t display anything from these sources that we don’t believe to be better than regular deep search results, i.e. Google results.

We map each source into an internal database that gets checked on each query.  This mapping creates topics.  For example, each Wikipedia page will be its own topic.  We then have code (including misspellings) that maps queries to topics, which determines what zero-click info we display and what gets populated in our explore box.

We have another set of human powered sources that helps us do name detection, which is why our name results are usually better.  We also work to simplify the look and feel of links through various regex substitutions.

Our deep search results are currently populated by Yahoo! BOSS.  We actually started out crawling the Web ourselves (and still do).  But Yahoo! BOSS does a better job at the moment than our crawler, and so using it lets us concentrate on our unique properties.  At some point, we will gradually incorporate more info from our own crawling.

Our removal of parked/spam pages is via a joint effort with The Parked Domains Project, which I am also involved in. This effort also crawls the Web, using various heuristics to identify these domains, which then get omitted from both the human powered sources and deep results.

CJ: Are you going to make plugins and other goodies available to bloggers?

GW: I’m really interested to know if you have anything in mind in this regard.  We have two ancillary(non-search-related) tools right now:

Karma

Profiles

But I’ve been thinking a lot about making a search tool for bloggers.

End of interview!

I think that this is a cool little engine to watch, I already use it when I need to tear around pulling information from lots of different resources to build a complete picture.  The best thing for me is that while I’m doing that I find information I had no idea was related.  It’s finding what you’re not looking for that’s really golden.  Could we have a new evaluation measure for search engines?

Thank you to Gabriel for answering my questions, I am sure he will be happy to answer yours too.  Take Duck Duck Go for a spin and enjoy yourselves.

Related Posts:

• Dr. Searcher and Mr. Browser
• Search engines to calculate implicit semantic relations
• Frowning at Google
• Search Engine Result Evaluation
• AI and its implications for SEO