IBM’s Watson on Jeopardy is an incredible accomplishment. The various domains of the problem are very interesting. It isn’t just a (un)natural language processor, a search engine, or a trivia machine. It is several very complex domains running to together and fast! You could see the frustration in Jennings’ and Rutter’s faces at being unable to buzz in most of the time.
It found one answer (Jeopardy reverses the normal question and answer format) very telling about Watson’s abilities. The answer was “It was the anatomical oddity of U.S. gymnast George Eyser, who won a gold medal on the parallel bars in 1904.” and Watson answered “What is leg?”. That was only partially true and Watson got the question wrong. Rutter answered “What is a missing leg?” and got it right. I think this shed a little bit of light into how Watson is reducing the answers into questions. Its algorithm probably removes anything other than noun or names for certain answer types.
This made me start thinking. If the Watson Jeopardy exhibition had more questions like this and fewer ones that are nouns, definitions, and proper names, would Watson still win? From that question, I would say no.
