In the past couple of days, there has been a lot of hype surrounding the forthcoming “Wolfram Alpha”, a search engine aimed at providing the answers to questions by computation rather than the existing approach to search in which potentially relevant resources are found and suggested as a good place to look for answers. It sounds incredibly promising, and the media have been quick to seize on it, but can it possibly deliver what it claims? I really don’t think so, and I’m going to take a few minutes right now and explain why.
First let me establish my credentials in the field – as many of you know I’m a research student in AI. I’ve studied AI now for 8 years, and been an enthusiast for closer to 15. Right now I’m working with one of the finest AI Planning research groups in the world, and although my work isn’t anything like that calibre, I’d like to think that I’ve learnt a bit about the state of the art based on the problems my colleagues are attempting to tackle. Although our work doesn’t directly relate to the overarching concept of “computing” answers to factual questions, it is most definitely related since both areas are concerned with modelling and reasoning about data, and inference – which is to say deriving new facts based on the presence of other facts. In short, I know a thing or two about what I’m about to talk about. More importantly, the media types that are involved in reporting on this kind of thing have nothing like that kind of background, so its hardly surprising that they are buying into the PR surrounding this product.
Lets take a step back though. There are basically two main components to this work as far as I can see. Firstly there is the notion that a question asked in natural language in a text input field on a website can be translated into some form of “formula” to which there a definite answer can be computed. Natural Language Processing is a hard problem in and of itself, and the ability to get a computer to recognise the functional-equivalence of “The cat sat on the mat” and “The mat is the location at which the cat was sat” (possibly not great examples) is something that, as far as I’m aware, does not exist yet. The distinction between meaning and structure is something that is so poorly represented currently that even having cracked this issue alone and being able to distil sentences down to formulaic representations ought to be cause for a seriously spectacular pizza party – however NLP is not really my area so I don’t want to get too into this bit. By far the most interesting part of WA to me is the concept of computing an answer based on resources on the web. Right now in the Planning community we operate broadly by representing a problem using a formal semantics that captures the essence of what we are working with, and then reasoning using this model of the problem to understand how changes can be made and how we can achieve what we need – the point of the work is to allow computers to work out automatically what needs to happen in order to satisfy objectives, and its technology that is already in use in many places, from the Mars Rover to scheduling train maintenance. The biggest issue we have in our field is that modelling problems is non-trivial, and attempts to do it automatically have fallen far short of what is necessary, so to suggest that the factual content of resources can somehow be modelled, the key aspects derived into a formal semantics that could be used for reasoning is really quite specious – that such a system could be used to compute the answer to any factual question with any sort of accuracy seems beyond the realm of current possibility. Typically knowledge is represented through the use of an Ontology, which more or less is a formalism that dictates how information will be modelled, with the idea that by standardising the format, information can be more readily found and reasoned about, and using an approach based on this it isn’t hard to see how (still assuming a question can be distilled into its ontological roots) such a system can be used, but the significant challenge in this method would be in creating a knowledge base conformant with the ontology automatically, without losing quality – to segregate the worthwhile resources from nonsense. Google does this through its Page Rank mechanic, but in order for WA to produce a definitive answer to a question, it needs some form of truth-sense to determine the validity of things in order to do this. This is the main point of my argument – that based on information available, there is no way of producing definitive answers to questions. You can maybe answer questions with a certain level of confidence, but actual certainty is beyond our capabilities while still remaining reliant on such a massive, noisy dataset. Moreover, attempts to produce certainty in results by their nature restrict the realm of potential questions to strictly those that have black and white answers on the internet, and in this day and age there are still, for example, many many web pages dedicated to the notion that the moon is made of cheese. Does that mean then that WA will not be able to answer “Is the moon made of cheese?”, and that being the case, what use is a system that can’t determine such a trivial fallacy – how can it be expected to answer questions with more subtle issues. The obvious approach to overcome this weakness is to restrict the computation to a set of facts that are known, thus limiting the WA to being able to answer with certainty just a small subset of questions. In AI we call search engines built around this approach “Expert Systems” and we’ve had them for many many years. Which leaves WA with either very very sophisticated technology far in advance of anything in use currently, or built on technology that is relatively uninteresting due to it being largely well understood.
Stephen Wolfram, the physicist behind WA attempts to allay sceptics : “I wasn’t at all sure it was going to work. But I’m happy to say that with a mixture of many clever algorithms and heuristics, lots of linguistic discovery and linguistic curation, and what probably amount to some serious theoretical breakthroughs, we’re actually managing to make it work.”. Of course, the guy who is the public face for the project and has his name on it is bound to say something along these lines, but it is interesting to note that this isn’t the first time that the validity of his claims has been called into question – in 2002 Wolfram publish “A New Kind of Science” detailing a branch of mathematics research called Cellular Automata. I won’t bore you with the details of what they are or how they work, the very very quick version is that you have a grid of cells, and each one can be in one of many states, but for the sake of simplicity call it “on” or “off”. At any time point t, it is possible to work out which cells are on and which are off using a predetermined set of rules as to what makes a cell on or off, based on which cells are on and off at time point t-1. The point is that even using a simple set of rules, very complex behaviours can be produced, and this can be seen in Conway’s Game of Life, which obeys four simple rules and can produce some amazingly intricate behaviours. Wolfram’s book was an investigation into this field, a summary of research he had undertaken previously and a speculation into the relevance of Cellular Automata for more broad underpinnings of the natural world, or put another way, that the behaviour we observe and actions we perform are created by the same kind of simple rule set expressing a complex pattern of interaction. Unfortunately for Wolfram, it might have been the first time such notions had been expressed in quite so mainstream a setting, but it certainly wasn’t the first time they’d ever been thought of. The book received criticism for over-emphasising the importance of Wolfram’s own work, of being misleading, and there were allegations that portions of work had been duplicated without any attribution – grave sins in the scientific community.
If this was a guy claiming to have built a spaceship that could travel faster than light, he’d be dismissed as a crank instantly – and yet crucially, the advances that are being claimed rely on just that kind of jump in technology, but they seem so much more plausible – possibly because so few people understand the way search engines work already. It would be awe inspiring if the claims weren’t being made by a known plagiarist and someone who has in the past made great efforts to overstate his contribution to a field, but given whose mouth they are coming from, and the kinds and number of breakthroughs required to actually make it work as advertised, I’m highly sceptical. I hope it works, I really do. The repercussions for the whole of industry, academia and, not to put too fine a point on it, the world, would be incredible. Now that Wolfram is back in the spotlight, it is hard not to think back to his moments of former ignobility and wonder whether the WA will live up to the hype he is creating around it.