Tips from your robot overlord

Until this century,“machine learning” was a field that excited mostly academics and sci-fi authors. But today’s recommender systems aren’t on the theoretical fringe—they’re part of a lucrative industry. Netflix is about to award $1 million to a team of researchers who improved the accuracy of the company’s recommendation tool by 10 per cent. Some 51,000 contestants from all over the world entered the competition,and two $50,000 “progress prizes” were awarded over the past three years. All this to improve a system that is already pretty good at predicting what movies people will enjoy: 60 per cent of Netflix rentals are a result of what a piece of software suggested.

As we live more of our lives online,we depend on computers to give us the kind of advice that other people once did. Movie critics and well-read librarians are being replaced by algorithms—well-tested,well-funded algorithms,but pieces of software code nevertheless. It’s possible that somewhere within our super-convenient video rental systems and online stores,we’re incubating a form of artificial consciousness. Could we be witnessing the birth of AI without realising it,every time we rent a movie?

Larry Page,a co-founder of Google,has said that he thinks artificial intelligence isn’t far off and could emerge from Google search. Google search is like an extremely complicated recommender that learns from links all over the Web to identify the most helpful answers to queries. For Page,artificial intelligence would essentially be a perfect search engine that always gave you the most accurate,helpful results.

Surely,intelligence must be more than the ability to recommend good links or enjoyable movies—but what exactly is it? AI researchers have been asking that question since the 1950s.

Marvin Minsky,a pioneer in the field and founder of MIT’s artificial intelligence lab,once said,“In general,we are least aware of what our minds do best.” When we finally see our brains reflected in silicon,we may be surprised to discover that,for example,consciousness emerges out of an ability to anticipate what the people around us will enjoy. Like a really good movie.

This assertion is hardly an insult to human intelligence. It turns out that coming up with accurate recommendations requires both sophisticated software and massive amounts of data. To find out how the mind of a recommender works,I talked to AT&T Labs scientist Robert Bell,a member of the winning team in early rounds of Netflix’s contest.

The most obvious obstacle for Bell and his colleagues were big gaps in the data. Netflix asks its customers to rate movies as often as possible,but they rarely do. Trying to predict enjoyment based on so little feedback is like a trying to pick out a present for a friend—you have some information about what she likes,but it’s not as if she’s ranked every item in the store. So how do you pick the perfect gift?

You extrapolate. If your friend likes detective novels,she’ll probably like other detective novels. And if she mentioned once that she’s a fan of a particular author,that narrows the search even more: Maybe you should get her the latest detective novel by that author.

Bell and his team taught their recommender system to think about the Netflix problem in roughly the same way. First they designed their software to group movies using the “nearest neighbour method”,which associates movies with others a person rated highly. Then the team trained the software using a method that sounds almost like intuition. Called the “latent-factor approach”,it searches for connections between movies based on all the ratings every person has given them. Some of these connections are obvious: Action movies would be grouped together,as would romances. But as Bell and his team wrote in an essay on their research,“Because the factors are determined automatically by algorithms,they may correspond to hard-to-describe concepts such as quirkiness,or they may not be interpretable by humans at all.”

Though Bell doesn’t think his Netflix solution is artificial intelligence,he and his team did have to pick a side in what he describes as an ongoing controversy within the AI field: Should you have a machine imitate humans or let it figure out how to do a task in a way humans wouldn’t? They wound up letting the machine figure out the task on its own,coming up with connections that a human never could. And yet those machine-discovered links were what gave Bell’s recommender such good taste in movies.

So it’s not at all like a human,but it has good taste. Which could make it the next step in the evolution of artificial intelligence. Or just a super-accurate data-crunching software program. It all depends on how you define intelligence. Luckily,neither answer gives you a new robot overlord.

Newitz is the author of Pretend We’re Dead: Capitalist Monsters in American Pop Culture and the editor of the science fiction blog io9.com.