Robots learn by watching how-to-do videos

Robots learn

Scientists are teaching robots to watch how-to videos and derive a series of step-by step instructions to perform a task, an advance that may help future `personal robots’ to do everyday housework. The researchers at Cornell University in New York call their project “RoboWatch”.

There is a common underlying structure to most how-to videos and there is plenty of source material available, researchers said. By scanning multiple videos on the same task, a computer can find what they all have in common and reduce that to simple step-by-step instructions in natural language.

The work is aimed at a future when we may have “personal robots” to perform housework -cooking, washing dishes, feeding the cat -as well as to assist the elderly and people with disabilities, researchers said. In the new method, a robot with a job to do can look up the instructions and figure them out for itself. Faced with an unfamiliar task, the robot’s computer brain begins by sending a query to YouTube to find a collection of how-to videos on the topic. The learned knowledge from the YouTube videos is made available via RoboBrain, an online knowledge base robots anywhere can consult to help them do their jobs.