The TalkMiner Lecture Webcast Search Engine developed at FXPAL is now available at talkminer.com. The system allows users to search for lecture webcasts published on the web using words from slides in the lectures and related text material (e.g., title, abstract, etc.). The current system has indexed over 12,000 lectures published at YouTube, the U.C. Berkeley webcast.berkeley site, and the blip.tv site. Many YouTube channels are being indexed including Google Tech Talks, Stanford University, MIT Open Courseware, O’Reilly Media, TED Talks, and NPTEL Indian Institute of Technology. Most of the material indexed is in English, but the system supports Japanese so some lecture webcasts from Japanese Universities are included. Over time we anticipate that we will add more lecture webcast sites to the system.
So how does TalkMiner work? Put briefly, the system automatically identifies slides in the video, applies optical character recognition to the slides to create a set of words with time codes into the video, and creates a searchable index using those terms. The user interface allows you to search for talks given keyword(s) and displays a list of talks that match the keywords (i.e., the keyword(s) appear in the slides or text associated with the talk). You can browse thumbnail images of the slides in a talk and view the lecture webcast. We only create an index – all video is left at the original publisher’s site. A paper describing the algorithms used to detect slides in the video and the design and implementation of the system will appear at ACM Multimedia 2010 later this year. The challenging research problem was developing algorithms to identify slides in the video – it is much harder than people think (e.g., camera switching production, picture-in-picture, poor video captured from the back of the room, etc.).
This project was initiated to solve a problem reported by students who used the Berkeley webcasting system I developed. Analysis of system use showed that students almost always watched the lectures on-demand rather than in real-time, and they rarely watched the entire lecture. Students use the webcasts to study for exams – we could see this clearly by patterns of usage – and, they primarily wanted to review selected material covered by the instructor. In one class we discovered that for over 50% of the lectures, students watched less than 10 minutes from a 50-minute lecture and students watched the entire lecture only 10% of the time. Consequently, for using the system, effective search is a big issue. We did some experiments with speech-to-text to create search indexes, but they did not work very well for obvious reasons (e.g., poor quality audio, untrained speaker, and specialized vocabulary). TalkMiner is an example of a different search index to lecture webcasts. It searches text in the video rather than relying on text on web pages where the video is embedded.
Give the system a try and let us know what you think about it. We have many ideas for further development and use for the technology, but we would like to know what you think. You can send email to me directly or use the contact mail page(s) at the TalkMiner web site.
FX Palo Alto Laboratory