The Panoptic Studio: Social Signal Processing in a 500 Camera Facility

ECE Seminar: The Panoptic Studio: Social Signal Processing in a 500 Camera Facility

Starts at: August 28, 2014 4:30 PM

Ends at: 6:00 PM

Location: Hamerschlag Hall, 1107

Speaker: Yaser Sheikh

Affiliation: Robotics Institute, Carnegie Mellon University



In this talk, I will present the design of a massive multi-view system designed to capture thesubtle details of a social interaction. Social interactions present three challenges to motion capturemethods:

(1) subtle details that encode crucial information about the interaction need to be resolved overa volume large enough to house several people, (2) severe occlusions frequently occur in any smallsubset of views, and(3) no dominant direction of observation exists.

Our system uses 500 synchronized cameras of hybrid resolutions mounted over the surface of ageodesic dome to capture detailed motion of approximately fifty thousands of points over volumessufficient for polyadic interactions of ten people. Using a dataset with participants involved in polyadicgames (to evoke natural interactions), we establish the performance of this large-scale multi-view systemby demonstrating robustness to occlusion, increased capture density, and agnosticism to the orientationof activity.

I will show raw 3D reconstruction of unprecedented motion detail over an order of magnitude longerdurations than have been previously recorded and describe current and future directions of research.


Yaser Sheikh is an Associate Research Professor at the Robotics Institute, Carnegie Mellon University,with appointments in the Mechanical Engineering Department and the Quality of Life Technology Center.His research interests span computer vision, computer graphics, and robotics, and are focused on themachine perception of social behavior. He has won the Honda Initiation Award (2010), best paper awardsat SAP (2012), WACV (2012), SCA (2010), and ICCV THEMIS (2009), and the Hillman Fellowship forExcellence in Computer Science Research (2004). He has over 50 publications in leading conferences incomputer vision, computer graphics, and machine learning, and hold four patents on his research. Yaserhas served as a senior committee member at leading conferences in computer vision, computer graphics,and robotics including ICCP (2011), SIGGRAPH (2013, 2014), CVPR (2014), ICRA (2014), and CVPR(2015). His research is sponsored by various government research offices, including NSF and DARPA,and several industrial partners including the Intel corporation, the Walt Disney Company, Nissan, Honda,and the Samsung Group. His research has been covered by various media outlets including The NewYork Times, BBC, MSNBC, New Scientist, slashdot, and WIRED. He received his PhD in 2006 from theUniversity of Central Florida advised by Prof. Mubarak Shah, and his BS degree from the Ghulam IshaqInstitute of Engineering Science and Technology in 2001.