Keynote Speech 

Referring Expressions: How to point by not pointing

Tsuhan Chen, Cornell University, USA

With recent advance in computing, humans interact and collaborate with machines more and more frequently. For example, one may tell a robot to pick up "the red bottle on that round table". Vice versa, a video surveillance system may tell a human guard to chase "the man in sunglasses carrying a brown suitcase". Likewise, a navigation system may tell the driver to go "50 meters beyond the restaurant on the right with a yellow awning", or to follow "that white car that's turning left at the upcoming T-interaction". In this talk, we will present our findings in automatic generation of these expressions, which are often called the "referring expressions." Combining techniques in computer vision and key concepts in language and psychology, we can generate efficient referring expressions that take into account the saliency of objects, uncertainty due to imperfect visual attributes, and object location and relative attributes. With crowdsourcing to collect user data and to validate our hypotheses, we show that our referring expressions are effective in referring the viewer to a specific image, or a specific object within an image. Such a multimodal approach to human-machine language interaction presents exciting future research opportunities. 


Tsuhan Chen has been with the School of Electrical and Computer Engineering, Cornell University, Ithaca, New York, since January 2009, where he is the David E. Burr Professor of Engineering, and served as the Director of the School of Electrical and Computer Engineering from 2009 and 2013. From October 1997 to December 2008, he was a professor with the Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, where he served as the Associate Department Head from 2007 to 2008. From August 1993 to October 1997, he was with AT&T Bell Laboratories, Holmdel, New Jersey. He received the M.S. and Ph.D. degrees in electrical engineering from the California Institute of Technology, Pasadena, California, in 1990 and 1993, respectively. He received the B.S. degree in electrical engineering from the National Taiwan University in 1987.

Tsuhan served as the Editor-in-Chief for IEEE Transactions on Multimedia in 2002-2004. He also served on the Editorial Board of IEEE Signal Processing Magazine, and as Associate Editor for IEEE Trans. on Circuits and Systems for Video Technology, IEEE Trans. on Image Processing, IEEE Trans. on Signal Processing, and IEEE Trans. on Multimedia. He co-edited a book titled Multimedia Systems, Standards, and Networks.

Tsuhan received the Charles Wilts Prize at the California Institute of Technology in 1993. He was a recipient of the National Science Foundation CAREER Award, from 2000 to 2003. He received the Benjamin Richard Teare Teaching Award in 2006, and the Eta Kappa Nu Award for Outstanding Faculty Teaching in 2007. He was elected to the Board of Governors, IEEE Signal Processing Society, 2007-2009, and a Distinguished Lecturer, IEEE Signal Processing Society, 2007-2008. He was elected as the Vice President of ECE Department Head Association in 2012, and served as the President 2013-2014. He is a member of the Phi Tau Phi Scholastic Honor Society, and Fellow of IEEE.


Copyright © 2014, IEEE International Symposium on Multimedia