Vision Enhancement Using Augmented Reality

We’re developing novel methods to enable people with different abilities to perform daily tasks more effectively with augmented reality. For example, in one project we developed an augmented reality application that identifies specified products on a grocery store shelf and provides visual cues to quickly direct a user’s attention to these products. We’re exploring augmented reality applications in other domains as well such as wayfinding.

  • Yuhang Zhao, Sarit Szpiro and Shiri Azenkot. ForeSee: A Customizable Head-Mounted Vision Enhancement System for People with Low Vision. ASSETS 2015. ACM, New York, NY, USA, 239-249.​

Locally Connected Experiences

We look at how data from mobile devices and sensors can enable a socio-technical infrastructure to provide awareness, trust and meaningful connections between physically co-located individuals. Such infrastructure will empower people to make better connections and communication in their local communities, with long term impact on participation and democracy.

  • McLachlan, R., *Opila, C., *Shah, N., *Sun, E., & Naaman, M. (2016). “You Can’t Always Get What You Want: Challenges in P2P Resource Sharing” 2016 CHI Extended Abstracts, pp. 1301-1307, ACM Press, New York, NY.
  • Ma, X., McLachlan, R., Lee, D., Naaman, M. & Sun, E. (2016). “Movement: A Secure Community Awareness Application and Display.” In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, pp. 106-109, ACM Press, New York, NY.
  • Emily Sun, Ross McLachlan, and Mor Naaman. 2017. “MoveMeant: Anonymously Building Community Through Shared Location Traces.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA.
  • Emily Sun, Ross McLachlan, and Mor Naaman. 2017. TAMIES: A Study and Model of Adoption in P2P Resource Sharing and Indirect Exchange Systems. In Proceedings of the Conference on Computer Supported Cooperative Work (CSCW ’17). ACM, New York, NY, USA

Understanding Attention and Engagement

The goal of this project is to advance our understanding of the psychological mechanisms behind people’s attention, as reflected through their interactions with digital content. In particular, we focus on the context of actions that people take online without any experimental intervention and examine how context affects behavior. We draw on theories from a wide range of fields to address questions that pertain to individual’s attention to content, expectations for attention from others and the value in getting that attention. To that end, we harness machine learning methods as well as language and statistical modeling to analyze signals of human attention as they occurs naturally outside of lab settings.

  • Minsu Park, Mor Naaman, Jonah Berger. A Data-driven Study of View Duration on YouTube. ICWSM 2016, May 2016, Cologne, Germany
  • Max Grusky, Jeiran Jahani, Josh Schwartz, Dan Valente, Yoav Artzi, Mor Naaman. 2017 “Modeling Sub-Document Attention Using Viewport Time.”  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA.

Computer Vision and Human Preferences

Humans have all kinds of preferences for things we can’t describe formally. We all have our favourite music, food, writers, and TV shows. Sometimes we can’t articulate quite why we like something — we just do. That means it’s difficult to teach machines about these intuitive ideas like food taste or music similarity.

Our work is about how to reach into people’s minds and pull out their intuitive notion of similarity, like food taste. To do this, first we build a deep learning system that can classify different foods. We then augment this visual machine expertise with the human expertise of thousands of crowd workers. The final result is a learned food embedding that captures which foods taste similar. Now we can teach machines that broccoli tastes more similar to carrots than cake, even though a human can’t always articulate why.

  • Veit*, Andreas; Kovacs*, Balazs; Bell, Sean; McAuley, Julian; Bala, Kavita; Belongie, Serge. Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences. ICCV, Santiago, Chile, 2015, (*The first two authors contributed equally).
  • Veit, Andreas; Matera, Tomas; Neumann, Lukas; Matas, Jiri; Belongie, Serge. COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images. Scene Understanding Workshop SUNw at CVPR 2016.
  • Veit, Andreas; Wilber, Michael; Vaish, Rajan; Belongie, Serge; Davis, James; others. On Optimizing Human-Machine Task Assignments. HCOMP, San Diego, CA ,2015.
  • Andreas Veit, Michael Wilber, Serge Belongie.” Residual Networks Behave Like Ensembles of Relatively Shallow Networks.” Presented at NIPS 2016 in Barcelona.

Immersive Recommendation Systems

We introduce a new user-centric recommendation model, called Immersive Recommendation, that incorporates cross-platform and diverse personal digital traces into recommendations. Our recent work includes (1) creative content recommendation with unstructured application usage traces, (2) food and restaurant recommendation with food photos, and (3) news and events recommendation with personal text data.

  • Cheng-Kang Hsieh, Longqi Yang, Honghao Wei, Mor Naaman, Deborah Estrin. “Immersive Recommendation: News and Event Recommendations Using Personal Digital Traces”, WWW, 2016.
  • Honghao Wei, Longqi Yang, Cheng-Kang Hsieh, Deborah Estrin.”GroupLink: Group Event Recommendations Using Personal Digital Traces.” CSCW ’16 Companion, February 27 – March 02, 2016, San Francisco, CA, USA
  • Longqi Yang, Yin Cui, Fan Zhang, John P. Pollak, Serge Belongie, Deborah Estrin. “PlateClick: Bootstrapping Food Preferences Through an Adaptive Visual Interface”, CIKM, 2015 ACM
  • Estrin, D. and Juels, A. 2016 Winter. Reassembling Our Digital Selves. Daedalus, 145, 1, 43-53 (doi: 10.1162/DAED_a_00364).

Augmented 3D Prints

3D printing technology is becoming mainstream, and offers a potential alternative to tactile graphics. However, current 3D printed graphics can only convey limited information through their shapes and textures. We are developing tools to enable users to interact with 3D prints with gestures.

  • Lei Shi, Idan Zelzer, Catherine Feng, and Shiri Azenkot. Tickers and Talker: An Accessible Labeling Toolkit for 3D Printed Models. CHI 2016 . ACM, New York, NY, USA, 4896-4907.
  • Lei Shi. Talkabel: A Labeling Method for 3D Printed Models. ASSETS 2015, 361-362. *1st Place, Student Research Competition*