top of page

In Cookie We Trust

In Cookie We Trust is an interactive window installation display that reinterpret the classic fortune cookie in new media to know one's future. It's a bit chunk of fun that divines people's fortune specific to the person using Artificial Intelligence.


Experience & Interaction Designer, Creative Technologist

Project Type

Interactive window installation display


Arduino, Processing, Facial Recognition


Chelsea Chen, Daniel Castaño, Arnav Wagh

Project Timeline

September 5th - November 13th 2018


November 9th - 13th 2018 at New York University Skirball Windows, LaGuardia Pl, New York City

Desktop HD_3x.png

How fun is it to know your fortune?

Fortune-telling has been part of human life for millenia. Some people does take it to heart some just find it for fun. Fortune cookie is a unique phenomenon of fortune-telling which people get their fortune on a small slip of paper inside a cookie. The most interesting part of fortune cookie is how people experience the anticipation and enthusiasm while breaking the cookie. For some people, the fortune coming from the cookie defines how the Universe provides a belief to reinforce what one already wants and wishes for. Even though some others don't take it seriously, people always find it fun to see what a cookie can tell human about life. The art of classic fortune cookie is the randomness of fortunes. But, what if it can predict your fortune specifically to you? Do you believe it? Or just find it for fun?

The most interesting part of fortune cookie is how people experience the anticipation and enthusiasm while breaking the cookie.

To enhance people's enthusiasm and surprise, we hide the camera so that user wouldn't notice if the fortune was based on their facial recognition. The goal of this project is to bring fun into fortune-telling and make people ask their beliefs if the fortunes are made specifically for them.

Facial recognition to divine people's fortune

One of many technologies to know specific human presence is facial recognition. In order to bring more value to personalized fortunes, we used facial recognition technology to detect audiences' faces and categorize them based on the highest confidence rate. This data then will be used to map the categories into our fortunes database. We have hundreds of fortunes for each appearance category. That way, each person will get personalized fortune based on their facial recognition. We used Microsoft Azure API to detect human faces.

Face Recognition using Microsoft Azure
Facial Recognition_3x.png

Engaging people on a window display

The biggest challenge of this project is how to engage people to interact with window installation display. Interactive window installation is one of the hardest installations to interact with. To engage the audiences, we paid attention to every detail interaction.

Interaction Flow_3x.png

Interaction design

Visual display also plays a big role in engaging people's attention for window display. We used contrast colors and fabricated a giant fortune cookie as part of affordance to people's attention. For the interaction itself, we put hand sign to let people know how to interact with the installation.

double take.jpg

Display at NYU Skirball

In the making


Process 2_3x.png

Code and fortunes database

Making a giant fortune cookie interactive installation

The process took about 2 months with the first month for developing the concept. Roughly we took 3 weeks of production from turning the visual sketch into an actual window installation display. The production was very intense from fabricating the giant fortune cookie by cutting woods using CNC machine, creating the cookie surface using foam, making the movement mechanism to open and close the cookie, and developing all physical computing parts; from LED strips, vibration speaker, spotlight, and linear actuator.

Process 1_3x.png

We used four separate microcontrollers (Arduino UNO) to make the installation robust and to reduce problems rate. Each Arduino connects to: LED 12V as spotlight, linear actuator to move the cookie open and close, capacitive touch as a main trigger from user, and Fadecandy LED driver to animate the LED Strips. We used processing as a main controller to control the entire sequence; starting from the user touched capacitive touch, then it would take user's picture, triggered LED thunder animation, thunder sound, spotlight increased brightness, and the cookie started to open. When it opened, the fortune message would be shown on the monitor display for 7 seconds until the cookie started to close. We had a fortune database contained of hundreds messages. A message that appeared was based on user's appearance using facial recognition technology.


Make people smile

I was so happy to see how people were smiling and excited to interact with the installation and very enthusiastic to see their personalized fortune. They were very surprised about how could the cookie knew them and gave the message very related to their personal appearance.

"It's hard not to feel happy when you make someone smile." - Unknown

bottom of page