talkTable : Project Update


There has been significant progress with the talkTable. The code looks in good shape. I have integrated speech recognition into the program. I’m using Luke Dubois’ speech.js library to do this. The library basically acts as an interface between p5 and Google’s speech recognition kit. There were certain issues that I faced while doing this :

  1. speech.js library doesn’t hold the capability to end the speech detection instruction. I made changes to the library in order to get around this. The changes were very basic. I simply added a method to stop the speech detection process.
  2. The Google API automatically stops running after a certain amount of time (roughly 1.5 – 2 minutes). I figured this might be happening because the way Google’s API has been designed is to trigger speech detection on a keypress, and it’s purpose revolves around detecting a single search query. I added a running interval in the program to re-initiate the API every 30 seconds. If it’s already running, the program does nothing.
  3. The API calls couldn’t be made using the server created by p5 IDE. Instead to diving into the server code for the p5 editor, I resorted to using a different server. With Sharif’s help, I created a python server that runs on my local host, and can easily talk to the API.

Now that I have worked around these challenges, I have a continuous workflow that uses speech recognition to interpret what the users are talking about. The speech detection event is triggered when the users select a topic discussion. The only thing that I wish to add to the code is to broaden the data set of words that are being recognized by the program. I was considering using an AI extension but I figured that the AI chat bots serve a different objective than what I’m trying to achieve. The bots are meant to chat with the user, whereas my program aims at making the users talk amongst themselves, while trying to maintain the conversation intermittently. Therefore, I will creating my own data set for the program. Advantages of doing this are:

  1. The program can be personalized to have certain opinions and characteristics. I much prefer this over a generic response machine.
  2. The program can be curated to listen to the users and instigate them to keep talking.


Physical Computing:

I ran into many issues with the physical computing bit. The issues are as follows:

  1. FSRs have minds of their own. First of all, there is a lot of noise in the analog results that they produce. However, since the results are within a certain range I can work around this problem. The bigger problem is that if their position is disturbed, they start producing different ranges. They are highly sensitive to placement, and are also very fragile. I constructed my own FSRs and they lack the durability of a manufactured one.
  2. Finishing and Fabrication. I’m very new to fabrication and it intimidates me. This is my first proper attempt at fabricating and consolidating circuits. Right now, the circuit is all over the place. It doesn’t fit into the designated box, and is highly susceptible to losing connections on physical disturbance.

To solve these issues, I’m considering the solutions stated below:

  1. I might be using a switch instead of FSRs (as suggested by Benedetta). The only advantage with an FSR is that it gives me an approximate idea of the weight of the object, which can help me in confirming that it is a phone that’s been inserted into the socket and not some other object. But at this point, using a switch seems like a more favorable approach.
  2. I have fabricated the boxes, the switch panels, sockets and socket covers. The overall structure looks swell. However, I’m facing issues with optimizing my circuit and trying to make it highly durable. Mithru is helping me out with solutions on better circuit design techniques. I will be updating this post with pictures on how it progresses.


User Testing:

User testing has been extremely helpful to give me implementation ideas. Following is some of the feedback that I’m planning to incorporate into the project:

  1. Projecting the visualization on the table instead of using a computer screen. This was suggested by Dominic, and sounded like a great idea. It makes the project actually look like a talkTable, first of all. Also, it takes away the awkward positioning of the laptop in the middle of the table. I think it just makes the whole conversation setting more natural.
  2. Using synthesis with/instead of the text. This was suggested to me by Benedetta. It’s a good idea to use spoken word instead of text. Text can be distracting in the middle of conversations. I will be implementing depending on the time that I can find.
  3. I also tested the project in the ITP lobby to make sure that the audio inputs work fine in the presence of noise. This test was performed to make sure that the device doesn’t malfunction during the show. The words were comprehensible if the user spoke into the mic, which was great.

To  send across an idea, the structure of the device will look something like this. Right now, the project looks in pretty dismal state since I’m still working on the fabrication and rewiring the circuit. I will soon be posting pictures of the progress that I make.


Animation Assignment 3 : Unity 3D Basics

Animation / Week 6 and 7 / Unity 3D

The final two weeks of the class covered some of the basics of animating in Unity 3D. It also briefly featured Fuse CC and Mixamo, and how they facilitate creation and animation of characters using pre-defined rigs and skins.

Concept :  I wanted to create a simple controller for a drunk character. Due to the finals week and shortage of time, I kept the application as simple as possible.

For the assignment I imported a character and some animations from Mixamo, using which I scripted some movements on key presses. The character has the ability to walk, turn, walk backwards, dance, and fall down. And it does all of that as an inebriated person would do. The staging of the character was done in a medieval setting. To create the environment, I imported some terrains and objects from the asset store. In order to exaggerate the drunk effect, I parented the character’s body to the camera. The result was very interesting.

Tools used : Fuse CC, Mixamo, Unity 3D, QuickTime, Unity Asset Store

Conclusively, the course was intriguing and opened me up to the possibilities of experimenting with moving visuals in both 2D and 3D formats. I would love a dedicated course on animation (at the school or through personal study), wherein some of this experimentation can be formally realized and constructed into something that is more finished and conceptually stronger.

Animation Assignment 2 : After Effects

Animation / Week 2,3,4,5 / After Effects

The second, third and fourth week were about learning and playing around with different techniques in Adobe After Effects. Personally, it turned out to be a good learning experience in terms of the abilities of video editing and possibilities of creating/editing simple and complicated animations using the tool. I worked with Roi and Wenqi on this project.

Concept : Roi already knew what he wanted to do when we started. He was keen on making a promo video for a non-existent product. One source of reference that we looked out for was the animated startup videos that are prevalent these days. We were aiming to construct something that promoted a product but also talked about a societal issue at the same time. The theme of global warming was decided, and we went along with the idea about a product that can modulate weather. The animation was staged in a dystopian society taken over by global warming, one where people are trapped in floods, but are happy with the consumerist satisfaction of being capable of modulating weather for their individual selves.

Sketches :

  weatherama-character-design-bear weatherama-character-design-boy weatherama-character-design-girlboy-holding-box


Work-distribution and Techniques Used : The work was equally split between the three of us. I picked up the character designs and part of the animation. Roi picked up the opening scene  and the sound-over. Wenqi worked on the informercial and the sound effects. I used Photoshop to create the environments and the characters. I made sure to construct everything in high resolution so that we don’t encounter pixelation. It didn’t work out so well, though. The buildings in the opening scene had some pixelation issues since I underestimated the resolution of the video. Eric made some seamless informercial visuals using SketchUp and After Effects. And Roi constructed a brilliant sound-over for the animation, along with a great opening scene.

Tools Used : Adobe After Effects, Adobe Photoshop, SketchUp

Animation Assignment 1 : Stop-Motion

Animation / Week 1 / Stop-Motion

First week of animation dealt with the principles of animation and stop motion, examples and techniques of which were shared with us. For a long time, I had been considering attempting stop-motion animation using my illustrations. However, due to the one week timeline, I decided to make something that was simpler and quicker to achieve. I worked with Mithru and Scott on the animation piece.

Concept : We were circling our heads around many ideas. One thing that we agreed on was that we didn’t want to do a funny story, and do something with an idea/purpose instead. We settled on presenting the evolution of interaction through the decades. The idea was to star two people at a table, who begin interacting with conversations, gradually move to written letters, text messages and eventually VR.

Technique : We used pixelation for creating the piece. I worked on the camera, while Mithru and Scott starred in the video. Through the process, we were continuously exchanging ideas and how to execute the next steps. Both Mithru and Scott held their positions for long stretches of time, which was quite commendable. The output, as a result, turned out to be great.

Tools used : DragonFrame, Canon 5D.

Credits : Mithru Vigneshwara, Scott Reitherman, Utsav Chadha

This initial experiment with stop motion was very interesting, and has opened me up to exploring the technique further to make moving images using illustrations.

Pcomp: Final Project Proposal: TalkTable

Introduction to Physical Computing / Following Weeks / Final Project Proposal

Project Description

With the increasing affinity and accessibility to mobile phones, it’s oftentimes noticed that people tend to resort to their handheld devices in social situations such as family dinners, restaurants and bars. The interaction with the cellular device gains prioritization over actual conversations with the people present in that space.

Talktable is an attempt to separate the individual from the mobile phone and try to encourage them to engage in first-hand conversations.

Design: TalkTable is a surface with designated sockets at each edge, that can hold a mobile phone. When phones are inserted in each socket, a screen activates with content that the participants can talk about. If any one of the phones is withdrawn from the device, the visual interaction is interrupted and the participants have to start over.

Participants: Two or more

Phase 1: The initial phase aims at instating the physical capability of detecting phones in the designated sockets and triggering a visualization. This visualization acts as a third party that starts the conversation and tries to maintain it. For the initial prototype, I’m planning to display a list of topics ranging from films, art, television to daily lives and religion. Once the participants select a topic of discussion, the interface displays selected questions. I’m also planning to integrate speech recognition into this program so as to be responsive to the discussion and interrupting it only when necessary. Tools: I’m planning to execute the initial phase using a computer screen and physical sockets for the mobile phones. In regards to tools, I’ll be using an Arduino Uno for the physical interactions and p5 for the visualizations.

Phase 2: The second phase involves taking this interaction to a surface model. This will require some fabrication, and perhaps an Raspberry Pi to control an LCD screen.

Phase 3: The third phase requires more time and work. It involves pulling data from the social feed (Facebook, Twitter, Spotify and Intagram) of the participants and curating the topics and discussions. This phase is out of the scope for the present semester.

The eventual idea is to make social spaces more interactive, and appealing to people. The idea aims to make it easier for participants to get to know each other in situations such as conferences, orientation events and the like.

System Diagram

Below is a rough system diagram of the device as it’s intended to be.


Estimated Bill of Materials



Week 1 : November 16 – 23

  • Collect all the materials needed for the first phase of the project
  • Figure out the circuit diagram and conduct a basic test of the sockets, using breadboards and mock set up
  • Figure out the visualization. Will it be a game or a list of topics or a visualization of the users’ social feeds

Week 2 : November 23 – 30

  • Art Strategies final project
  • Animation After Effects project
  • While fabricating for art strategies, also fabricate the casing for the sockets

Week 3 : November 30 – December 7

  • Create a barebones circuit within the fabricated sockets
  • Learn p5.js and start writing the interface
  • Complete the initial workflow, which includes introductory messages, topic lists, and basic computer responses based on the choices that the user makes
  • Run the circuit in sync with the visualization

Week 4 : December 7 – 14

  • Add speech recognition to the program
  • Make the program as responsive as possible
  • Consolidate the circuit and finish the fabricated parts
  • Develop a presentation


Present Status and Video Demo

Presently, on December 6, I’m running behind on the circuit bit, due to the failure of phone detection using conventional FSRs. I will be using making my own FSRs customized for the sockets. Besides that, the code looks in pretty good shape. The introduction and the topic listing workflows are ready. Things pending with the code are speech recognition (which I’ll initiate post December 7) and addition of more information and responsive messages to the data set that I’m using for the visualization. Regardless, I’ve prepared a video demonstration for the project which gives an idea about how the final project will look like. If there’s time on my hands, I can consider taking the project to Phase 2 (refer the introduction).