Somnolent Listener

Nature of Code / Week 4-7 / Final Project

Title

Somnolent Listener – The hearing impaired note taker and visualizer

Introduction

Somnolent Listener is a speech recognition system that listens to speech, interprets it and produces results in various forms. It is prone to making some mistakes while listening, and hence has been titled using the adjective ‘somnolent’.

The entire idea is centered around making speeches and lectures more interactive. The listener records the data producing real time subtitles, maintains a transcript of the entire recording and also takes notes. There are three associated visualizations that can be switched using the arrow keys or the number keys. Visualizations one and two are particle systems that change based on the volume level of the input. Visualization three uses the recorded transcript to generate keywords. Each keyword has a weight based on the number of times it is repeated. And each keyword also has associations with other keywords based on how close together they were spoken in time. It basically acts as an automatic note-taking system for the student/listener.

Video Demo

Applications

1. The system acts as an automatic note making device for a student or listener. This would ensure that a person can focus undivided attention to the speaker or lecturer and not worry about capturing everything that’s spoken.
2. For a person whose first language isn’t English, it can often become difficult to follow what is being spoken. The subtitles assist such an audience member to understand the words.
3. The transcript can be used to look back at the lecture and use references. Or perhaps, if one is sleepy or inattentive in class, they could use it to know what had been spoken.
4. The visualizations provide information around the amplitude levels. They can also be changed to represent frequencies, wavelength et cetera. This is not really something new, but it looks cool haha. And perhaps, can make a speech more interesting.

Future Steps

I’d like to add more visualizations and more capabilities to this system. I also intend on making the listener more sophisticated. Plus, the keyword selection and matching algorithm requires further work. I’m interested in classifying keywords and generating different methods to group words together.

I’m also interested in using this project in an installation context wherein, the listener listens to an input, records it and then reiterates it with the mistakes. In different ways. Perhaps representing what a computer listens and what it understands.

Source Code

The entire code directory can be found here : link

Credits

Thanks to
Daniel Shiffman, instructor and mentor for Nature of Code
Luke DuBois, for p5.speech library
Jason Sigal, for p5.sound library

 

Polar Roses

Nature of Code / Week 3 / Oscillations and Particle Systems

I wanted to explore oscillations for my third assignment, particularly set in a circular motion. Polar roses gave me a lot of ideas, and I tried to experiment with their values. The results were really in accordance with what I had in mind. But it looked cool regardless (haha).

Source code link.

Moving Mandala

Nature of Code / Week 2 / Forces and Vectors


What is a Mandala?
A mandala is a spiritual symbol in Hinduism and Buddhism that was used to represent the universe or the cosmos. It has gradually been adapted as a form of designing structures and psychedelics. The pattern can be found in various places ranging from rangoli designs and album covers to internet art and even hypnosis videos (haha).

Design and Process
Moving Mandala is a simple sketch drawn using the Mover and Attractor classes created by Daniel Shiffman. In fact, the sketch has been drawn on top of Shiffman’s example on the basic attractor.

The whole idea was conceived while I was playing around with the sketch, and trying to develop it into a mandala. I created two partitions and two Mover objects that simultaneously move and constantly change the mandala as they go. The transformations occur as illustrated in the overhead gif.

Ideas
Never having been in the position of creating generative art (but always having wanted to do so), I was struggling to come up with ideas to begin with. I started tweaking with Shiffman’s sketches and soon realized how the force of attraction can be used to generate mandalas. More so, transforming mandalas. I guess force of attraction is not solely applicable to love and magnets (haha?). The results of this small experiment were definitely amusing, and have given me a lot of leads to work with going forth:

– Modulating alpha values (perhaps, using Gaussian distribution) and looking at the results
– Using more partitions and different patterns
– Changing the acceleration in a controlled manner to produce variant mandalas
– Defining forces between the two objects, along with the force between the objects and the center


Source Code

The associated pde files can be found on Nature of Code’s drive.

Life is random()

Nature of Code / Week 1 / Random Walkers, Gaussian Distribution and Perlin Noise

Since I waived Introduction to Computational Media last semester, I didn’t really get enough bandwidth to play around with p5. I did learn it however, as part of my final project for Introduction to Physical Computing. So as soon as I was exposed to the random and noise methods, I cracked my knuckles and started fooling about with p5. Through using the random functions, I discovered how I could easily form celestial visualizations. Below are a few instances :

Links for source code and moving sketches :

Abyss : https://alpha.editor.p5js.org/utsav/sketches/SkMRS_POe
Galaxy : https://alpha.editor.p5js.org/utsav/sketches/HyMef_P_l
Comet : https://alpha.editor.p5js.org/utsav/sketches/BJFF1ldux