I am a Swedish computer science & media technology student who loves to work with exciting projects that involve Visualization, Software Development, Human Computer Interaction and Education. View full presentation

This blog contains random thoughs and some notes from the development of Continuous — a modern WebGL based math visualization tool to help students learn algebra and calculus. Show archive

emil axelsson

Castafiore - an audio visualizer for Spotify

Fall 2013 Spotify released a new Audio API which lets you create apps within Spotify that make use of the waveform and the spectrum of the currently playing song. I wanted to try it out and see if I could create something similar to the old visualizations from Winamp and Windows Media Player back in the early 2000's (while also trying to create something new and interesting).

Luckily Spotify has support for WebGL content inside third party apps, which makes it possible to take advantage of the computation power of modern graphics hardware.

Meet Madame Castafiore! Lately I have been naming my hobby projects after Tintin characters, and since this one is very much about music I had no other option than borrowing the name from The Milanese Nightingale.

The idea

A good friend of mine recently made a painting by letting paint drops run down on a canvas (not a HTML5 canvas, I am talking about the real deal now). I thought it would be cool to create a Spotify app that automatically splatters paint on a surface depending on the dynamics of the playing song. The paint should then blend with other colors while flowing down the surface.

How to do it?

I found this paper on realtime simulation of water drops on the GPU. To simulate water drops, the authors suggest using a texture where each pixel intensity represents the amount of water in the corresponding area. Once a pixel intensity exceeds a certain limit, the gravitational force on the water will start overcoming the frictional force and it will start flowing down the surface, leaving some traces behind it. The more water, the more speed.

The paper also presents a way to render the water by adding another render pass, calculating its surface normals and deriving the directions of reflection and refraction. I stole many of these ideas right away and implemented them in JavaScript and GLSL.

I wanted my drops to have colors, so I needed to come up with a way for the simulation to take this information into account as well. My first approach was to store the amount of cyan, magenta and yellow pigments in the texture's r, g, and b channels, and the amount of water in the alpha-channel. When blending colors I would add up the pigments, and before rendering I would convert from CMY color to RGB. However, this makes it impossible to represent white paint. Using (0, 0, 0, 1) as CMYA components would naturally mean no pigments and a lot of water.

I would need to store both absorption and reflectance for the three color channels in order to allow for both black, white and transparent drops. But since a texture in WebGL can only store four values per pixel, and multiple render targets is not yet supported in the WebGL standard, I decided to use a simpler model where I assume the pigment concentration to be constant. This means that transparent water cannot longer be represented. I consider it a valid tradeoff, since white color probably will look more interesting than transparent water in a audio visualization. When blending colors, I simply calculate an average of the colors being blended, weighted on the contribution of each color.

This is what I had come up with after two days of frenetic coding. The colors were here selected from The Scream by Edvard Munch and blended using the first approach discussed above.

How to drop the drops when the bass drops?

The whole point with an audio visualization is its way to react to the dynamics of the music. I wanted color splats to pop up when there are large changes in the volume of the song. Through Spotify's Audio API, the application is fed with short-time Fourier transforms about ten times per second. By calculating the spectral flux between the frames, I could detect distinct volume changes in the audio track.

Some songs are much more dynamic in their volume than others, and it is not uncommon that one song has parts with much more dynamics than other pieces of it. I found that emitting splats when the flux exceeded a fixed threshold would cause the visualizer to go completely bananas during some songs and become totally inactive during others. I solved this with a buffer keeping track of the latest calculations of the flux, and normalizing the current value with respect to the average flux in the buffer. The size of each splat depends on how much the flux exceeds the threshold.

Colors from the album art

Thanks to Spotify's API, it was easy to fetch the album art of the currently playing song. To make the visualizations more different from song to song, the colors are chosen from the cover. At some volume peaks I also make a slight random color adjustment to the dark background color, giving the whole thing a little more disco feeling.

Listen to Håkan Hellström, or try it yourself!

I made a video of what Castafiore painted while listening to Håkan Hellström's Du är snart där. You can watch the video here or download the source code from my Github repo.

Client-Server Communication in Continuous

The last few days, I have been working on what I call the SimpleParser, which has as its responsibility to read math expressions as they are entered by the user and create the expression trees that I mentioned in my last post.

For the prototype/demo of Continuous I have previously implemented a relatively simple shunting yard algorithm, which could handle the basic precedence rules of mathematical operators. The implementation was very compact - expressed in only two or three JavaScript functions - and was pretty much a translation of this pseudocode to JavaScript. My lexer (the preprocessing step that separates all the symbols, operators and numbers in the text) consisted of the longest regex string I have ever written. It was a quick hack and it worked as a proof of concept but it was completely unmaintainable.

So I decided to rewrite the whole thing and add support for parsing vectors and matrices. The parser I wrote is based on the design by Vaughan Pratt, who wrote a paper on "Top down operator precedence" back in 1973. Using this technique, I succeeded in writing a much more readable and modular design than the previous one. I will not describe further right now how it works, but I recommend this excellent article if you are interested in doing something similar.

Now, I want to talk about today's breakthrough!

First round-trip to the Sage Server

With an almost fully-functional parser and a formatter ready, I had all the ingredients I needed to start building on the communication between the the Continuous browser application and a node.js-server, that in turn communicates with Sage. I made this UML-inspired illustration of the program flow when the client is asking the server to simplify an expression, for example sin(x)^2+cos(x)^2 (which always simplifies to 1).

Oh, gosh, that is a lot of modules, you might say. Let me justify some of my design decisions.

• Super-simple interface for the client. All the client needs to know is that is has some kind of connection where it can send request objects and receive response objects. This is good because if the current web socket implementation needs to be replaced, this can be done without changing any of the code in the actual visualizations.
• The client-side SocketWrapper. I have wrapped the web socket inside a SocketWrapper which I assume always has a send-method that is ready to be invoked. The browser native WebSocket's send-method will through an error if you try to write to a socket that has been closed if the connection was lost. What I did here was to implement a queue of messages inside the SocketWrapper, which is flushed as soon as a connection is available again. This further simplifies things for the client, that would otherwise have to put all of its communication code in a callback that would be invoked from the WebSocket when it was ready for data transfer.
• Formatters and Parsers. The client and server work with Request and Response objects. One approach would just be to just call JSON.stringify and JSON.parse and hope that all the information in these objects would be sent over correctly. But since I want to be able to wrap expression trees inside the requests, they would then recursively be JSON-formatted as well. I would rather use my SimpleFormatter and SimpleParser for this, since these deal with a much more compact format than JSON. Writing my own Formatters and Parsers gives me much more control.

Promising results

I managed to put the above structure together today, and successfully sent my first requests from the Continuous web app to the node.js server. I put together a tiny application where you can edit a mathematical expression and see Sage simplify it live on every keystroke. I was stunned when I saw the results. The possibilities that this brings to dynamic 3D visualization of math are incredible, and I can hardly wait to add a graphics engine on top of all this...

Kalkyl getting some help from Sage

Kalkyl is the module in Continuous that takes care of symbolic math operations. Why does Continuous need to handle symbolic math? Well, it would be possible to create visualizations with only numerical calculations, but the whole application would become much less flexible.

How about numerical calculations using eval?

Let's say that we want the user to be able to fill in a function representing a surface of their own choice. Since JavaScript has an eval-function, we could simply substitute x and y in the user's expression by numerical values and then evaluate the string. Do this systematically with small intervals for x and y and Voilà, we have enough data to feed into our 3D engine to create a surface.

But then let's say that we want to visualize the partial x derivative by applying a color gradient to the surface, where red means large positive derivative and blue means large negative derivative. Of course we would need to calculate the derivative in every point. We could calculate a numerical approxiamation, but come on! The results would be inaccurate, and if we wanted to show a mathematical expression of the derivative to the user, it would be impossible.

Or let's say that we have a function that operates on matrices and outputs a vector. JavaScript does not natively handle matrix arithmetic, so the eval-idea fails.

Hello, expression trees

What Kalkyl does is that it parses input from the user and builds up data structures called expression trees. Finding the derivative of an expression is just a matter of traversing the tree and applying differentiation rules from calculus. These trees also make some simplification of expressions possible by searching the trees for known patterns that can be rewritten in a simpler way.

Symbolic integration and solving equations?

An expression tree and some differentiation rules is a good start, but thins get a bit more complex if you want the program to be able to solve equations or integrate arbitrary expressions Wolfram Alpha-style. Although it would most certainly be amazingly exciting to try to implement a full-fledged Computer Algebra System (like Mathematica or Maple) in JavaScript, I think I pass. At least for now. So what to do? Should we be satisfied with the limited set of symbolic operations that was feasible to implement in JavaScript without devoting an entire life to mathematics? No way!

The open source community comes to the rescue, I hope.

I found an open source alternative to Mathematica and Maple, written in Python and published under the GPL license. Sage is capable of integrating expressions, solving equations and much more. My plan right now is to get Sage running on a web server, and let Continuous ask Sage for help when Kalkyl is not smart enough to do the math.

As some of the computation moves from the browser to a server, the application has to be designed to be much more async in its nature. It will be a real challenge to avoid callback soup. This article seems to explain some common ways to tackle this challenge, so I suppose reading it thoroughly will help me on my quest.

Classes in JavaScript?

I am right in the middle of the process of rewriting Kalkyl which is the module of Continous responsible for working with symbolic math. Since this module will act as a core part of the application, I find it extremely important that the code that goes into this module is childishly simple to read and understand. JavaScript is a language that allows the programmer to structure applications in a thousand ways. In Java, for example, a file must contain excactly one class – not less, not more. To see what methods that are available on an object, you can simply scroll through the code in the class definition.

In JavaScript, on the other hand, methods and fields can be dynamically added to any object at any time. But, if one were to do this, one would quickly end up with a complete mess. Instead, there are some different patterns that I have seen many programmers use. Some of these patterns are completely different from how a class in Java, or C++ is structured and some are trying to mimic the "classical" approaches.

The Module Pattern

The top hit on Google in a search for "JavaScript module pattern" descibes techniques that involve using closure to keep variables and functions private. To achieve something similar to class inheritance, it is suggested that you loop through the members of another object and add references to these to this. This approach completely ignores the prototypal nature of JavaScript. It breaks the instanceof operator, and I find it very weird to scan a file for loops copying references to methods to figure out who the parent class is.

Assigning a new prototype to the constructor

Another pattern that for example Three.js uses is to first create a constructor function, and then add methods to the prototype. The functions end up being stored on the __proto__ member of the object, which means that calls to this.foo() are delegated to this.__proto__.foo(). The files end up with a more clear list of the methods available, which I like. But things get a bit messy when you want inheritance. To inherit from another "class", you have to create a new instance of that parent class and set the prototype of the subclass' constructor to the instance of the parent class. It's not only difficult to say, it's also difficult to understand.

Quack.js

I borrowed ideas from both these patterns when I wrote Quack.js, which is an extremely lightweight library that lets JavaScript programmers define classes like this:

```  var SomeClass = quack.createClass({
constructor: function (bar) {
this.foo = bar;
},

someMethod: function () {
...
}

});

var SomeSubclass = quack.createClass(SomeClass, function () {
someNewMethod: function () {
...
}
});

```

I also added Java-style abstract classes and interfaces to Quack. quack.createClass will complain if you try to create a class with abstract metods without declaring it abstract. You can not instanciate abstract classes. I do not perform any type checking to see if methods are called with arguments of correct types, but I still find these features useful. Declaring classes and methods abstract is a very nice way of showing your intentions, and I would say that it makes code more readable.

```  var SomeAbstractClass = quack.createAbstractClass({
constructor: function (bar) {
this.foo = bar;
},

someMethod: new quack.AbstractMethod()

});

var SomeInterface = quack.createInterface([

'someOtherMethod'

]);

var SomeSubclass = quack.createClass(SomeClass, [SomeInterface], function () {
someMethod: function () {