In the Information Visualization - Lab Course the aim was to implement a scientific paper in the field of visualization. We choose the audio library visualization by Muelder et al.[1] for implementing.
The basic idea of this paper is, to analyze audio files and generate a fingerprint out if them. Those fingerprints are then checked for similarity, which results in a adjacency matrix and can be presented in graph.
In genral the program consits of two parts:
We implemented the analysis part two times, because of reasons stated below.
Our first approach to generate the fingerprints and the similarity matrix was to use MATLAB. In version R2012b functionality for reading audio files was added.
For generating the MFCCs out of the audio files we used the voicebox library.
The parsing of the audio tags (which later should be used in the graph representation) was done via the id3parse script.
However, this approach proofed to be incredibly time consuming. It would take at least 5 times the length of the song to generate the fingerprint out of it. Furthermore the entire computer was blocked by the haevy workload during the calculations.
therefore we decided to not further use MATLAB but C++ instead. The MATLAB code can be found in src/matlab_implementation and is seperated into two functions, namly addFingerprint.m (which uses searchDirForFiles.m) and genSimilarityMatrix.m.
In our second approach we used C++ to analuze the programm. Our audio analysis application is splitted into three parts which can be found in the subdirectories of src/c_implementation:
This is the only part of the application done in C#, because C++ proofed to be very annoying when it comes to file handling and strings.
The application scans a directory and all subdirectories for audio files, supported by the BASS library (see below).
Source:
This part of the application generates the fingerprints and a list with meta information for the audio files from the previous part.
The application uses the BASS library to read the audio files (including their tags). Lukily it also features a built-in FFT.
For generating the MFCCs out of the audio data, we used external code by Jeremy Sawruk.
To speed the whole thing up we used OpenMP to parallelize the calculation of the MFCCs (each component can be calculated seperatly). Without this optimization it would take around 2 times the length of the song to get the fingerprint. With OpenMp this factor can be reduced to 0.6.
Source:
This small program simply generates the similarity matrix out of the fingerprint file using the inverse euclidean distance.
Source:
Again we implemented this part two times. Both times we used HTML5 and JavaScript to visualize the results from above in a browser. Furthermore both applications feature a dialoge to select the files from the previous programs and include an audio player as well.
The difference between the two parts are the libraries to help us display the graph:
Jit is a very simply tool te generate graphs on a canvas object. However it did not have enough flexibility for our needs. To be more specific: we were not able to take control over the drawing operation, which led to small labels on the display and just straight lines. Furthermore we were not satisfied with the placement of the nodes. Even after several placement iterations, the Jit library failed to deliver a good result.
In this version we used audio.js to play the music. Apart from supporting only the Browser-native file formats, this tool sometimes produced hard cuts during the playback.
Arbor.js is basically a particle system, that visualizes connected particles. The only thing calculated by the library are the positions of the nodes, based on parameters like friction and mass as well as the connection of the nodes. Arbor places the nodes during execution, which made it possible to drag nodes and such. This time we used the buzz library to play the audio files (if they are supported by the browser). This library produced good results.
The User starts with selection the files, which were generated from the previous programs. This is done in a dialog box in the upper right corner. There the user can also specifie the treshold for which similarity value no edges are generated (those similarity values can range from 0 to 1, with 0 beeing complety different audio files and 1 beeing the exact same). Furthermore the user has the option to scale the lables on zoom or not. We recommend to leave this box unchecked for bigger graphs.
After the button is pressed, the particle system is initalized and filled with nodes and edges in the funtion init() of the main.html file.
The display now shows the graph with the nodes moving towards each other and the centrum of the canvas. The entire drawing process is done in the function redraw of the main.js file. Selecting a node by clicking close to it was implemented so the user has the option to play the audio file. When a node is selected it is displayed with an red border and its neighbors (and the edges towards them) have a slightly darker red.
The user can drag a node, which makes it untouchable for the physics simulation. After the user releases the mouse, the physics kick in and the graph adapts accordingly. The implementation of this is done in function initMouseHandling of the main.js file.
To catch every detail of the graph, we can zoom and move around. The former is done by using the mousewheel and the latter by pressing the arrow keys on the keyboard. The implementation of this is done in functions initMouseHandling:scroll and initKeyHandling of the main.js file.
When a node is selected by the user, a small (draggable) window is displayed featuring the audio player. This player not only features basic controls such as play, stop, pause and valume-adaption but also displays meta-data of the audio files. The implementation of this is done in functions in the marked functions of the main.html file.
[1] Muelder, C., Provan, T., and Ma, K.-L. (2010). Content based graph visualization of audio data for music library navigation. In International Symposium on Multimedia.