How does librosa work?

Before getting into the details of how do handle large files, it will help to understand librosa’s data model more generally.

Early in the development of librosa, we made a conscious decision to rely only on numpy datatypes, and not develop a more structured object model. This decision was motivated by several factors, including but not limited to:

1. ease of implementation,
2. ease of use,
3. ease of interoperability with other libraries, and
4. syntactic similarity to previous MATLAB-based implementations, as well as theoretical (mathematical) definitions.

What this means, is that rather than having object-oriented code like:

x = not_librosa.Audio(some parameters)  # an object of type not_librosa.Audio
melspec = not_librosa.feature.MelSpectrogram(x)  # an object of type not_librosa.feature.MelSpectrogram

you instead get a more procedural style:

y, sr = librosa.load(some parameters)  # a numpy array
melspec = librosa.feature.melspectrogram(y, sr)  # another numpy array

As a result, it’s fairly easy to move data out of librosa and into other python packages. Back in 2012, theano and scikit-learn were prime targets back in 2012. These days, it’s more about tensorflow or pytorch, but the principle is the same. Having our own object interface to audio and features would get in the way, even if it would have made some design choices easier.

What’s the problem with large files?

Both approaches (objective and procedural) described above have their pros and cons. The pros of the procedural approach are listed above, but one of the drawbacks that we inherit from numpy is that the entire input y must be constructed before melspec can be produced. This is ultimately a limitation of the numpy.ndarray type, which is explicitly designed as a container for fixed-length, contiguous regions of memory with consistent underlying data types (eg int or float). For short recordings — our most common case in librosa — this is fine: recordings typically fit in memory and are known in advance. However, when the audio you want to analyze is long (e.g., hours) or streaming from a recording device, ndarray is not an appropriate container type. We knew this in 2012, but decided to optimize for the common case and deal with the fallout later.

Now, if we had gone for an objective interface, we could have handled these problems in a variety of ways. For instance, it would have been easy to abstract away time-indexing logic so that data is only loaded when it’s requested, eg Audio.get_buffer(time=SOME_NUMBER, duration=SOME_NUMBER). Or we could have used the object’s internal state to maintain a buffer in memory, but not load the entire recording from storage, and provide an interface to seek to a specific time position in the signal. Various other libraries implement these kinds of solutions, and they can work great! But they do come with a bit of additional interface complexity, and might limit interoperability.

Streaming and generators

The solution that we ultimately went with in version 0.7 is to use Python generators. Rather than load the entire signal at once, we rely on soundfile to produce a sequence of fragments of the signal, which are then passed back to the user. At a high level, we would like to have an interface of the form:

for y in librosa.stream(some_parameters):
    some_function(y)
    ...

where y now refers to a short excerpt of the much longer recording in question. However, this raises a few more questions:

1. How big of an excerpt should we use?
2. How do two neighboring excerpts relate to each-other?
3. Can this be used with every function in librosa?

To dig into those, we have to think a bit more about how librosa represents data.

Samples, frames, and blocks

An audio buffer y is typically viewed as a sequence of discrete samples y[0], y[1], y[2], .... Most audio analyses operate at the level of frames of audio, for instance, taking y[0] ... y[2047] as one frame, followed by y[512] ... y[2047 + 512], and so on. Each frame here consists of exactly 2048 samples, and the time difference from one frame to the next is always 512 samples. (These are just the default parameters, of course.)

For most analysis cases, e.g. those based on the short-time Fourier transform, frames are modeled independently from one another. This means that it would be completely valid to process one frame entirely before moving on to the next; and indeed, many implementations operate in exactly this fashion. However, this can also be inefficient because it makes poor use of memory locality, as well as data- and algorithm-parallelism. It is generally more efficient, especially in Python/numpy, to operate on multiple frames simultaneously. This naturally incurs some latency while buffering data, but the end-result leads to improved throughput.

Now, a naive solution here would be to simply load a relatively long fragment y consisting of multiple frames, and process them in parallel before moving on to the next fragment. The tricky part is handling the boundaries correctly. If the hop length (number of samples between frames) is identical to the frame length (number of samples in each frame), then frames do not overlap, and we will not get into trouble by processing data in this way. However, if frames can overlap in time, then so should the longer fragments if we are to get the same answer at the end of the day. This is where we need to be a bit careful.

filename = librosa.util.example_audio_file()
sr = librosa.get_samplerate(filename)
stream = librosa.stream(filename,
                           block_length=256,
                           frame_length=4096,
                           hop_length=1024)
for y_block in stream:
    # Process y_block

Details

There are a few things to be aware of when using stream processing librosa.

First, following on our previous post, librosa.load will (by default) resample the input signal to a given sampling rate. However, this resampling operation needs access to the full signal (or at least quite a bit of the future) to work well, so resample-on-load is not supported in streaming. Practically, this means that you’ll need to be aware of your sampling rate and analysis parameters in advance, and be sure to carry them over across all downstream processing.

Second, librosa’s analyses are frame-centered by default. This means that when you compute, say, D = librosa.stft(y), the kth column D[:, k] covers a frame which centered around sample y[k * hop_length]. To do this, the signal is padded on the left (and right) so that D[:, 0] is centered at sample y[0]. This will cause trouble if you call librosa.stft(y_block), since the beginning (and end) of each block will be padded, and they would not have been padded had the entire sequence been provided to stft at once. Consequently, librosa does not support frame-centered analysis in streaming mode: frames are assumed to start at sample y[k * hop_length] rather than be centered around them.

As a general rule, always remember to include center=False when doing stream-based analysis:

for y_block in stream:
    D_block = librosa.stft(y_block, n_fft=4096, hop_length=1024, center=False)

and of course, be sure to match the frame and hop lengths to your block parameters.

What does and does not work?

Not all analyses support stream processing. For instance, anything that requires total knowledge of a sequence, such as recurrence matrix generation, will clearly not work. A bit more subtle are methods that rely on resampling, such as librosa.cqt.

However, any STFT-based analysis (such as most of the librosa.feature module) should work fine, and this already covers a large proportion of use cases.

The example gallery includes a notebook which demonstrates how to do stream-based processing with STFT and pcen normalization.

Summary

Block-based processing allows some, but not all of librosa’s functionality to apply easily to large audio files.

While, in principle, this could also be applied to online streaming from live recording devices, we don’t yet have a stable underlying implementation to rely upon for this, and hesitate to make any general recommendations.

If, at some point in the future, streaming sample rate conversion becomes viable, we will look at relaxing some of the constraints around resampling (e.g., on load or within cqt).

What is a sampling rate?

Before diving into the details, we first need to all get on the same page about what a sampling rate is. Audio in the real world happens in continuous time, but computers don’t have infinite precision, so we approximate continuous signals by collections of discrete samples. The sampling rate — typically in the digital signal processing literature, or sr in librosa — is defined as , where is the amount of time (in seconds) between successive samples. Equivalently, is the number of observations per second in the discretely sampled signal. It’s a basic fact, a theorem due to Nyquist and Shannon, that if a continuous signal has no content above some frequency , then a sampling rate suffices to reconstruct the signal without introducing aliasing artifacts. Typically we go the other way: fix a sampling rate , and then filter the signal to eliminate any content above before sampling.

For a fixed sampling rate , a digital signal is represented as a sequence of samples: (for ), where the sample corresponds to the value of the signal at time . This gives a general rule for converting between units of samples and units of time, which is helpful to have in the back of your head when reasoning about software interface design later on.

Compact discs (remember those?) used a standard sampling rate of 44100 Hz. This is partly because typical human perception tops out around 20000 Hz (hence ), and partly due to historical accidents. Unlike physical media (CDs), digital audio files (.WAV, .MP3, and so on) can have arbitrary sampling rates. Rates of 16000, 22050, 32000, 44100, and 48000 are all relatively common, and you can’t rely on consistency from one file to the next. Fortunately, sample-rate conversion (or resampling) methods allow us to change the sampling rate of a digital signal as needed.

Why not use the file’s native sampling rate?

When designing the librosa API, we had a few goals that weren’t always necessarily in agreement.

First, we wanted it to be relatively simple to use, and have consistent default parameters shared across all functions. As Gael Varoquaux reiterated in his keynote at SciPy 2017: consistency, consistency, consistency! Having standardized default parameters means that a user is less likely to be surprised by unexpected behavior when working with different parts of the library. This makes the software easier to learn and use.

Second, we wanted the default parameters, such as frame length (number of samples in one frame of a short-time Fourier transform), to be expressed naturally. In audio signal processing, there are two ways this could have gone: either specify the frame length as a duration (in seconds), or as a number of samples.

• Expressing frame length as a duration is nice because it uses real, physical units and is independent of the sampling rate: a 1-second frame occupies the same amount of “content”, whether the sampling rate is 8000 or 16000 Hz. However, this would mean that the same function applied to two signals with different sampling rates would produce outputs of different dimensionality, and would therefore not be directly comparable.

• Expressing frame length as a number of samples, on the other hand, always produces outputs of comparable dimension. However, the meaning of the contents can change, depending on the sampling rate. As it turns out, designing for dimensional compatibility is much more convenient when you consider that subsequent processing stages will need to know the dimension of the input data to operate correctly. It’s easier to fix the sampling rate first, and then design around that, than vice versa.

• Expressing frame length in terms of samples has the added bonus that we can design for efficiently calculable Fourier transforms. Most fast Fourier transform (FFT) implementations work best when the number of samples is an integral power of 2, and worst when the number is a large prime. Although the latter case is unlikely in general, defining frames in terms of samples leaves us in a better position to guarantee efficient implementation.

Third, we wanted to minimize the chance of users (i.e., myself) making simple mistakes by not accounting for the sampling rate. An analysis script, once written, should behave consistently across different input signals, and not depend strongly on the exact sampling rate. In practice, this meant that every analysis script involved immediately standardizing the sampling rate of a file after it was loaded, so it made sense to combine the two steps into one (by default) since it’s the most common case when dealing with collections of audio.

After a bit of discussion, we pretty quickly decided that resample-on-load was the best compromise available for achieving consistency and simplicity at the API level.

Okay… but why 22050 Hz? Why not 44100 or 48000?

It’s true: 44100 Hz is essentially the standard for “high (enough) quality” audio storage, and it would have been a sensible default.

However, we decided for the lower rate of 22050 for two reasons:

1. It cuts down on memory consumption,
2. 44100 was overkill for our most common tasks.

The first point is obvious, but the second point deserves a bit more discussion.

When we were initially developing librosa, our main use cases were analyzing corpora of old jazz recordings, music more generally, and speech signals. While humans (young ones, anyway) can hear up to around 20000 Hz, it’s possible to successfully analyze music and speech data at much lower rates without sacrificing much. The highest pitches we usually care about detecting are around , well below the 11025 cutoff implied by . There’s certainly content above 11025 Hz, but it often turns out to be noisy or redundant with the lower parts of the spectrum, and not so informative for semantic analysis tasks like instrument classification, rhythm analysis, chord recognition, and so on.

I still don’t like it. What can I do?

Sure, 22050 isn’t right for every situation. It’s a default setting, but not a requirement!

You have a few options though. First, you can always bypass resample-on-load by specifying sr=None:

y, sr = librosa.load(filename, sr=None)

You will need to remember to pass sr around to all relevant functions, and make sure your frame and hop lengths are tuned accordingly.

A slightly fancier alternative is to use the presets package, as illustrated in the example gallery to change the default. This approach uses some pythonic hackery to intercept function calls into a package (like librosa, but it works more generally), and gives you the option to override default parameter values. The end result is not so different from carrying the specific sr value around with you, but it does make for slightly cleaner code since the defaults can all be set globally in a preamble, rather than replicated everywhere. For example:

from presets import Preset
import librosa as _librosa
librosa = Preset(_librosa)
librosa['sr'] = 44100
librosa['n_fft'] = 4096
librosa['hop_length'] = 1024

would effectively change the default sampling rate to 44100, and double the frame and hop lengths n_fft and hop_length from their standard default values. These new defaults would persist throughout your coding session.

Summary

Resample-on-load was ultimately a usability choice. We felt that the initial computational effort at load time was a worthy trade-off if it could simplify software usage without sacrificing accuracy or quality in the most common cases.