MIR Overview

MIR means Music Information Retrieval: the part of audio analysis that turns sound into musical answers — tempo, beat positions, key, chords, pitch, timbre, and structure. This page is a map. It groups the terms you will meet across the docs and shows how they build on one another, so you know which feature to reach for and where it is computed.

These terms are grouped on purpose. They are not isolated functions — almost every MIR task is built on the same time–frequency foundation. Understanding that shared foundation once means the individual features stop looking like a long, unrelated list.

New here? Read this as orientation, not reference

This page explains how the pieces relate. For call signatures, go to the JavaScript API or Python API; for how each one is computed, see DSP Implementation Notes.

The shared pipeline

Most MIR features are derived from a small set of intermediate representations. You rarely build these by hand — libsonare computes them internally — but seeing the flow explains why so many features share parameters like nFft and hopLength.

Shared analysis pipeline

One STFT feeds every branch; the onset envelope is read from the mel spectrogram. Section analysis is a timbre-and-harmony task, not a rhythm one: its boundaries come from the mel/MFCC and chroma branches, never from the onset envelope.

Sharing these intermediates only pays off inside a single whole-track analysis. analyze (MusicAnalyzer) computes the STFT and its derivatives once and reuses them across BPM, key, chord, and section results.

The per-feature helpers do not share anything

detectBpm, detectKey, detectChords, and the section helpers each start again from the raw samples, and there is no cache keyed on the audio. If you want more than one answer about the same track, call the whole-track analysis once instead of assembling it from the individual helpers: it computes the STFT and chroma a single time.

Which question, which feature

You want to answer…	Reach for	Built on	Call (JavaScript / Python)	CLI
How fast is it? Where are the beats?	BPM / beat tracking	onset strength	detectBpm, detectBeats / detect_bpm, detect_beats	bpm, beats
What key is it in?	key detection	chroma	detectKey / detect_key	key
What chord is playing?	chord recognition	chroma	detectChords / detect_chords	chords
Where does the chorus start?	section analysis	MFCC (timbre) + chroma (harmony) + energy	analyzeSections / analyze_sections	sections
What note is the melody?	pitch / melody tracking	pitch tracking (see Separation and pitch below)	pitchPyin, pitchYin / pitch_pyin, pitch_yin	pitch
What does it sound like (timbre)?	MFCC	mel spectrogram	mfcc / mfcc	none
Can I separate drums from the rest?	HPSS	spectrogram structure	hpss / hpss	hpss
What is the raw frequency content over time?	STFT / spectrogram	the waveform	stft / stft	none
What does the recording space sound like?	room-acoustic analysis	impulse-response (IR) decay or blind free-decay estimates	analyzeImpulseResponse, detectAcoustic / analyze_impulse_response, detect_acoustic	acoustic

Every call takes the samples and the file's real sample rate. See Audio Basics for why the rate matters.

Browser

import { init, detectBpm, detectKey, analyze } from '@libraz/libsonare';

await init();

const bpm = detectBpm(samples, sampleRate);     // one question, one pass
const key = detectKey(samples, sampleRate);

const result = analyze(samples, sampleRate);    // the whole track in one pass
// result.bpm, result.key, result.chords, result.sections

Python

import libsonare as sonare

audio = sonare.Audio.from_file("song.wav")
bpm = audio.detect_bpm()                        # one question, one pass
key = audio.detect_key()

result = sonare.analyze(audio.data, audio.sample_rate)   # the whole track in one pass

CLI

sonare bpm song.wav
sonare key song.wav
sonare analyze song.wav    # the whole track in one pass

Timing: BPM, beat, onset, section

The timing family builds up in layers:

Feature	What it answers
Onset detection	Where notes, drums, or consonants begin: the spikes in an onset-strength envelope.
BPM	How periodic those onsets are.
Beat tracking	Where pulses land on the timeline.
Section analysis	Where longer spans such as intros, verse-like sections, chorus-like sections, and breaks begin and end. Unlike the three rows above it, this one does not read the onset envelope: boundaries come from timbre (MFCC) and harmony (chroma).

Onset is the root of the rhythm family

BPM, beats, and tempograms all start from the same onset-strength envelope. If you want the time × tempo picture behind a BPM estimate, see the tempogram family in Realtime and Streaming.

DETECTOR · ONSET / BEATIDLE

Onsets vs beats — from attacks to a pulse

Scroll to load

Onset detection marks every attack in the audio; beat tracking distils those into the steady pulse you would tap along to. Switch the view, then press play to watch each marker fire as the playhead reaches it.

Detect

Harmony: key, chord, chroma

Chroma compresses frequency content into 12 pitch-class bins (C, C♯, … B), folding every octave of the same note together. That makes it the natural substrate for harmony: key detection estimates the tonal center from the overall chroma distribution, and chord recognition estimates local harmony frame by frame.

Chroma trades octave and timbre detail for harmonic clarity

Folding octaves together is exactly what makes chroma good for key/chord work — and exactly what makes it the wrong tool for melody or timbre, where octave and spectral shape matter. Match the representation to the question.

CHROMA · PITCH CLASSIDLE

Chromagram — harmony folded into 12 bins

NOTETIME →

Scroll to load

Every frequency is folded onto one of twelve pitch classes, so octave is forgotten and the harmony remains. This clip contains only four separated triads, C–Am–F–G, with a one-beat gap between bars. Watch the lit rows move, then play the turnaround.

Spectrum: FFT, STFT, spectrogram

The FFT (Fast Fourier Transform) is an efficient algorithm for the DFT (Discrete Fourier Transform), which converts a block of samples into frequency content.

The STFT (short-time Fourier transform) repeats that over many short, overlapping windows so frequency content can be tracked over time.

A spectrogram is the visual result: time on one axis, frequency on another, intensity as brightness.

Two parameters recur everywhere: nFft (window size — bigger means finer frequency resolution but blurrier timing) and hopLength (step between windows — smaller means more frames and smoother motion). The trade-off between frequency and time resolution is fundamental, not a libsonare quirk.

STFT · SPECTRALIDLE

STFT — seeing time and frequency at once

FREQ · REL dBTIME →

Scroll to load

A tone gliding from 400 Hz to 1 kHz, with four clicks. Each column is one short-time spectrum; brighter means more energy. Change nFft: at 256 the clicks are thin lines and the tone is a band about 170 Hz wide; at 4096 the tone is a line about 20 Hz wide and each click smears across about 280 ms.

nFft (window)

Perceptual features: mel, MFCC, CQT, VQT

These perceptual features answer different questions:

Feature	What it emphasizes
Mel spectrogram	Frequency resolution shaped toward human hearing: fine detail low, coarser detail high.
MFCCs (mel-frequency cepstral coefficients)	A compact "timbre fingerprint". It is computed by taking the mel spectrogram, compressing its loudness with a logarithm, then summarizing each frame into a handful of numbers that capture overall spectral shape rather than exact pitch.
CQT / VQT (constant-Q / variable-Q transform)	Musically spaced bins — geometric spacing with a configurable binsPerOctave (12 by default, so one bin per semitone; 24 or 36 are common for chroma and tuning work). Useful when pitch relationships matter more than equal-Hz spacing.

You can also run these transforms backwards for previews and debugging — see Inverse Features.

Separation and pitch: HPSS and pitch

HPSS means Harmonic/Percussive Source Separation. It splits sustained pitched material from transient hits by using their spectrogram shapes.

Component	Spectrogram shape
Harmonic content	Mostly horizontal lines.
Percussive content	Mostly vertical lines.

Separating them first often improves downstream tasks because drums and pitched instruments otherwise confuse each other.

HPSS · FULL MIXIDLE

HPSS — splitting the tune from the drums

FREQ · REL dBTIME →

Scroll to load

Sustained harmonic sounds tend to form horizontal ridges in a spectrogram; transients tend to form vertical streaks. HPSS uses median filters and masks to separate these patterns. Switch between the full mix, harmonic and percussive layers and audition them. This separates spectral structure rather than instrument identities: pitched attacks can enter the percussive layer, and sustained cymbals can enter the harmonic layer.

Layer

Pitch estimation tracks F0, the frequency that governs waveform periodicity and, for harmonic sounds, the spacing between partials (the individual frequency components; see Audio Basics). F0 need not appear as a spectral component, so it is not necessarily the lowest or strongest observed peak. An upper harmonic can dominate a bright brass note or a bass heard through a small speaker, and the pitch can remain perceptible even when the F0 component is absent. Inharmonic sounds such as bells and cymbals may have no single well-defined F0. Estimators therefore use waveform periodicity rather than the loudest spectral peak. This is useful for melody, vocals, monophonic instruments, tuning checks, and transcription-style workflows.

Adjacent: room acoustics

Room-acoustic analysis is adjacent to MIR. It describes the space captured by the recording rather than the notes, rhythm, or form of the music.

Use direct IR analysis when you have a clean impulse response. That path measures RT60 (the time reverberation takes to decay by 60 dB), EDT, C50, C80, D50, and band decay.

Use blind acoustic estimation when you only have a normal recording. That path reports room-decay cues with a confidence value because the free-decay evidence may be weak or missing. See Room Acoustics.

Implementation notes

libsonare exposes MIR functions across browser/WASM, JavaScript, Python, native bindings, CLI, and C++ APIs.

Many features share intermediate representations such as STFT, chroma, and spectral energy curves. That sharing lives inside MusicAnalyzer: a whole-track analyze computes each intermediate once and reuses it across BPM, key, chord, and section results. The single-feature entry points hold no cross-call cache, so each one rebuilds what it needs from the samples.

The browser demos are built for interactive use, but each one emphasizes a different part of the library:

Demo	Main role
Music Analysis Studio	Full-file MIR: BPM and a local tempo curve, key, chords with their Roman-numeral function, sections, and the A4 tuning estimate.
Realtime views	BPM, key, and chord estimates that update over time through StreamAnalyzer.
Mastering Studio	Measurement-style APIs such as loudness measurement, reference comparison, and report export.

Seeing those demos side by side shows which pieces are reusable across analysis work and finishing work.

Related: Introduction, Audio Basics, JavaScript API, Room Acoustics, DSP Implementation Notes, librosa Compatibility