Architecture

This document describes the internal architecture of libsonare.

Read this page once you are comfortable with the Getting Started guide and your language's runtime page. It is an internal map for people extending libsonare or wiring it into a larger system, not a tutorial. If you only need to call an API, start with Getting Started. It shows how public APIs connect to the C++ core.

How to read the layers

The outer API layers are what apps call. The core and feature layers are where reusable signal-processing work happens. Bindings should stay thin: they translate language shapes into the same C++ behavior rather than reimplementing DSP.

What You Will Learn

By the end of this page you should be able to:

• trace a public call from WASM, C, quick API, or sonare.h into analysis, streaming, effects, mastering, mixing, and engine modules;
• identify which source directories own each subsystem;
• understand where shared DSP, feature extraction, realtime processing, and language bindings meet;
• decide whether a change belongs in a core module, a language binding, a demo component, or documentation.

Module Overview

The layers below run top to bottom: the API layer is what apps call. Spectral analysis and some effects use the shared transform primitives; filters, dynamics, mixing, and other processors have their own signal paths. Within one MusicAnalyzer, selected intermediate features are cached, but separate processors may compute their own transforms. Groups mirror the src/ subdirectories from the Directory Structure section below; nodes inside a group are representative members, not an exhaustive class list — see the Page Map for where each subsystem's full API is documented.

Module Overview

Bindings stay thin: WASM/C/Quick/sonare.h all reduce to the same C++ core, they never reimplement DSP per binding.

The API layer is itself split by runtime. The Python package calls the C ABI through ctypes. The WASM (Embind) and Node native (N-API) bindings wrap the C++ classes directly and use the C ABI for some entry points. Each CLI is built on a runtime: the pip-installed sonare command on the Python package, and the source-built native CLI on C++.

From each runtime to the core

• calls the C++ classes directly
• goes through the C ABI

The C ABI is a layer only some paths cross. Whichever path a call takes, it ends in the same C++ core.

Page Map

If you are looking at...	Read...
analysis/ and feature/	JavaScript API, Python API, librosa Compatibility
analysis/acoustic_analyzer.*, analysis/room_estimator.*, src/acoustic/, or effects/acoustic/	Room Acoustics, Algorithm References
streaming/	Realtime and Streaming
mastering/	Mastering Processors, DSP Implementation Notes, Mastering Assistant
mixing/	Mixing Engine, Mixing Scene JSON, Channel Formats (downmix)
playback/	Playback Renderer, Channel Formats
engine/, transport/, automation/, graph/, rt/	Realtime Engine
midi/ and midi/synth/	Built-in Instruments, SoundFont 2 Player, MIDI Input
arrangement/ (edit model)	Project Editing, Recording and Takes, Project Bounce
mir/ (warp, grid snap, key context)	Warp and Tempo, Realtime and Streaming
editing/ and effects/	Editing DSP, DSP Implementation Notes
sonare_c*.h and binding folders	Binding Parity, Native Bindings, C++ API

Directory Structure

src/
├── util/               # Level 0: Basic utilities
│   ├── types.h         # MatrixView, ErrorCode, enums
│   ├── exception.h     # SonareException
│   └── math_utils.h    # mean, variance, argmax, etc.
│
├── core/               # Level 1-3: Core DSP
│   ├── convert.h       # Hz/Mel/MIDI conversion
│   ├── window.h        # Hann, Hamming, Blackman
│   ├── fft.h           # Transform seam (PFFFT / KissFFT backends)
│   ├── spectrum.h      # STFT/iSTFT
│   ├── audio.h         # Audio buffer
│   ├── audio_io.h      # WAV/MP3 loading, optional FFmpeg-backed formats
│   └── resample.h      # r8brain resampling
│
├── filters/            # Level 4: Filterbanks
│   ├── mel.h           # Mel filterbank
│   ├── chroma.h        # Chroma filterbank
│   ├── dct.h           # DCT for MFCC
│   └── iir.h           # IIR filters
│
├── feature/            # Level 4: Feature extraction
│   ├── mel_spectrogram.h
│   ├── chroma.h
│   ├── cqt.h
│   ├── vqt.h
│   ├── inverse.h
│   ├── spectral.h
│   ├── onset.h
│   └── pitch.h
│
├── effects/            # Level 5: Audio effects
│   ├── hpss.h
│   ├── phase_vocoder.h
│   ├── time_stretch.h
│   ├── pitch_shift.h
│   ├── normalize.h
│   ├── preemphasis.h
│   ├── silence.h
│   ├── decompose.h
│   ├── remix.h
│   ├── delay/ modulation/ reverb/
│   ├── acoustic/       # room_morph
│   └── common/
│
├── acoustic/           # Geometric room acoustics
│   ├── room_model.* room_types.* material.*
│   ├── image_source.*  # early reflections
│   ├── late_reverb.*   # deterministic late tail
│   └── rir_synthesizer.*
│
├── analysis/           # Level 6: Music analysis
│   ├── music_analyzer.h
│   ├── bpm_analyzer.h
│   ├── key_analyzer.h
│   ├── beat_analyzer.h
│   ├── downbeat_analyzer.h
│   ├── meter_analyzer.h
│   ├── chord_analyzer.h
│   ├── section_analyzer.h
│   ├── boundary_detector.h
│   ├── melody_analyzer.h
│   ├── rhythm_analyzer.h
│   ├── timbre_analyzer.h
│   ├── dynamics_analyzer.h
│   ├── acoustic_analyzer.h
│   ├── room_estimator.h
│   └── ...
│
├── streaming/          # Level 6: Real-time streaming
│   ├── stream_analyzer.h   # Main streaming analyzer
│   ├── stream_config.h     # Configuration options
│   └── stream_frame.h      # Frame and buffer types
│
├── mastering/          # Mastering engine
│   ├── api/            # Chain, 30 presets, named-processor and pair/stereo registries
│   ├── eq/ dynamics/ spectral/ stereo/ final/
│   ├── maximizer/ multiband/ saturation/ repair/ utility/
│   ├── match/ assistant/                 # Reference match + assistant/profile
│   └── common/        # Shared biquad/loudness helpers
│
├── mixing/             # Mixing engine
│   ├── channel_strip.* # Strip: trim/insert/pan/width/sends
│   ├── bus.* sends.* vca_group.* panner.*
│   ├── downmix.*       # ITU-R BS.775 channel fold
│   └── api/            # Scene JSON + scene presets
│
├── midi/               # MIDI I-O and the built-in instruments
│   ├── synth/          # NativeSynth voices + SoundFont player
│   │   ├── native_synth.*      # 17 engine modes: 11 physical models + 6 others
│   │   ├── ks_/piano_/pipe_organ_/bowed_string_/reed_/brass_/flute_/... voice.*
│   │   ├── sf2_player.* sf2_file.* sf2_voice.*   # SoundFont (SF2) playback
│   │   └── synth_presets.* gm_fallback_map.* gs_layer.* gs_effects.*
│   ├── sequencer.* smf.* smf2.* ump.*   # Sequencer, SMF and MIDI 2.0 (UMP)
│   └── program_map.* cc_map.* midi_fx.* routing.*
│
├── playback/           # Playback renderer: speaker layouts, binaural HRTF, loudness
├── arrangement/        # Non-destructive edit model, edit compiler, edit history
├── engine/             # Realtime engine: transport, clips, instrument rack, track mixer, metronome, telemetry
├── automation/         # Automation lanes + curve shapes
├── metering/           # LUFS, true-peak, phase scope/goniometer
├── mir/                # Warp, grid snap, key context, tempo estimator bridge
├── graph/  rt/  transport/   # DSP graph, RT-safe primitives, transport
├── editing/            # Pitch editor, voice changer, note stretch
├── serialize/          # Project (de)serialization
├── host/               # Audio-device / MIDI-I-O / plugin-host backends (native only)
│
├── quick.h             # Simple function API
├── sonare.h            # Unified include header
├── c_api/              # C API implementation (public headers in include/sonare/sonare_c*.h)
└── wasm/
    └── bindings.cpp    # Embind bindings

Data Flow

Audio Analysis Pipeline

This diagram shows the main feature relationships, not a single shared STFT instance. Within a MusicAnalyzer, the ordinary chromagram uses a cached spectrogram, and a cached mel representation feeds the onset strength used by BPM and beat tracking. Harmonic/CQT chroma for key and chord analysis can compute additional transforms. MusicAnalyzer.analyze() builds an AnalysisResult, including optional fields enabled by its configuration, and reuses intermediates already available on that analyzer.

Audio Analysis Pipeline

This is a feature map: some intermediate results are cached within one MusicAnalyzer, while other analyses use their own transforms.

Audio Effects Pipeline

HPSS — harmonic/percussive source separation, which splits a signal into its sustained-tone and transient parts — and the phase vocoder both operate on complex STFT data and reconstruct audio with an inverse STFT. The diagram shows their shared type of processing, not one transform instance or a guarantee that their window and hop settings match. timeStretch can use the native spectral backend or a phase-vocoder fallback; pitch shift combines time stretch with resampling.

Audio Effects Pipeline

What is a phase vocoder?

A phase vocoder time-stretches audio by analysing it with one hop size and resynthesising with a hop r times larger or smaller. Each bin's phase is advanced by its measured frequency times the new hop, so every sinusoid stays continuous across frames instead of jumping. See Time Stretch and Pitch Shift for the picture. Combined with resampling, it can also shift pitch. It approximately preserves pitch during a stretch, but transients may smear and extreme ratios can sound artificial. libsonare offers a phase-vocoder backend for timeStretch; other editing paths may use a different spectral backend.

Streaming Pipeline

The streaming pipeline processes audio in real time, maintaining overlap state between chunks. Streams above 44.1 kHz are first resampled to 44.1 kHz by a stateful resampler; mel, chroma and onset read the power spectrum, while the spectral features read the magnitude. Once a full frame's features land in the ring buffer, quantization is an opt-in trade: the default keeps full Float32 precision, while 8-bit/16-bit packing shrinks the buffer for transfer at the cost of precision.

Streaming Pipeline

Progressive Estimation

As more audio streams in, the pipeline accumulates chroma and onset data, so its BPM/key estimates have more evidence to work from. Estimates are refreshed periodically (default: BPM every 10s, key every 5s) and grow more confident the longer the stream runs. The key comes from the stream's own 186 ms chroma, so it trails the frame outputs by up to about a quarter second.

Key Design Decisions

Lazy Initialization

MusicAnalyzer initializes sub-analyzers on demand. Each intermediate (STFT, chroma, onset envelope, etc.) is computed the first time it's needed and reused afterwards.

// BPM only (computes onset envelope)
float bpm = analyzer.bpm();

// Key detection triggers chroma computation
Key key = analyzer.key();

// All-in-one analysis fills in what's left
AnalysisResult result = analyzer.analyze();

Why this matters

Asking just for the key does not force chord recognition or section detection to run. Calling analyze() reuses intermediates already computed on that MusicAnalyzer; distinct feature paths can still perform additional transforms.

Zero-Copy Audio Slicing

Audio uses shared_ptr with offset/size for zero-copy slicing:

auto full = Audio::from_file("song.mp3");

// Both share same underlying buffer
auto intro = full.slice(0, 30);     // 0-30 sec
auto chorus = full.slice(60, 90);   // 60-90 sec

WASM Compatibility

"Decoded samples" means raw audio amplitude values (a Float32Array), not the bytes of a .mp3 or .wav file — decoding is the step that turns the compressed file into those values. Most WASM calls expect samples that are already decoded.

The npm/WebAssembly package exposes mostly sample-based APIs. Most calls expect decoded mono Float32Array samples. For encoded bytes, Audio.fromMemory(...) decodes WAV/MP3 in memory, while Audio.fromMemoryWithBrowserFallback(...) can switch to the Web Audio API or another browser codec path before calling the same sample-based methods. Both fold a multichannel file to mono with the ITU-R BS.775 rule; decodeChannels(...) keeps every channel and downmix(...) applies the same rule on demand.

WASM builds avoid native file I/O and FFmpeg-backed decoding. Runtime behavior is single-threaded.

librosa Compatibility

Many DSP defaults intentionally mirror common librosa values, but libsonare is not a drop-in replacement. In particular, libsonare usually requires the caller to provide the sample rate; it does not implicitly resample to 22050 Hz the way librosa.load() does by default.

Parameter	Default
sample_rate	User-provided
n_fft	2048
hop_length	512
n_mels	128
fmin	0
fmax	sr/2

Third-Party Libraries

Library	Purpose	License
KissFFT	FFT	BSD-3-Clause
PFFFT	SIMD FFT, native builds	BSD-3-Clause
Eigen3	Matrix ops, build-time only	MPL-2.0
dr_libs	WAV decode	Public Domain
minimp3	MP3 decode	CC0-1.0
FFmpeg	Optional extended file decoding	LGPL/GPL depending on linked build
r8brain	Resampling	MIT
SADIE II HRTF subset	Default binaural HRTF set of the playback renderer (data, not code)	Apache-2.0

Two FFT backends behind one header

core/fft.h is the seam, not a binding to one library. In a native build PFFFT serves every transform length it can factor and KissFFT serves the rest, so the accepted n_fft range is the same either way — only the speed differs. Configure with -DSONARE_USE_PFFFT=OFF for a KissFFT-only native build.

The WebAssembly build is KissFFT-only on purpose: carrying both transforms costs the analysis-only module more than its size budget allows, for a gain that does not survive the move off native SIMD.

Eigen is not a consumer dependency

Eigen is used inside the core's own translation units and no installed header includes it, so it is a build-time dependency rather than something a consumer links or includes. Building against an installed libsonare needs a C++17 compiler and nothing else.

WASM Compilation

Output: ~7,808 KB WASM (~2,656 KB gzipped) plus the JS binding code;
        ~8,500 KB with the JS loader and API bundle (~2,792 KB gzipped); see src/wasm/meta.json
Build:  Emscripten with Embind
Flags:  -sWASM=1 -sMODULARIZE=1 -sEXPORT_ES6=1

The full mastering + mixing + analysis API set accounts for the bundle size. The published @libraz/libsonare/analysis build is substantially smaller because it leaves out the mastering, mixing, realtime, and project surfaces.

Non-goals

libsonare is the headless engine, not an application. The boundary is deliberate, and everything below is outside it:

Not included	Why
UI or DAW workflow	The engine returns data structures. Arrangement, editing and playback UI belong to the application on top of it.
Third-party plugin hosting (VST/CLAP)	Hosting means adopting another plugin ABI, its threading model and its license terms.
Cross-platform real-time I/O abstraction	Every host already has one, and wrapping them all would pull in per-platform dependencies for no gain.
Bundled audio samples or instrument sample libraries	Audio corpora carry their own licences, which would attach to the library. The one data set shipped is the small HRTF set listed above.
Note-level composition	Writing melodies and harmony is a separate concern from analysis and rendering; it belongs to the separate @libraz/libcantus.
Deep-learning models	Weights are large, hard to license cleanly, and would make the build depend on a runtime.
Windows	Use Linux, macOS, WebAssembly, or WSL2.
Owning the audio callback	Callers own both the callback and the UI. The experimental macOS backends are the one opt-in, unpublished exception to that boundary.

Together these keep the library dependency-free and limited to compatible permissive licences (FFmpeg is optional and off in WASM), which is what makes it safe to link into something else.