Live MIDI Input & Web MIDI

Live MIDI input turns the realtime engine into an instrument you can play. A note pressed on a USB keyboard becomes a note-on event, the engine routes it to a bound synthesizer, and you hear it on the next audio block — no file, no offline render.

libsonare's RealtimeEngine accepts live MIDI through the same realtime-safe entry points that drive transport, clips, and automation. In the browser, a small Web MIDI bridge (bindWebMidi) wires the platform's MIDI ports straight into that engine for you.

There are two common starting points:

If you have	Start with
An on-screen keyboard or sequencer step written in your app	immediate pushMidiNoteOn / pushMidiNoteOff calls
A USB keyboard or external controller	bindWebMidi, which uses the live input-source calls for you

MIDI destination

A MIDI destination is not a speaker output. It is an internal instrument slot. MIDI events go to a destination id such as 0; the instrument bound to that id decides what sound is produced.

Where this sits

This page is about playing the engine from a controller. To bind the instruments those notes reach, see Native Synth (patch-driven synthesizer) and SoundFont Player (GS/GM .sf2 playback). To record what you play into a timeline, see Recording and Takes. Microphone audio input is a separate path — see the note at the end. New to MIDI? Start with MIDI Basics.

The live MIDI path

The browser receives MIDI bytes, bindWebMidi converts them into engine events, and the destination's instrument produces audio during process(...). Each arrow below is one hop on that path — from raw key press to sample data — and the loop label marks the parts that repeat on every note.

USB keyboard to audio output

Everything left of the engine happens once per event; everything right of it happens once per audio block.

If you hear silence, check the path in this order: browser permission, bindWebMidi input list, destination id, bound instrument, then the AudioWorklet/output wiring.

The MIDI destination model

The engine does not play notes directly — it routes them to MIDI destinations, and each destination has an instrument bound to it. A destination is identified by a small integer id (default 0). You bind an instrument once, then every live event or scheduled clip aimed at that id renders through it.

Three instrument kinds can sit on a destination:

Bind with	Instrument	See
setBuiltinInstrument(config, destinationId)	The built-in waveform synth (the data-free floor)	—
setSynthInstrument(patch, destinationId)	The patch-driven NativeSynth	Native Synth
setSf2Instrument(config, destinationId)	A GS-compatible SoundFont player	SoundFont Player

MIDI 2.0 and MPE expression

MPE gives each sounding note its own channel inside a zone, so bend and pressure can shape one note of a chord; MIDI Basics explains zones, manager and member channels, and UMP groups.

The built-in waveform synth (setBuiltinInstrument), NativeSynth (setSynthInstrument), and SoundFont player (setSf2Instrument) receive MIDI 2.0 channel-voice UMPs at full value width. Note velocity, controllers, pressure, and pitch bend retain the precision supplied by the MIDI 2.0 message.

The built-in synth and NativeSynth apply channel pressure, polyphonic key pressure, and per-note pitch bend. The SoundFont player applies full-width note velocity, channel pressure, pitch bend, and MIDI 2.0 per-note pitch, but intentionally ignores polyphonic key pressure. MPE-style member channels can therefore carry channel expression to each player, while poly-pressure expression is unavailable on SoundFont.

A zone-wide command sent on an MPE manager channel, such as CC64 sustain, reaches every member channel of the zone on the built-in synth.

NativeSynth and the SoundFont player use RPN 0 for bend range, and the built-in synth handles its MPE zone configuration. Generic high-resolution RPN/NRPN and 14-bit CC parameter mappings live one level up, in the engine's bindMidiCcBinding(...) layer described below. See Native Synth for NativeSynth's bend and expression details.

import { init, RealtimeEngine } from '@libraz/libsonare';

await init();

const engine = new RealtimeEngine(48000, /* maxBlockSize */ 128);

// Destination 0 → a NativeSynth preset (see synthPresetNames()).
engine.setSynthInstrument('saw-lead', 0);

// You can run several destinations at once, each with its own instrument.
engine.setSf2Instrument({ gain: 1 }, 1);

Use clearMidiInstrument(destinationId) to unbind one, and midiInstrumentCount() to see how many are live. Multiple destinations let one engine host a layered rig: a lead synth on 0, drums on 1, and so on.

Rebinding a destination while the transport rolls fades the outgoing instrument over 10 ms and starts the new one from its default channel state. Resend any program or bank with pushMidiProgram after the change; see Realtime Engine.

Queueing live events

Live events are queued, not played synchronously. Each call hands the engine a sample position at which the event should fire; the next process(...) block consumes everything due in that block. That is what makes timing tight: the event lands at an exact frame, not "whenever the message arrived".

There are two queueing paths, and you should pick one per destination. Rule of thumb: use the immediate commands when your own code generates the events (a sequencer step, an on-screen keyboard); use the input source when events arrive from outside with their own timestamps (a hardware keyboard via bindWebMidi), because that lane carries the per-port timestamp the Web MIDI bridge needs.

Call	Targets	Timing argument	Notes
pushMidiNoteOn / pushMidiNoteOff / pushMidiCc	One destination	renderFrame?	Omit it to fire as soon as possible
pushMidiProgram	One destination	renderFrame?	One MIDI 2.0 Program Change; see below
pushMidiSysex	One destination	renderFrame?	Complete frame, 0xF0 to 0xF7, 1..512 bytes
pushMidiUmp	One destination	renderFrame?	1 to 4 words; MT 0x4 keeps full width, MT 0x3 / 0x5 are rejected
pushMidiPanic	Every destination	renderFrame?	Releases every sounding note
pushMidiInput* (NoteOn, NoteOff, Cc, PitchBend, ChannelPressure, PolyPressure, Program, Ump)	The input source	offsetInBlockSamples	Needs setMidiInputSource(destinationId); the lane the Web MIDI bridge feeds

• Immediate engine commands — pushMidiNoteOn / pushMidiNoteOff / pushMidiCc each take a destinationId and an optional renderFrame; omit it to fire as soon as possible. pushMidiPanic(renderFrame?) takes only the optional renderFrame and releases every sounding note on all destinations at once.
• Refusals — every live push refuses a value outside its MIDI range with an error naming the argument and the range, such as channel must be in [0, 15] or bend14 must be in [0, 16383], on destination pushes and input-source pushes alike: RangeError from the package classes on Node and WASM (the raw embind engine refuses with a CppException), SonareValueError in Python, InvalidParameter in C, each prefixed with the calling function's name. A negative renderFrame is refused the same way; "not given" is absence, never -1.
• The engine-owned live input source — setMidiInputSource(destinationId) opens a dedicated input lane, then pushMidiInputNoteOn / pushMidiInputNoteOff / pushMidiInputCc send events with a offsetInBlockSamples timestamp. pushMidiInputPitchBend, pushMidiInputChannelPressure, pushMidiInputPolyPressure, pushMidiInputProgram, and pushMidiInputUmp add the expression, program-change and raw-packet paths. Pushing while the input source is disabled fails with InvalidState, and a full input queue with OutOfMemory; a WASM build without the input source throws NotSupported.
• Live SysEx — pushMidiSysex(destinationId, data, renderFrame?) queues a full SysEx (System Exclusive, a free-form manufacturer-specific message) frame to a destination; data is the complete message including the leading 0xF0 and trailing 0xF7 (1..512 bytes), and renderFrame follows the same omit-for-immediate convention as the other pushMidi* calls. Its primary use is delivering a GS/GM reset or a GS insertion-effect (EFX) selection to a live SF2-bound destination without stopping playback — see SoundFont Player for what that SysEx selects.
• Raw UMP messages — pushMidiUmp(destinationId, words, renderFrame?) accepts an array or Uint32Array of 1–4 32-bit words, most significant first. The length must match the message type in the first word. MIDI 2.0 channel-voice messages (MT 0x4) keep their full value width; SysEx7/data messages (MT 0x3/0x5) are rejected, so use pushMidiSysex for those. The SonareEngine worklet facade accepts the same complete packet through pushMidiUmp(trackId, words, renderFrame?) (a legacy one-word number is also accepted), and its pushMidiInputUmp(words, offsetInBlockSamples = 0) covers the engine-owned input source.

Every binding exposes the same SysEx call; only the naming convention changes. data is the complete frame (leading 0xF0, trailing 0xF7, 1..512 bytes) and the last argument is the same optional render frame as the other pushMidi* calls (omitted, or None in Python, for immediate). Only the C ABI spells "immediate" as -1.

The destination instrument prepares a live SysEx payload on the control thread before it is queued. If preparation is refused, the call reports a refused argument (RangeError; InvalidParameter in C); retrying the same payload cannot fix that refusal. setMidiClips prepares every scheduled SysEx before publishing a replacement, so one refusal leaves the previously accepted schedule in place. Rebinding a destination also stages preparation for scheduled and already queued SysEx; a refusal (InvalidState) leaves the previous instrument binding and pending operations unchanged. A later prepare() rechecks scheduled payloads; if the bound instrument refuses one, the schedule is cleared and InvalidState is reported.

Browser

engine.pushMidiSysex(/* destinationId */ 0, gsResetOrEfxBytes);

Node

// Same surface as the browser; data is a Buffer or Uint8Array.
engine.pushMidiSysex(0, gsResetOrEfxBytes);

Python

# data is bytes / bytearray / memoryview; render_frame=None (the default) is immediate.
engine.push_midi_sysex(0, gs_reset_or_efx_bytes)

C ABI

/* size must be 1..512; the C ABI keeps render_frame -1 = immediate. */
sonare_engine_push_midi_sysex(engine, /* destination_id */ 0,
                              data, size, /* render_frame */ -1);

// Immediate path: fire a note at the start of the next block.
engine.pushMidiNoteOn(/* destinationId */ 0, /* group */ 0, /* channel */ 0, /* note */ 60, /* velocity */ 100);
engine.pushMidiCc(0, 0, 0, /* controller */ 1, /* value */ 64);
engine.pushMidiNoteOff(0, 0, 0, 60, 0);

// Scheduled: the same call with a block-relative render frame.
engine.pushMidiNoteOn(0, 0, 0, 64, 100, /* renderFrame */ 64);

// Input-source path (what bindWebMidi uses under the hood):
engine.setMidiInputSource(0);
engine.pushMidiInputNoteOn(/* group */ 0, /* channel */ 0, 60, 100, /* offsetInBlockSamples */ 0);
engine.pushMidiInputCc(0, 0, 1, 64, 0);
engine.pushMidiInputNoteOff(0, 0, 60, 0, 0);
// engine.midiInputPendingCount()  -> events waiting for the next process() block

group and channel are MIDI nibbles (0..15); note, velocity, controller, and value are 7-bit (0..127). A note-on with velocity 0 is treated as a note-off, exactly as the MIDI spec requires.

Program change and the receivers that follow it

pushMidiProgram(destinationId, group, channel, program, bankValid?, bankMsb?, bankLsb?, renderFrame?) and pushMidiInputProgram(group, channel, program, bankValid?, bankMsb?, bankLsb?, offsetInBlockSamples?) (Python push_midi_program, push_midi_input_program; C sonare_engine_push_midi_program, sonare_engine_push_midi_input_program) queue one MIDI 2.0 Program Change, timed like a note or CC. The bank is optional: it is applied only when bankValid is true, and a non-zero bankMsb / bankLsb with bankValid false is refused. The SonareEngine worklet facade takes pushMidiProgram(trackId, group, channel, program, bankValid?, bankMsb?, bankLsb?, renderFrame?).

The message goes through the receiver's own program gates, so only some receivers act on it:

Receiver	Follows a program change?
SoundFont player	Yes, unless the part's GS receive switches for program change or bank select are off
NativeSynth bound with useGmPrograms: true (Python use_gm_programs)	Yes; each channel's voice is resolved from the bank and program
NativeSynth playing the GM kit (the drum-kit preset, or a percussion patch with gmKit on)	Yes; the program picks the GS drum set on every channel
Built-in synth, any other NativeSynth patch, a part whose GS program-change receive switch is off, an MPE member channel	Accepted and ignored

Binding MIDI CCs to engine parameters

A CC can do double duty: reach the instrument and drive an engine automation parameter. bindMidiCc(channel, controller, paramId, options) maps a controller's 7-bit value into [minValue, maxValue] for a registered parameter, while the CC still flows to the destination instrument.

// Register the parameter the engine should drive, then map mod wheel (CC1) to it.
engine.addParameter({ id: 42, name: 'cutoff', minValue: 0, maxValue: 1, defaultValue: 0.5 });
engine.bindMidiCc(/* channel */ 0, /* controller */ 1, /* paramId */ 42, { minValue: 0, maxValue: 1 });

// engine.midiCcBindingCount()  -> 1
// engine.clearMidiCcBindings() -> remove all mappings

For high-resolution controllers, call bindMidiCcBinding(...) with a descriptor. Its kind selects the controller format and which fields it reads:

kind	Format	Fields used
0	7-bit CC	ccNumber
1	Paired 14-bit CC	ccNumber, ccLsbNumber
2	RPN	selectorMsb, selectorLsb
3	NRPN	selectorMsb, selectorLsb

channel: 255 matches every MIDI channel.

engine.bindMidiCcBinding({
  kind: 3, // NRPN
  channel: 0,
  ccNumber: 99,
  selectorMsb: 1,
  selectorLsb: 2,
  paramId: 42,
  minValue: 0,
  maxValue: 1,
});

addParameter also accepts unit (a display string), rtSafe, and defaultCurve. The one that matters here is rtSafe (default true): it declares whether the audio thread may change the parameter mid-playback. Keep it true for anything you intend to CC-bind and move while notes sound. Register it with rtSafe: false and the engine rejects every live write to it outright — from automation and from a bound CC, and from a direct setParameter(...) call — regardless of whether the transport is playing or stopped.

CC "learn" workflows

For an offline "wiggle a knob, capture which CC moved" flow, Project.midiCcLearn works on captured ProjectMidiEvent data rather than the live engine. See Learn a CC and turn it into automation.

Mapping a device gesture to an expression axis

A controller profile says what a gesture means to the instrument on a destination: CC2 as breath force, channel pressure as loudness, a pitch bend as a pitch offset in cents. This is separate from bindMidiCc, which drives an engine parameter; a profile drives the instrument's own expression axes.

• controllerProfileNames() lists the presets: 'gm', 'breath', 'breath-aftertouch', 'mpe'.
• engine.setControllerProfile(destinationId, name) replaces the destination's profile with a preset and clears every channel's accumulated axis values. An unknown name, a destination with no instrument, or an instrument with no profile throws.
• engine.bindController(destinationId, binding) adds one ControllerBinding on top: input ('control-change', 'channel-pressure', 'poly-pressure', 'pitch-bend', 'velocity'), index (the CC number, used only by 'control-change'), axis ('excitation', 'position', 'brightness', 'morph', 'loudness', 'pitch-cents', 'vibrato-depth'; 'none' is refused), and optional lo, hi and curve. lo > hi inverts the gesture, and curve defaults to a linear 1.
• clearControllerBindings(destinationId), controllerBindingCount(destinationId) and setControllerVelocityMeaningful(destinationId, meaningful) complete the set. Set the last to false when the device sends a constant velocity, so the bound axes carry the dynamics alone.
• setControllerNoteTracking(destinationId, dimension, tracking) chooses which note a value addressed to a whole channel reaches when several sound on it: dimension is one of MPE_DIMENSIONS ('bend', 'pressure', 'timbre') and tracking one of NOTE_TRACKINGS ('last', 'lowest', 'highest', 'all'). It matters only inside an MPE zone while a channel holds more than one note.

import { controllerProfileNames } from '@libraz/libsonare';

controllerProfileNames();                         // ['gm', 'breath', 'breath-aftertouch', 'mpe']
engine.setControllerProfile(0, 'breath');         // breath CC drives the excitation force
engine.bindController(0, { input: 'channel-pressure', axis: 'loudness', lo: 0.2, hi: 1 });
engine.controllerBindingCount(0);                 // 5

Python uses bind_controller(destination_id, input=..., index=0, axis=..., lo=0.0, hi=1.0, curve=1.0) with keyword-only arguments, set_controller_profile, clear_controller_bindings, controller_binding_count, set_controller_velocity_meaningful, set_controller_note_tracking, and the module-level controller_profile_names(). Which axes each instrument voice reads is listed in Physical Models.

Swapping MIDI FX without hanging notes

Each destination can carry one MIDI-FX insert — a non-destructive transform on the event stream (transpose, channel filter, velocity curve, …) configured from JSON.

Browser

// Transpose every incoming note up an octave.
engine.setMidiFx(/* destinationId */ 0, JSON.stringify({ transpose_semitones: 12 }));
engine.clearMidiFx(0);   // clears this destination only (the id defaults to 0 if omitted)

Node

engine.setMidiFx(0, JSON.stringify({ transpose_semitones: 12 }));
engine.clearMidiFx(0);   // destinationId defaults to 0

Python

engine.set_midi_fx(0, '{"transpose_semitones": 12}')
engine.clear_midi_fx(0)   # destination_id defaults to 0

C ABI

sonare_engine_set_midi_fx(engine, /* destination_id */ 0,
                          "{\"transpose_semitones\": 12}");
sonare_engine_clear_midi_fx(engine, /* destination_id */ 0);

The JSON schema is the one bakeMidiFx accepts: stages are keyed by their parameters, so include a stage's keys to enable it and omit them to skip it. Valid keys include transpose_semitones, velocity_scale / velocity_offset / velocity_gamma, quantize_ppq / quantize_strength, chord_intervals, and arpeggiator_intervals / arpeggiator_step_ppq / arpeggiator_gate_ppq. See Bake a MIDI-FX chain for the full key table.

setMidiFx replaces the insert in place without resetting the instrument's voices, but swapping onto a destination that already has FX releases its sounding notes: the audio thread flushes (all-notes-off) any destination whose FX configuration changed, on the next block. For keys held while you change FX:

• The very first setMidiFx call on a destination with no active FX flushes nothing, since there is no live configuration to replace yet.
• Every later swap releases the sounding notes automatically, so no follow-up panic is needed. If a swap must not audibly cut a held note, time it for a gap between notes rather than mid-sustain.

The call publishes an immutable configuration from the control thread and the audio thread adopts it at the next block boundary, so it may run while engine processing is active.

MIDI panic and stuck-note recovery

A stuck note is a note-on whose matching note-off never arrived — a yanked cable, a dropped Bluetooth packet, an FX swap that ate the off. The cure is a MIDI panic: an all-notes-off that releases every sounding voice.

engine.pushMidiPanic();   // immediate; or pass a renderFrame to schedule it

Panic is realtime-safe and cheap — wire it to a visible "panic" button in any instrument UI. The Web MIDI bridge does not auto-panic on disconnect, so if you handle hot-unplug yourself, send a panic when a port you were playing goes away.

Note-offs are not dropped under load: one that a full MIDI output refuses is retried every block, and a full MIDI-FX queue releases the note at once and withdraws its pending note-on. Scheduled MIDI lands on the same frame whatever block size the host runs, including MIDI-FX output, releases at a clip's end, and events after a loop wrap.

The browser Web MIDI bridge

A few wire-format terms

UMP (Universal MIDI Packet) is the MIDI 2.0 message format; the bridge accepts it as well as classic MIDI 1.0 bytes. SysEx (System Exclusive) is a free-form, manufacturer-specific message — used for things like a GS Reset — and browsers gate it behind a separate permission. RPN / NRPN ((non-)registered parameter numbers) address extra parameters via CC, for example RPN 0 sets the pitch-bend range.

In the browser, bindWebMidi(engine, options) does the plumbing: it requests MIDI access, enables the engine's live input source, attaches listeners to every matching input port, parses incoming bytes (including running status and one-data-byte messages), and queues them onto the engine with sample timestamps. A Web MIDI Uint32Array payload, or a payload whose first value is above 0xff, is read as a stream of UMP packets and split by each packet's message type. When the supplied engine has pushMidiInputUmp, each complete one-to-four-word channel-voice packet keeps its original MIDI 2.0 width; MT 0x3 and MT 0x5 data packets are left for a SysEx path. If the engine only supplies the convenience methods, MIDI 2.0 channel voice is reduced to the matching 7-bit note/CC/pressure or 14-bit pitch-bend method.

import { init, RealtimeEngine, isWebMidiAvailable, bindWebMidi } from '@libraz/libsonare';

await init();
if (!isWebMidiAvailable()) {
  // navigator.requestMIDIAccess is missing — fall back to an on-screen keyboard.
}

const engine = new RealtimeEngine(48000, 128);
engine.setSynthInstrument('saw-lead', 0);

const binding = await bindWebMidi(engine, {
  destinationId: 0,        // engine MIDI destination to play (default 0)
  group: 0,                // UMP group for MIDI 1.0 events (default 0)
  // inputIds: ['<port-id>'],  // restrict to specific ports; omit = all connected
  sysex: false,            // request SysEx-capable access (default false)
  software: true,          // request software ports where supported (default true)
  ccBindings: [
    { channel: 0, controller: 1, paramId: 42, options: { minValue: 0, maxValue: 1 } },
  ],
  // Optional: return a block-relative offset in samples for each event time.
  // Only the thread that renders can compute it; see "Why timestamp -> sample mapping matters".
  // timestampToOffsetInBlockSamples: (eventTimeMs) => offset,
  onInputsChanged: (inputs) => {
    // Called on hot-plug after the helper rebinds matching ports.
    console.log('MIDI inputs:', inputs.map((i) => `${i.name} (${i.state})`));
  },
});

// binding.inputs()  -> WebMidiInputInfo[] { id, name, manufacturer, state }
// binding.access    -> the underlying MIDIAccess object, if you need raw control

What each option does:

• destinationId / group — which engine destination receives the live events, and the UMP group stamped on MIDI 1.0 channel-voice events.
• inputIds — restrict binding to specific port ids (from binding.inputs()); omit or pass an empty array to bind every connected input.
• sysex / software — passed straight to navigator.requestMIDIAccess. SysEx access usually triggers a separate permission prompt; software requests software-synth ports where the platform offers them.
• ccBindings — bindMidiCc mappings applied before any port connects, so the very first knob move is already routed. Register the target parameters with addParameter(...) first.
• onInputsChanged — fires on hot-plug (MIDIAccess statechange) after the helper has rebound matching ports, with the fresh port list.

Why timestamp → sample mapping matters

The two clocks don't match. Web MIDI tags each message with a time in milliseconds (the page clock, DOMHighResTimeStamp), but the engine schedules events by sample frame. timestampToOffsetInBlockSamples(eventTimeMs) is the bridge between them: it converts a message time into the offsetInBlockSamples value the engine queues. That value is an offset, in samples, from the start of the render block that picks the event up, not a position on an absolute clock. An offset past the end of the block is clamped to the block's last frame. Measured with a 128-frame block, an offset of 40 starts the note at frame 41, while a page-clock time converted straight to samples (480000 after ten seconds) produces nothing until the block's final frame, so every event of a fast run lands on the same frame.

Why bother? Get the conversion right and tightly-timed passages (chords, fast runs) land on the exact frame you played them. Omit it, and every event is queued at sample 0 of the next block, so timing snaps to block boundaries: 2.7 ms steps at 128 frames and 48 kHz, 10.7 ms at 512 frames. That is fine for casual playing and audible on a fast run.

Only the rendering thread can compute the offset. The offset is relative to the block the engine renders next, and only the thread that renders knows which block that is. With the engine in an AudioWorklet, the normal browser setup, the main thread does not: a posted message takes time to arrive, and by then the block the main thread had in mind has been rendered. Its offset then points past the block the event actually lands in, gets clamped to that block's last frame, and the events cluster as described above. So do not subtract a block start on the main thread. Instead, the main thread converts the Web MIDI timestamp into a context time with AudioContext.getOutputTimestamp() (it returns the same instant on both clocks, contextTime and performanceTime), adds the output latency and a small fixed lookahead so the target lies ahead of the render head, and posts it with the event. The worklet holds the event until the block that contains that time and computes offsetInBlockSamples from its own currentTime and sampleRate. The fixed lookahead trades a constant added latency for jitter-free timing. The recipe below does exactly this.

Web MIDI timestamp to block offset

The offset is computed on the thread that knows which block comes next.

Lifecycle

bindWebMidi returns a WebMidiBinding. When you are done, call binding.close(): it removes the statechange listener, detaches every port listener, and calls engine.clearMidiInputSource(). It does not destroy the engine; release that separately with engine.destroy().

Browser support

Web MIDI support is uneven, so check at runtime with isWebMidiAvailable() and degrade gracefully:

• Chrome and Edge (desktop) — full Web MIDI, including hot-plug and SysEx (behind a permission prompt). The primary target.
• Firefox — has shipped Web MIDI; SysEx and add-on requirements have varied over time, so feature-detect rather than assume.
• Safari — historically did not expose navigator.requestMIDIAccess; support has been changing, so do not assume it is present. Always gate on isWebMidiAvailable() and offer an on-screen keyboard as the alternate input path.

Recipe: a USB keyboard plays a synth in the browser

A runnable path from "keyboard plugged in" to "sound out of the speakers". It takes two files because the work spans two threads:

• Audio thread — an AudioWorklet processor hosts the engine and renders a block every render quantum. The engine must live here: on the main thread, process(...) is not called by the audio callback, so it would stay silent.
• Main thread — owns the Web MIDI access (bindWebMidi) and forwards every event to the worklet through the node's port.

Why a stand-in object satisfies bindWebMidi

bindWebMidi only calls the live-input part of the engine it is given: setMidiInputSource, bindMidiCc, the three required pushMidiInput* methods, and clearMidiInputSource on close. It also uses optional pitch-bend, channel-pressure, poly-pressure, and raw-UMP methods when they exist. Any object implementing the required methods can stand in for the engine — so a small forwarder that posts each event over the worklet port carries the binding across the thread boundary. Give the forwarder pushMidiInputUmp to preserve MIDI 2.0 packet width; omit it only when the 7-bit/14-bit convenience fallback is intended.

Audio thread: the worklet hosts the engine

An AudioWorkletGlobalScope forbids dynamic import(), which rules out the high-level package entry point (its init() imports the WASM module dynamically). Statically import the Emscripten factory sonare.js instead and call the lower-level engine it exposes. A few things to know about that lower-level entry point:

• The worklet cannot fetch the .wasm bytes either — fetch them on the main thread and hand them in through processorOptions.
• The raw embind RealtimeEngine constructor takes four arguments — (sampleRate, maxBlockSize, commandCapacity, telemetryCapacity). The high-level RealtimeEngine class defaults the last two to 1024, which is why the two-arg call earlier on this page works; the embind class has no default parameters, so the worklet must pass them explicitly. commandCapacity is the ring-buffer capacity for queued MIDI/automation commands, and telemetryCapacity is the capacity for telemetry/meter readback.
• Some argument orders differ from the high-level JS package, notably setSynthInstrument(destinationId, patch) and setSf2Instrument(destinationId, config): the destination id comes first on the raw engine.

// synth-worklet.js — load with context.audioWorklet.addModule(...).
// Copy sonare.js and sonare.wasm from the @libraz/libsonare package into your
// static assets; the import specifier must be a URL the worklet can resolve.
import createModule from '/wasm/sonare.js';

const BLOCK = 128; // render-quantum size the engine scratch is prepared for

// message type -> [engine method, ...message fields in argument order]
const LIVE = {
  noteOn: ['pushMidiInputNoteOn', 'group', 'channel', 'note', 'velocity'],
  noteOff: ['pushMidiInputNoteOff', 'group', 'channel', 'note', 'velocity'],
  cc: ['pushMidiInputCc', 'group', 'channel', 'controller', 'value'],
  pitchBend: ['pushMidiInputPitchBend', 'group', 'channel', 'bend14'],
  channelPressure: ['pushMidiInputChannelPressure', 'group', 'channel', 'pressure'],
  polyPressure: ['pushMidiInputPolyPressure', 'group', 'channel', 'note', 'pressure'],
};

class KeyboardSynthProcessor extends AudioWorkletProcessor {
  constructor(options) {
    super();
    this.engine = null;
    this.channels = [];
    this.pending = []; // timed events waiting for their block
    this.port.onmessage = (event) => this.onMessage(event.data);

    createModule({
      wasmBinary: options.processorOptions.wasmBinary,
      locateFile: () => 'sonare.wasm', // never hit the network from the worklet
    })
      .then((mod) => {
        const engine = new mod.RealtimeEngine(sampleRate, BLOCK, 1024, 1024);
        engine.setSynthInstrument(0, 'saw-lead'); // native order: (destinationId, patch)
        engine.setMidiInputSource(0);
        // Zero-copy render path: fill the prepared scratch, call processPrepared.
        engine.prepareChannels(2, BLOCK);
        this.channels = [engine.getChannelBuffer(0, BLOCK), engine.getChannelBuffer(1, BLOCK)];
        this.engine = engine;
        this.port.postMessage({ type: 'ready' });
      })
      .catch((err) => this.port.postMessage({ type: 'error', error: String(err) }));
  }

  onMessage(msg) {
    if (!this.engine) return;
    if (msg.type === 'panic') {
      this.engine.pushMidiPanic(); // omitted frame = immediate
      return;
    }
    // Hold each event until the block it belongs to is the one being rendered.
    this.pending.push(msg);
    this.pending.sort((a, b) => a.time - b.time);
  }

  // `offset` is in samples from the start of the block about to be rendered.
  dispatch(msg, offset) {
    if (msg.type === 'ump') return this.engine.pushMidiInputUmp(msg.words, offset);
    const [method, ...names] = LIVE[msg.type];
    this.engine[method].call(this.engine, ...names.map((name) => msg[name]), offset);
  }

  process(_inputs, outputs) {
    const output = outputs[0];
    if (!output?.length) return true;
    if (!this.engine) {
      for (const channel of output) channel.fill(0);
      return true;
    }
    // Re-acquire the heap views if WASM memory growth detached them.
    if (this.channels[0].byteLength === 0) {
      this.channels = [this.engine.getChannelBuffer(0, BLOCK), this.engine.getChannelBuffer(1, BLOCK)];
    }
    const frames = Math.min(output[0].length, BLOCK);
    // currentTime is the context time of this block's first frame.
    const blockEnd = currentTime + frames / sampleRate;
    while (this.pending.length && this.pending[0].time < blockEnd) {
      const msg = this.pending.shift();
      const offset = Math.round((msg.time - currentTime) * sampleRate);
      this.dispatch(msg, Math.min(Math.max(offset, 0), frames - 1)); // late events play at offset 0
    }
    // The synth is a generator — clear the input scratch before rendering.
    for (const channel of this.channels) channel.fill(0, 0, frames);
    this.engine.processPrepared(frames);
    for (let ch = 0; ch < output.length; ch++) {
      output[ch].set((this.channels[ch] ?? this.channels[0]).subarray(0, frames));
    }
    return true;
  }
}

registerProcessor('keyboard-synth', KeyboardSynthProcessor);

Main thread: Web MIDI feeds the worklet

No init() here — in this architecture the WASM runs only on the audio thread. The main thread boots the worklet, then hands bindWebMidi a forwarder whose live-input methods post to the port.

import { bindWebMidi, isWebMidiAvailable, type RealtimeEngine } from '@libraz/libsonare';

async function startKeyboardSynth() {
  if (!isWebMidiAvailable()) {
    throw new Error('Web MIDI not available — use an on-screen keyboard instead.');
  }

  // --- Boot the worklet (call this from a user gesture so the context runs) ---
  const context = new AudioContext({ latencyHint: 'interactive' });
  await context.audioWorklet.addModule('/synth-worklet.js');
  const wasmBinary = await (await fetch('/wasm/sonare.wasm')).arrayBuffer();
  const node = new AudioWorkletNode(context, 'keyboard-synth', {
    numberOfInputs: 0,
    numberOfOutputs: 1,
    outputChannelCount: [2],
    processorOptions: { wasmBinary },
  });
  node.connect(context.destination);
  await new Promise<void>((resolve, reject) => {
    node.port.onmessage = (event) =>
      event.data?.type === 'ready' ? resolve() : reject(new Error(event.data?.error));
  });

  // --- Page-clock time -> the context time at which the worklet should render the event ---
  const LOOKAHEAD_S = 0.03; // fixed extra delay; raise it if some notes still come out late
  const toContextTime = (eventMs: number) => {
    const { contextTime = 0, performanceTime = 0 } = context.getOutputTimestamp();
    // contextTime/performanceTime are the same instant on the two clocks.
    // The render head runs ahead of the speakers by about outputLatency.
    return contextTime + (eventMs - performanceTime) / 1000 + (context.outputLatency || 0) + LOOKAHEAD_S;
  };

  // --- The forwarder: bindWebMidi's engine calls, posted over the port ---
  const forwarder = {
    setMidiInputSource: () => {}, // the worklet already bound destination 0 at startup
    clearMidiInputSource: () => {
      node.port.postMessage({ type: 'panic' }); // release held notes on close
    },
    bindMidiCc: () => {}, // no CC-to-parameter bindings in this recipe
    pushMidiInputNoteOn: (group: number, channel: number, note: number, velocity: number, timeMs: number) =>
      node.port.postMessage({ type: 'noteOn', group, channel, note, velocity, time: toContextTime(timeMs) }),
    pushMidiInputNoteOff: (group: number, channel: number, note: number, velocity: number, timeMs: number) =>
      node.port.postMessage({ type: 'noteOff', group, channel, note, velocity, time: toContextTime(timeMs) }),
    pushMidiInputCc: (group: number, channel: number, controller: number, value: number, timeMs: number) =>
      node.port.postMessage({ type: 'cc', group, channel, controller, value, time: toContextTime(timeMs) }),
    pushMidiInputPitchBend: (group: number, channel: number, bend14: number, timeMs: number) =>
      node.port.postMessage({ type: 'pitchBend', group, channel, bend14, time: toContextTime(timeMs) }),
    pushMidiInputChannelPressure: (group: number, channel: number, pressure: number, timeMs: number) =>
      node.port.postMessage({ type: 'channelPressure', group, channel, pressure, time: toContextTime(timeMs) }),
    pushMidiInputPolyPressure: (group: number, channel: number, note: number, pressure: number, timeMs: number) =>
      node.port.postMessage({ type: 'polyPressure', group, channel, note, pressure, time: toContextTime(timeMs) }),
    pushMidiInputUmp: (words: Uint32Array | readonly number[], timeMs: number) =>
      node.port.postMessage({ type: 'ump', words, time: toContextTime(timeMs) }),
  };

  const binding = await bindWebMidi(forwarder as unknown as RealtimeEngine, {
    destinationId: 0,
    timestampToOffsetInBlockSamples: (eventTimeMs) => eventTimeMs, // pass the page-clock time through
    onInputsChanged: (inputs) =>
      console.log('keyboards:', inputs.map((i) => i.name).join(', ')),
  });

  // Pressing a key on the USB keyboard now sounds the 'saw-lead' patch.

  return {
    stop() {
      binding.close();      // detach MIDI ports; calls forwarder.clearMidiInputSource()
      node.disconnect();
      void context.close(); // tears down the worklet and the engine inside it
    },
  };
}

The binding is given timestampToOffsetInBlockSamples: (eventTimeMs) => eventTimeMs, an identity pass-through. The forwarder is not the engine, so it receives the page-clock timestamp in the slot where the engine would take an offset, and converts it to a context time itself. The worklet then turns that into a real offset per block. Without the option, the forwarder receives 0, the target time lands in the past, and every event plays at the start of the next block: tight enough for casual playing, but with the block-sized steps described above.

Browser gestures and cleanup

An AudioContext must be created/resumed from a user gesture (a click), and most browsers only prompt for MIDI access from a secure context. Pair bindWebMidi with binding.close(), and close the AudioContext when the page tears down — in this recipe that is what releases the worklet and the engine's native memory.

On other runtimes

The live-MIDI engine API is not browser-only. The Node native and Python bindings expose the same RealtimeEngine input methods — only the Web MIDI bridge itself is browser-specific (it depends on navigator.requestMIDIAccess). In Python the names follow the snake_case convention:

import libsonare as sonare

engine = sonare.RealtimeEngine(sample_rate=48000.0, max_block_size=128)
try:
    engine.set_synth_instrument("saw-lead", destination_id=0)
    engine.set_midi_input_source(0)
    engine.push_midi_input_note_on(0, 0, 60, 100, 0)   # group, channel, note, velocity, offset_in_block_samples
    engine.push_midi_input_cc(0, 0, 1, 64, 0)
    engine.push_midi_input_note_off(0, 0, 60, 0, 0)
    out = engine.process([[0.0] * 128, [0.0] * 128])   # non-zero once the note sounds
finally:
    engine.close()

To feed those engines from real hardware, read MIDI with a platform library (for example CoreMIDI or ALSA bindings) and call the same push_midi_input_* methods — the timestamp-to-sample mapping is yours to supply, just as timestampToOffsetInBlockSamples is in the browser.

Experimental native macOS backends

A C++ source build can opt into native macOS host backends behind the off-by-default BUILD_COREAUDIO, BUILD_COREMIDI, and BUILD_AU_HOST CMake options — a CoreAudio output, a CoreMIDI input/output, and an Audio Unit instrument host. They add no C-ABI API and ship in no published package (npm / PyPI / WASM), so they are a macOS-only, source-build opt-in that may still change.

A note on microphone input

Live MIDI is control input — it tells the engine what to play. Audio input (a microphone, an instrument through an interface) is a different path: bindMicrophoneInput(context, engine, options) routes captured audio into the engine for monitoring and recording. The two are independent and can run at once. See Recording and Takes.

Related

• Native Synth — the patch-driven instrument you bind to a destination
• SoundFont Player — GS/GM .sf2 playback on a destination
• Recording and Takes — capture what you play (and microphone audio input)
• Edit MIDI Clips — project event lists, validation, routing, CC learn, and MIDI-FX baking
• MIDI 2.0, UMP, and Clip Files — full-width UMP builders, live delivery, and file fidelity
• Project Bounce — render a MIDI performance offline
• Realtime and Streaming — the AudioWorklet engine bridge that renders audio output
• Realtime Engine · Realtime Safety