/projects/dj-agent — n°005 — AUDIO · ML — LIVE
DJ Agent
Q:Can we leverage agentic intelligence to curate an authentic vibe?
A personal DJ that turns my real library into a vibe database, learns my taste from how I annotate tracks, and plans beatmatched sets a deterministic critic must pass before any audio renders.
On the bench since
JUN 2026
Last update
JUN 2026
Status
LIVE
Stage
Bench — raw, still moving
- representations
- 512-d clap (track + section) · 384-d taste · bpm / camelot / lufs columns
- agents
- 2 llm (architect, selector) — everything else deterministic
- critic gate
- ≥70% harmonic · ≤6 bpm jump — set per genre · ≤4 lufs arc-rmse · 0 artist repeats — always
- state
- allin1 segments for real · genre-aware sets · 256 fast tests
Spotify DJ is a recommender in a costume — it legally can't touch audio, and it optimizes for everyone. This bench goes the other way: personal use, my actual files, my taste as the center of gravity, and real beatmatched renders that need the raw decodable audio streaming services can never give an algorithm.
Here's the whole idea in one run — watch the booth think:
The booth, thinking
7A
104
intro
8A
105
chorus
8B
106
chorus
8B
107
bridge
9B
108
chorus
3A
109
break
9A
110
break
10A
112
intro
10B
111
break
10B
110
bridge
11B
108
break
12B
105
outro
12 slots — each is a section of a track, not the whole file · ◆ splice: legal key move · clash
critic — downtempo profile
approval — human
Approve this set for rendering? [y/N] y
xml, m3u8 and set sheet are written on the yes — only the render costs compute: stretch dst÷src, equal-power crossfade, bass swapped at 180 Hz, overlap = half the outgoing section, quantized to a power-of-two bar count → one .wav
violet = the model imagining · outline = the model has no say — code or a human decides · pulse = the set, playing
The rules here are real and running: the Camelot legality test, the four thresholds, the arc interpolation and RMSE math execute on this page exactly as coded in the repo. Profile selection is a keyword table, not a model — detect() takes the brief's first keyword hit (“sunset” sits in downtempo's list) and --genre overrides it. Only the jump and harmonic gates are genre-variable: downtempo's equal the open defaults shown, hip-hop loosens them to 12.0 / 40%, house tightens to 4.0 / 80%; arc-RMSE and artist repeats are critic-wide constants. The set itself is representative, built to exercise those rules.
2026.06.11
allin1 runs for real; the critic learns genre
Two ADRs landed in one day, and both close gaps between what the architecture claimed and what actually executed:
- allin1, finally (ADR 0012). The "allin1 → librosa fallback" chain had a secret: allin1 had never run. Dependency pins blocked the install, and once bridged, its attention layer materialized full T×T score matrices on CPU — 11.3 GB peak for a 30-second clip; a real track OOMed the machine. The fix is two parts: the
allin1CLI now runs as a subprocess from whatever Python can host it, JSON-cached per track, and a vendored memory-linear attention patch does the same math in O(T·k) — verified to float32 epsilon, and it restores the trained positional bias the upstream API migration silently dropped. Same clip now peaks at 2.07 GB, a full track at 5.2 GB. First real output: BPM 133 and 12 sensible sections on a bachata test track. Librosa is finally just the fallback. - Genre profiles as data (ADR 0013). The critic's transition gates — BPM jump and harmonic compatibility — stopped being constants. A frozen profile table — nine genres plus an open default that is exactly the old behavior — feeds the architect, selector, critic, mixer and set sheet from one place. The same set with a 7-BPM jump and one key clash now passes as hip-hop and fails as house; that exact case is a unit test. Slots grew into entry → core → exit play spans sized to the genre's airtime window, crossfades quantize down to powers of two capped per genre, and approval writes a third artifact: a printable set sheet with per-transition bar counts and blend-style notes.
The fast suite went from 227 to 256 tests, still green offline.
2026.06.10
Rescope: the pipeline runs end-to-end
The clap-first-rescope branch landed — the live pipeline now runs brief-to-render without a gap, and the test suite grew from 103 to 227 fast tests, all green without a database, a model, or audio on disk. Three ADRs went code-complete with it:
- Link ingestion (ADR 0009). Paste a Spotify or YouTube URL and the curator resolves metadata, pulls the audio via yt-dlp to FLAC, and feeds it the same idempotent ingest path as a local file. A bulk judge (
dj.taste.judge <url>) scores a whole playlist against my taste vectors without downloading anything. - Two-form export (ADR 0010). Every approved plan exports a
rekordbox.xmlfor manual decks and an.m3u8always;--renderstill produces the automatic .wav. The plan is the product — the render is one of its outputs. - Grid truth (ADR 0011). Real downbeat phase estimation, section bounds snapped to the beat grid, and
first_downbeat_sexported as the TEMPO anchor — so what rekordbox shows matches what the mixer assumed.
2026.06.05
Code-complete, audio-pending
A full review pass landed across phases 0–5: 103 fast tests green, every deterministic module covered. The best catch was real: librosa's beat tracker folds double-tempo genres — DnB, footwork — down to half-tempo, and a naive octave correction elsewhere was quietly un-folding them back. Fixed before any real audio ever got graded by it.
Next gate is contact with reality: the first full ingest of my actual library, where the allin1 → librosa fallback chain finds out which segmentation backend actually installs and which genres break the beat grid.
2026.06.04
The selector earns its loop
The Critic is the interesting half: harmonic compatibility, BPM continuity, energy-arc RMSE, artist spacing — pure math, reused unchanged as the eval scorecard. The Selector revises against its notes until pass or budget (three revisions, then best-so-far wins). The console at the top of this bench plays that exact loop. With no API key the whole pipeline degrades to a deterministic arc and greedy nearest-fit ordering, which is also the A/B baseline the agents have to beat.
2026.06.03
Capture taste before you own the file
The strongest taste signal happens while listening — usually on Spotify, where there's no file to tag. So capture decouples from ingest: a review left in the moment ("Bicep — Glue: dreamy 3am closer, 5") parks in a pending_taste table and auto-applies when the Curator ingests a confident match — ISRC, or name plus duration. Anything ambiguous surfaces for a one-line manual resolve instead of guessing.
That last part is the design: a wrong auto-match would poison the taste layer silently, and the taste layer is the only thing here that can't be recomputed from the audio.
2026.06.01
Bench assembled — two vectors, one row
"Sounds similar" and "is my kind of track" are different questions, so every track carries both: a 512-d CLAP acoustic vector and a 384-d taste vector embedded from my own notes — separate columns on the same Postgres row, blendable per query. "Surprise me" leans acoustic; "my favorites" leans taste. Sections are first-class rows with their own vectors, because a set doesn't mix tracks, it mixes the right parts of tracks on musical boundaries.
Open questions
- Q-01Does taste propagation hold at ~150 hand tags, or does it quietly need 2,000?
- Q-02Genre detection is first-keyword matching — good enough for one-genre briefs, but what should ‘melodic techno into trance’ do?
- Q-03Phase-locked downbeats at the splice: seams currently sit within ~1.5 BPM of each other — is that audible on a real floor or only in a DAW?
- Q-04Plan approval is the acceptance metric — does approving a set as data actually predict liking it as audio?
The thinking that runs underneath this bench
Four Agents, One Nervous System