Appendix
Under the hood
How a voice becomes a painting, and how the painting proves which performance made it. Written for engineers and judges; skippable by everyone else.
1 · The live pipeline
Audio comes off the desk as a stereo line feed. Two paths run in parallel from the same buffer: one transcribes and reads meaning, the other tracks tempo and energy. Both feed a director that decides what the painting does next. The renderer never restarts — it only receives instructions.
desk feed ──► ring buffer (2 s) ──┬──► transcription ──┐
│ (line, t0, t1) │
└──► music features ──┤
(bpm, key, energy)
▼
director (LLM)
│ instruction
▼
canvas renderer ◄── state of the painting
│
projector out (1080p) ──► QR overlay (last 30 s)
│
end of song ──► seal ──► provenance record
SCROLL SIDEWAYS FOR THE FULL DIAGRAM
2 · The director loop
The director is the only component that makes aesthetic decisions. It sees the last sung line, the running summary of the song so far, and a compact description of what is already on the canvas. It returns one instruction — a region, a subject, a palette move, an intensity. It is not allowed to clear the canvas.
Sees
Last line, song-so-far summary, canvas description, bar position.
Returns
One instruction: region, subject, palette shift, intensity, blend.
Cannot
Clear the canvas, move the horizon, or contradict a committed light source.
Logged
Instruction, causing line, model id, timestamp, hash.
Every instruction is logged with the line that caused it and the timestamp it fired. That log is not telemetry — it is the artwork's score, and it ships with the piece.
3 · Why one canvas is hard
Generating an image per line is easy and looks like a slideshow. Growing a single coherent image for four minutes means every new stroke must respect composition, lighting and colour already committed. We keep a persistent latent of the canvas plus a region map of what has been claimed, and constrain each edit to unclaimed or intentionally overpainted area. Overpainting is allowed — the older layer stays visible underneath, which is what makes it read as paint.
Latency budget, per instruction
Target is a visible change within one bar of the line being sung. Numbers here are the design budget, not a measured benchmark.
4 · Provenance of the moment
A painting without its performance is decoration. So when the song ends we seal the image together with the record that produced it, and mint that pair. The token points at the image and at an immutable record of the inputs — not at a promise that we behaved.
{
"performance": { "song": "…", "artist": "…", "venue": "…", "started_at": "…" },
"transcript": [ { "t": 12.480, "line": "…" }, … ],
"director_log":[ { "t": 13.021, "instruction": { … }, "model": "…", "hash": "…" }, … ],
"canvas": { "final_image": "ipfs://…", "keyframes": [ "ipfs://…", … ] },
"attestation": { "scheme": "…", "signature": "0x…" }
}
Anyone holding a print can check three things: the transcript matches the audio of that night, each instruction follows from a line that was actually sung, and the final image is the result of replaying that log. The audience never sees any of this — they see a picture and a code.
5 · Verifiable inference
The interesting claim is not "an AI made art". It is that a specific model, given a specific line at a specific second, produced a specific instruction — and that this can be checked afterwards by someone who was not in the room. We attest the model identity and the input/output pair for each director call, so the chain from voice to brushstroke is auditable end to end.
Where the attestation lives, and how strong it is, is the part still moving. Current build signs the log at the edge and anchors a digest on chain; stronger schemes are on the roadmap and we would rather state that plainly than overclaim it.
6 · Known limits
- Heavy vocal effects and screamed delivery degrade transcription; the system falls back to music-only mode.
- Songs under ninety seconds do not give the painting enough room to accumulate.
- Multilingual switching mid-song works, but the palette can lurch.
- It interprets. It will sometimes paint the wrong noun with conviction. We treat that as part of the medium.