Building a Low-Latency Virtual Piano and Drum Kit on iOS with AVAudioEngine

Where do the sounds come from? Either a SoundFont you have the rights to ship, or your own recordings, with one sample every few semitones loaded via loadAudioFiles(at:). Sample quality is the single biggest factor in "sounds real", and it's a licensing question as much as a technical one: check the licence of any sound bank before you bundle it.

4. Drums: preloaded buffers and a player-node pool

• Decode each drum sample into an AVAudioPCMBuffer once, at launch.
• Keep a small pool of AVAudioPlayerNodes per pad.
• On each hit, schedule the buffer on the next node in the pool, round-robin.

import AVFoundation

final class DrumKit {
    private let engine: AVAudioEngine
    private var buffers: [String: AVAudioPCMBuffer] = [:]
    private var pools: [String: [AVAudioPlayerNode]] = [:]
    private var nextIndex: [String: Int] = [:]
    private let voicesPerPad = 4     // how many overlapping hits per pad

    init(engine: AVAudioEngine) { self.engine = engine }

    /// Call once, before starting the engine.
    func load(pads: [String]) throws {
        for pad in pads {
            guard let url = Bundle.main.url(forResource: pad, withExtension: "caf") else {
                throw CocoaError(.fileNoSuchFile)
            }
            let file = try AVAudioFile(forReading: url)
            guard let buffer = AVAudioPCMBuffer(pcmFormat: file.processingFormat,
                                                frameCapacity: AVAudioFrameCount(file.length)) else {
                throw CocoaError(.fileReadCorruptFile)
            }
            try file.read(into: buffer)
            buffers[pad] = buffer

            pools[pad] = (0..<voicesPerPad).map { _ in
                let node = AVAudioPlayerNode()
                engine.attach(node)
                engine.connect(node, to: engine.mainMixerNode, format: buffer.format)
                return node
            }
            nextIndex[pad] = 0
        }
    }

    /// Call after engine.start().
    func startNodes() {
        pools.values.flatMap { $0 }.forEach { $0.play() }
    }

    func hit(_ pad: String, velocity: Float = 1.0) {
        guard let buffer = buffers[pad], let pool = pools[pad], let i = nextIndex[pad] else { return }
        let node = pool[i]
        nextIndex[pad] = (i + 1) % pool.count
        node.volume = velocity
        // .interrupts: if this voice is still ringing from 4 hits ago, restart it cleanly.
        node.scheduleBuffer(buffer, at: nil, options: .interrupts, completionHandler: nil)
    }
}

The whole setup:

let engine = AVAudioEngine()
let piano = PitchedInstrument(engine: engine)
let drums = DrumKit(engine: engine)

try InstrumentSession.activate()
try piano.loadSoundBank(named: "Piano")
try drums.load(pads: ["kick", "snare", "hihat-closed", "hihat-open", "crash", "tom-high", "tom-low"])
engine.prepare()
try engine.start()
drums.startNodes()

5. A multi-touch keyboard that feels like a keyboard

The direct approach is one UIKit view for the whole keyboard that tracks every UITouch itself:

import UIKit

final class KeyboardView: UIView {
    var firstMidi: UInt8 = 60           // C4
    var octaves = 2
    var onNoteOn: ((UInt8) -> Void)?
    var onNoteOff: ((UInt8) -> Void)?

    private var touchNotes: [ObjectIdentifier: UInt8] = [:]   // which finger holds which note

    override init(frame: CGRect) {
        super.init(frame: frame)
        isMultipleTouchEnabled = true      // essential: off by default
    }
    required init?(coder: NSCoder) { fatalError("init(coder:) is not supported") }

    // MARK: Touch tracking

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { press(t) }
    }

    override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches {
            let id = ObjectIdentifier(t)
            let newNote = note(at: t.location(in: self))
            guard newNote != touchNotes[id] else { continue }    // still on the same key
            if let old = touchNotes[id] { onNoteOff?(old) }       // glissando: release old…
            touchNotes[id] = newNote
            if let n = newNote { onNoteOn?(n) }                   // …and press new
            setNeedsDisplay()
        }
    }

    override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { release(t) }
    }

    override func touchesCancelled(_ touches: Set<UITouch>, with event: UIEvent?) {
        for t in touches { release(t) }     // never leave a note stuck on
    }

    private func press(_ t: UITouch) {
        guard let n = note(at: t.location(in: self)) else { return }
        touchNotes[ObjectIdentifier(t)] = n
        onNoteOn?(n)
        setNeedsDisplay()
    }

    private func release(_ t: UITouch) {
        if let n = touchNotes.removeValue(forKey: ObjectIdentifier(t)) { onNoteOff?(n) }
        setNeedsDisplay()
    }

    // MARK: Geometry

    private static let whiteOffsets: [UInt8] = [0, 2, 4, 5, 7, 9, 11]          // C D E F G A B
    private static let blackOffsets: [(after: Int, semitone: UInt8)] =
        [(0, 1), (1, 3), (3, 6), (4, 8), (5, 10)]                               // C# D# F# G# A#

    private var whiteKeyWidth: CGFloat { bounds.width / CGFloat(7 * octaves) }

    /// Black keys sit on top, so test them first.
    func note(at p: CGPoint) -> UInt8? {
        guard bounds.contains(p) else { return nil }
        let w = whiteKeyWidth
        if p.y < bounds.height * 0.6 {
            for octave in 0..<octaves {
                for (after, semitone) in Self.blackOffsets {
                    let x = CGFloat(octave * 7 + after + 1) * w - w * 0.3
                    if CGRect(x: x, y: 0, width: w * 0.6, height: bounds.height * 0.6).contains(p) {
                        return firstMidi + UInt8(octave * 12) + semitone
                    }
                }
            }
        }
        let index = Int(p.x / w)
        let octave = index / 7, step = index % 7
        return firstMidi + UInt8(octave * 12) + Self.whiteOffsets[step]
    }
}

The essential details:

On iOS 12 this is the only option, since SwiftUI needs iOS 13. Even on a SwiftUI app today, I'd wrap this view in a UIViewRepresentable. Touch handling for an instrument is one of the places where UIKit is still simply the better tool.

Drum pads follow the same idea in simpler form: a grid of rectangles, touchesBegan triggers hit(_:), and there's no note-off.

6. Teaching note names and positions

The listing describes the app as "teaching beginners the names & positions of notes on the piano". On the keyboard view, that's two small things: draw a label on each key, and let learners switch between naming systems.

enum NoteNaming: String, CaseIterable {
    case letters      // C D E F G A B
    case solfege      // Do Re Mi Fa Sol La Si

    static let letterNames = ["C", "C♯", "D", "D♯", "E", "F", "F♯", "G", "G♯", "A", "A♯", "B"]
    static let solfegeNames = ["Do", "Do♯", "Re", "Re♯", "Mi", "Fa", "Fa♯", "Sol", "Sol♯", "La", "La♯", "Si"]

    func name(for midi: UInt8, withOctave: Bool = true) -> String {
        let pitchClass = Int(midi) % 12
        let base = self == .letters ? Self.letterNames[pitchClass] : Self.solfegeNames[pitchClass]
        return withOctave ? "\(base)\(Int(midi) / 12 - 1)" : base      // MIDI 60 = C4
    }
}

// NoteNaming.letters.name(for: 60) == "C4"; NoteNaming.solfege.name(for: 64) == "Mi4"

In draw(_:), draw white-key labels near the bottom of each key and black-key labels only when a "show sharps" toggle is on; for beginners, too many labels is noise. Highlighting the pressed key matters more than any label. The immediate visual "this is the key you just played" is how position is learned.

7. Real-world example: Drums n Piano

This is the approach I'd use, and recommend, for Drums n Piano.

At the time of writing, its App Store listing shows:

How the listing maps to this article:

| What the listing says | Where it's covered | | --- | --- | | "Simply play and learn the piano, drums or xylophone wherever you are" | A sampler per pitched instrument; a player-node pool for drums (sections 3–4) | | "This virtual app sounds real, just like the original instrument!" / "Premium sound quality" | Real samples loaded once, a low-latency session, natural note-off releases (sections 2–3) | | "Teaching beginners the names & positions of notes on the piano" | Note labels and pressed-key highlighting (section 6) | | "Play or learn your music chords" | Per-touch tracking with multi-touch enabled (section 5) |

The listing's audience is broad: "a music teacher, singer, songwriter, or beginner or joyful kid". That's a good reminder that kids mash several keys with their whole hand. A multi-touch view that never leaves a stuck note is worth more than any feature.

8. Before you ship a virtual instrument: a checklist

• Nothing loads on tap. Samples and sound banks load at launch or on instrument switch; taps only trigger notes.
• The session is .playback (or a deliberate .ambient), with a small preferred buffer, and you've logged what you actually got.
• Polyphony is tested. Ten-finger chords on the piano; rapid repeated hits on every drum pad.
• There are no stuck notes. Test cancelled touches, a phone call mid-chord, backgrounding mid-note and Control Center swipes.
• isMultipleTouchEnabled is on, and each touch owns its note.
• Glissando works: sliding across keys releases and presses notes as you go.
• Sample licences are cleared for everything you bundle.
• You've tested on real hardware with wired and Bluetooth audio, and the app tells users that wired is tighter.

Wrapping up

A virtual instrument is mostly preparation: decode everything up front, pick the right player for the job (a sampler for pitched instruments, a node pool for one-shots), keep the audio buffer small, and track every finger yourself. Get those right, and the sound arrives with the touch, which is all a player really asks for.

Play it, or explore my other apps: