pianosu Dev Notes

Phaser3 boilerplate and JS game dev links: https://github.com/Sidaroth/Phaser3.Boilerplate

Phaser Multitouch Piano: http://phaser.io/examples/v3/view/input/multitouch/multi-touch-test

Phaser 100 line keyboard piano: https://github.com/akrolsmir/Phaser-Piano

Phaser + ToneJS: https://douglastarr.com/keyboard-boogie

Better Phaser docs: https://rexrainbow.github.io/phaser3-rex-notes/docs/site/timer/#start-timer

Piano Typewriter: http://www.picatino.com/piano_typewriter/the-entertainer/

• Nice backing track, sheet music display, but tells you exactly what to press. Familiar to sheet music readers
• Also waits until you hit the right key, so you get it right

Synesthesia Game: https://www.synthesiagame.com/

• Nice visualization, also: waits until the right key is hit
• Also shows sheet music

Tensorflow js AI for piano midi generation: https://magenta.tensorflow.org/performance-rnn

Using webcam to generate lipsync: https://blog.tensorflow.org/2020/07/using-machine-learning-in-browser-to-lip-sync.html

Shawn Wasabi making music on Nintendo Labo: https://www.youtube.com/watch?v=D5KHP0mQ5fE

@March 20, 2021

Sheet music as a service: https://flat.io/

@March 21, 2021

Pitch Shifting (Autotune) in JS: Paper, GIthub

EarSketch: Online IDE for making music

Music Source Separation

Separating out voices from songs is a great first step

Audacity vocal removal works okay, 7/10

https://vocalremover.org/ works great out-of-the-box, 9/10

LaSAFT: Cool demo, 8/10 paper, but code doesn't repro ☹️

Demucs: No demo, but lots of GH stars and sponsored by FB research, 8/10 https://github.com/facebookresearch/demucs

Waveform to midi notes

Essentia PitchMelodia (C++ song extraction library with WASM bindings): https://mtg.github.io/essentia.js/examples/offline/pitchmelodia/example.html

Suggestion: Manually move 3 songs into

Pipeline:

1. Youtube to Mp3
2. Mp3 to voice/drums/bass/instruments waveform
3. Waveform to notes?

Hard things about transcribing:

• Want to hear the backing music, with a 4/4 beat indicator
• Want to line up the transcribed notes to the song
• Want to be able to skip backwards to perfect an area (pretty sure the first section is good)

Unsolved questions:

• Where does data go?
• Is it in Vue (and we pass a VueApp reference around?)
• In Phaser itself?
• Phaser has a data builtin
• Or could integrate with LokiJS, an in-memory firestore-like db
• Rex Finite State Machine?
• What widgets do we use?
• HTML/Vue based
• More familiar, faster dev speed, debuggable
• Phaser-based eg Rex
• Better user experience, no weird routing design,

TODO

Sinclair:

• Want to hold down notes to play longer
• Feels like there should be two keys between F and J
• Could make starting notes G and H, or configurable
• Maybe QWER or 1234 for the home row?
• Bryant also had this feedback
• Virality:
• Tiktok virality: half video is someone else's, half is yours
• AKA rounds

Eric: need hit feedback

Adrian: Non diatonic songs?