🧺

pianosu Dev Notes

Phaser3 boilerplate and JS game dev links: https://github.com/Sidaroth/Phaser3.Boilerplate

Phaser Multitouch Piano: http://phaser.io/examples/v3/view/input/multitouch/multi-touch-test

Phaser 100 line keyboard piano: https://github.com/akrolsmir/Phaser-Piano

Ableton in-browser how to compose music: https://learningmusic.ableton.com/index.html

This is a ton of fun!

Phaser + ToneJS: https://douglastarr.com/keyboard-boogie

Better Phaser docs: https://rexrainbow.github.io/phaser3-rex-notes/docs/site/timer/#start-timer

Piano Typewriter: http://www.picatino.com/piano_typewriter/the-entertainer/

  • Nice backing track, sheet music display, but tells you exactly what to press. Familiar to sheet music readers
  • Also waits until you hit the right key, so you get it right

Synesthesia Game: https://www.synthesiagame.com/

  • Nice visualization, also: waits until the right key is hit
  • Also shows sheet music

Tensorflow js AI for piano midi generation: https://magenta.tensorflow.org/performance-rnn

Using webcam to generate lipsync: https://blog.tensorflow.org/2020/07/using-machine-learning-in-browser-to-lip-sync.html

Shawn Wasabi making music on Nintendo Labo: https://www.youtube.com/watch?v=D5KHP0mQ5fE

@March 20, 2021

Youtube tutorial on levels of Jazz:

Jazz Levels

Sheet music as a service: https://flat.io/

Hm... a lot of smart people I know have taken an interest in music.

  • Angie
  • Eric Zhang
  • Eric Jang
  • Is it Asian competition? Does it go deeper?

@March 21, 2021

Pitch Shifting (Autotune) in JS: Paper, GIthub

EarSketch: Online IDE for making music

Music Source Separation

Separating out voices from songs is a great first step

Audacity vocal removal works okay, 7/10

https://vocalremover.org/ works great out-of-the-box, 9/10

LaSAFT: Cool demo, 8/10 paper, but code doesn't repro ☹️

Demucs: No demo, but lots of GH stars and sponsored by FB research, 8/10 https://github.com/facebookresearch/demucs

Waveform to midi notes

Essentia PitchMelodia (C++ song extraction library with WASM bindings): https://mtg.github.io/essentia.js/examples/offline/pitchmelodia/example.html

Suggestion: Manually move 3 songs into

Pipeline:

  1. Youtube to Mp3
  2. Mp3 to voice/drums/bass/instruments waveform
  3. Waveform to notes?

Hard things about transcribing:

  • Want to hear the backing music, with a 4/4 beat indicator
  • Want to line up the transcribed notes to the song
  • Want to be able to skip backwards to perfect an area (pretty sure the first section is good)

Unsolved questions:

  • Where does data go?
    • Is it in Vue (and we pass a VueApp reference around?)
    • In Phaser itself?
  • What widgets do we use?
    • HTML/Vue based
      • More familiar, faster dev speed, debuggable
    • Phaser-based eg Rex
      • Better user experience, no weird routing design,

TODO

Timeline: Ability to scrub back and forth
Notes

Sinclair:

  • Want to hold down notes to play longer
  • Feels like there should be two keys between F and J
    • Could make starting notes G and H, or configurable
    • Maybe QWER or 1234 for the home row?
    • Bryant also had this feedback
  • Virality:
    • Tiktok virality: half video is someone else's, half is yours
      • AKA rounds

Eric: need hit feedback

Adrian: Non diatonic songs?

Adrian: Tools inspire you to build things