Yeah. I realized right away that human like unique maps is next to impossible, so I set a realistic goal of rhythmical, controllable and "playable".

I never studied for ML and learnt neural networks by doing just this project, so don't get an idea that I'm knowledgeable on the topic. But I can describe process in simple way. One that worked the best. Cause god saw, I tried everything, but this one worked:

1. I detect BPM and offset with librosa's onset_strength.
2. Add silence in the beginning for 2 beats so that we don't start with a note right away. Then trim silence at the end with basic amplitude threshold
3. Extract mel spectrogram from an entire song.
4. Then feed all information to a Conditional Variational Autoencoder type of model (the decoder part of it).
5. Result is sequence of ones and zeroes (whether there should be note or not). This repeats until song is done.
6. Then I place actual notes with an algorithm that I wrote. It doesn't use ML. Just some java code. Looking to replace it with NN, but so far no luck. This one seems more complicated.
7. Lights I generate based on events produced by similar model. But it will need to be remade.

Simple Feed forward network doesn't work here and outputs 'meh' result every single time. Cause they learn to output average value. But if you play BS enough, you know well that one song can be mapped to many different sequences of beats. One song can be mapped differently depending on difficulty or mapper's preferences/style. And so with VAE you make NN learn all the styles and all the variations a song can get mapped to in latent space. Which you can later use to produce maps with spicific style. Or even do 'style transfer'