Coding a musical game using MIDI

[bruno.dumoulin]

Hello,

I'm an beginner musician who tinkers with code in his spare time and I would like to write a little video game to help me make progress in music. I'm a bit of a jack of all trades, I have a guitar, a keyboard and a dualo.

I would like to write a simple video game: the game plays a midi melody, I listen to it, and then I have to reproduce it, by ear, with my instrument.

At first, I would like to be able to play the game using my dualo connected as a MIDI keyboard.

The game plays a MIDI melody then pauses and waits for me to play the notes. Then I use my dualo to play the melody back to the game, and the game simply checks that I am playing the right notes in the right order, ignoring the rhythm.

In other words, if the midi file contains the melody for "frère jacques", the game simply checks that I am indeed playing "do re mi do, do re mi do, mi fa sol, mi fa sol..."

(Later on, I would like to be able to play it with my guitar or piano, but that means capturing audio, transforming the captured sounds into midi notes... and uh... for now, writing the code to play directly in midi is complicated enough for me!)

I have a little bit of experience in programming, but I'm a beginner in coding video games... and then MIDI is a strange beast... although it looks quite simple at first glance, but in fact i find it... a bit confusing. And the documentation I found is rather cryptic.

Here is precisely what I am trying to do
Step 1: I just want to write the code that opens the midi file, plays it in audio, and gets the sequence of notes of the melody

Step 2: write the code that plays the sound when I press a key on my dualo and compares my note sequence with the one from the file

In my mind, that sounded easy... and then... well...

If I understand correctly, MIDI is only text information that describes sound events, but a MIDI file, in itself, does not include any sound. If the MIDI file contains an event 0x92 0x3D 0x78... , that means "note on", note 61 (meaning C# 4), velocity 78, right?

And my code must forward the event to some virtual synthetizer library that will synthesize sound, right?

Same thing if it's my dualo that sends the event, I guess? My kb will send a "note on" when I press a key, and a "note off" when I release the key?

From what I understand, to have sound, I need a virtual synthesizer, I guess it's a library?

In my research, I found fluidsynth.lua, a lua library that interfaces with fluidsynth Does this sound like a good idea? Should I keep digging there? Or do you have something better to recommend? It looks like the people at dualo are developping their new app around SurgeXT, would that be a good option for my game? If so, how can I make surgeXT calls from Lua/Löve2d?

A lot of things I find mention ALSA... but that stands for "Advanced Linux Sound Architecture", I guess that means it won't run on windows?

Then the sounds themselves are organized in a sound font, right?

The midi file also contains the info telling the synth which instrument to use, right? So, I would need to create some kind of mapping to say that this instrument corresponds to this sound font?

Has any of you ever coded musical games using MIDI? Can anyone here orient me to the appropriate libraries/tools?

Any pointers would be very appreciated. I bought the book "the midi manual" by David Miles Huber, but it's a difficult read... do you have other resources to recommend?

Thanks
Bruno

In case you're wondering, a dualo is a small synth that looks like that (https://dualo.com/dualo/)

dualo.jpg (34.53 KiB) Viewed 3527 times

[zorg]

Hi and welcome to the forums.

I would like to write a simple video game: the game plays a midi melody, I listen to it, and then I have to reproduce it, by ear, with my instrument.

I have to break it to you, that's not simple.

At first, I would like to be able to play the game using my dualo connected as a MIDI keyboard.
The game plays a MIDI melody then pauses and waits for me to play the notes. Then I use my dualo to play the melody back to the game, and the game simply checks that I am playing the right notes in the right order, ignoring the rhythm.
In other words, if the midi file contains the melody for "frère jacques", the game simply checks that I am indeed playing "do re mi do, do re mi do, mi fa sol, mi fa sol..."

LÖVE doesn't have a standard way of handling MIDI devices, and that needs a per-OS implementation for any library wanting to implement such a thing. (https://github.com/SiENcE/lovemidi exists, but i haven't tried to use it yet so i can't say whether that works decently or not...)
That said, it isn't that hard to store input data and then check the notes against whatever notes you were trying to mimic, especially since you don't want to care about the timing, only the pitches.

(Later on, I would like to be able to play it with my guitar or piano, but that means capturing audio, transforming the captured sounds into midi notes... and uh... for now, writing the code to play directly in midi is complicated enough for me!)

Getting the strongest pitch from actual audio is a relatively difficult problem, and you'll probably need to use an FFT lib in the process... and there's no guarantee about that giving you the most accurate pitches either.

I have a little bit of experience in programming, but I'm a beginner in coding video games... and then MIDI is a strange beast... although it looks quite simple at first glance, but in fact i find it... a bit confusing. And the documentation I found is rather cryptic.

The docs/specs i found on MIDI liked to not explain that MIDI is both a file format and a transmission format, and some details only pertain to one or the other, but not both.

Here is precisely what I am trying to do
Step 1: I just want to write the code that opens the midi file, plays it in audio, and gets the sequence of notes of the melody
Step 2: write the code that plays the sound when I press a key on my dualo and compares my note sequence with the one from the file

With löve, you can open a MIDI file, but it will be treated as an audio stream, like wav or mp3; you can't get the notes from it, and there's a high chance that it will be played back without any instruments whatsoever (due to löve using libmodplug to handle midi files, and that requires Timidity to exist on your system, on a specific path.) ...i probably should fix my own midi reader library, add a soft synth to it, and release it as a library.

If I understand correctly, MIDI is only text information that describes sound events, but a MIDI file, in itself, does not include any sound. If the MIDI file contains an event 0x92 0x3D 0x78... , that means "note on", note 61 (meaning C# 4), velocity 78, right?

Something like that, but the standard is a bit more complex in places; sometimes some values are allowed to be skipped, for instance.

And my code must forward the event to some virtual synthetizer library that will synthesize sound, right?

Considering there's nothing inbuilt in löve to handle it, yes.

Same thing if it's my dualo that sends the event, I guess? My kb will send a "note on" when I press a key, and a "note off" when I release the key?

Probably.

From what I understand, to have sound, I need a virtual synthesizer, I guess it's a library?

It's whatever that can handle reading midi messages from a midi port; could be a standalone application, could be a DAW like FL studio, could be a library for löve or something else.

In my research, I found fluidsynth.lua, a lua library that interfaces with fluidsynth Does this sound like a good idea? Should I keep digging there? Or do you have something better to recommend? It looks like the people at dualo are developping their new app around SurgeXT, would that be a good option for my game? If so, how can I make surgeXT calls from Lua/Löve2d?

Fluidsynth is a standalone software synthesizer that can use sf2 soundfonts to turn midi notation into audio; i have found like two lua libraries that wrap fluidsynth, but i'm not sure if those would work with löve or not; technically it's not a requirement if you just want to deal with midi messages; one can write a simple synth that just outputs a specific frequency sine or square or whatever waveform without any libraries (this is something you can do with löve) so you can deal with non-polyphonic midi stuff this way too, if you don't want to mess around with external dependencies too much (except for the midi interface stuff for input, i mean)

A simpler way to play sounds would be to have one sound file of one instrument, say a piano note or a guitar string being strung, then you could load that into löve and play it back with whatever pitch it needs to be. (as long as the sound's own pitch is not too far from what the midi note is, it should not be too bad.)

A lot of things I find mention ALSA... but that stands for "Advanced Linux Sound Architecture", I guess that means it won't run on windows?

OSS, ALSA, ASIO, WASAPI and others are just audio drivers that work on whatever OS they were written for; even löve uses these through OpenALSoft if it can; you don't need to worry about any of this if you want to use löve.

Then the sounds themselves are organized in a sound font, right?

You don't need to use soundfonts; midi are just events, what you do with them is up to you; show a color, play a sound, vibrate your controller.

The midi file also contains the info telling the synth which instrument to use, right? So, I would need to create some kind of mapping to say that this instrument corresponds to this sound font?

Yes and no;
the currently used midi standard does define a set of 128 "instruments" in bank 0 (plus a few drumkits for midi channel 10) that people can assume should sound more or less how those instruments would sound... but even that's not a guarantee; it's just the names of the instruments. other vendor-specific extensions like Roland's GS and Yamaha's XG also contain alternative/extra "instruments" but there's no real concensus.
Also, soundfonts can take up a lot of space, considering the better ones do multisampling, so each one might contain 10+ high quality samples... some might have different ones for the same pitch range, and would play back in a round-robin (one after the other) or in a semi-random fashion... that bloats it even more.

Has any of you ever coded musical games using MIDI? Can anyone here orient me to the appropriate libraries/tools?

I did write my own screamtracker3 replayer, but that's not exactly midi... in some ways it's a simpler format, in other ways it's more complicated.
I did mention and link science's midi library above, you could try using that.

Any pointers would be very appreciated. I bought the book "the midi manual" by David Miles Huber, but it's a difficult read... do you have other resources to recommend?

Nope, i personally tried to write a midi parser using specs like this: https://www.cs.cmu.edu/~music/cmsip/rea ... pdated.pdf
And yes, it's not an easy read.

Good luck

[BrotSagtMist]

I salute this extensive answer. Everyone can learn from that.

[bruno.dumoulin]

Hi and welcome to the forums.

I have to break it to you, that's not simple.

Yes, I appreciate that now... oh my! Sounds like this little project might keep me busy for a while
Thanks for taking the time to write such an extensive response

LÖVE doesn't have a standard way of handling MIDI devices, and that needs a per-OS implementation for any library wanting to implement such a thing. (https://github.com/SiENcE/lovemidi exists, but i haven't tried to use it yet so i can't say whether that works decently or not...)

I'll look into lovemidi, thanks for the link. Could it be that Löve is not the best choice of langage for what I'm trying to do?

The docs/specs i found on MIDI liked to not explain that MIDI is both a file format and a transmission format, and some details only pertain to one or the other, but not both.

That's a very interesting remark. Reading it was a little light bulb moment for me. You're right! I need to look into this further and understand the nuances better...

So reading/parsing the midi file is one thing
Playing it is another
And handling input from a MIDI device is yet another thing

Hmmm... food for thought

Thanks for the insight.

With löve, you can open a MIDI file, but it will be treated as an audio stream, like wav or mp3; you can't get the notes from it, and there's a high chance that it will be played back without any instruments whatsoever (due to löve using libmodplug to handle midi files, and that requires Timidity to exist on your system, on a specific path.) ...

That's where I'm hoping Peter Billam's set of MIDI lua libraries might come in handy https://pjb.com.au/midi/
Else I guess I will need to write my own MIDI parser... later in your reply, you said you "tried to" write one? You gave up?

i probably should fix my own midi reader library, add a soft synth to it, and release it as a library.

That sounds like a nice idea How far along are you? A soft synth is what turns the midi info into audio, right? Are you going to write your own?

Fluidsynth is a standalone software synthesizer that can use sf2 soundfonts to turn midi notation into audio; i have found like two lua libraries that wrap fluidsynth, but i'm not sure if those would work with löve or not;

i'll have to look into that

technically it's not a requirement if you just want to deal with midi messages; one can write a simple synth that just outputs a specific frequency sine or square or whatever waveform without any libraries (this is something you can do with löve) so you can deal with non-polyphonic midi stuff this way too, if you don't want to mess around with external dependencies too much (except for the midi interface stuff for input, i mean)

Hmmm... simply outputing a frequency will result in audio that won't sound very musical, will it?
Writing a virtual synth that will make a piano piece sound like an actual piano... feels way beyong my programming abilities... And down the line, I'm hoping to add some chord recognition, that will take polyphonic support... so I'd rather find an appropriate open source virtual synth and use that.

A simpler way to play sounds would be to have one sound file of one instrument, say a piano note or a guitar string being strung, then you could load that into löve and play it back with whatever pitch it needs to be. (as long as the sound's own pitch is not too far from what the midi note is, it should not be too bad.)

Again that sounds like trying hard not to use an existing synth? Why would I want to do that?

You don't need to use soundfonts; midi are just events, what you do with them is up to you; show a color, play a sound, vibrate your controller.
(...)
the currently used midi standard does define a set of 128 "instruments" in bank 0 (plus a few drumkits for midi channel 10) that people can assume should sound more or less how those instruments would sound...

So if I understand you correctly, any virtual MIDI synth will handle those instruments (using a soundfont or not) and I don't really need to worry about using the soundfont, the synth will handle it if it needs it?

I did write my own screamtracker3 replayer, but that's not exactly midi... in some ways it's a simpler format, in other ways it's more complicated.

I'm curious, why did you choose not to use MIDI then?

Nope, i personally tried to write a midi parser using specs like this: https://www.cs.cmu.edu/~music/cmsip/rea ... pdated.pdf
And yes, it's not an easy read.

Good luck

Thanks, sounds like I will need some!

[Nikki]

Hi, nice plan and what a great detailed answer by zorg (great detailed question too btw!),

I have some experience in using lovemidi, its great when it's working, (its a very thin wrapper around a very small c library) but it's using a so/dll file with the complexities that come from that ( I needed to compile it for M1/ and x64 for example).

Anyway, if you aren't married to the idea of using love2d for this, I think I would advise using p5.js for this.
It has almost all your requirements covered;
(midi in (via webmidi), various synth options, pitch detection of microphone in check its ml5 lib ) over at the p5 libraries.

The only thing missing from it, is the part where you 'read a midi file and have control about it' for that you could use tonejs-midi and have a human readable json representation of a midi file at your fingertips.

It also has quite a few examples floating around of various audio programming sketches.

[bruno.dumoulin]

Anyway, if you aren't married to the idea of using love2d for this, I think I would advise using p5.js for this.
It has almost all your requirements covered;

I'm definitely not married to using Löve or even Lua. I enjoy learning and I'm not afraid of hard work, but choosing the best langage/coding environment is clearly a very important step. There is no reason to struggle doing it in langage x when langage y is a more appropriate option. Having said that, I find it funny/intriguing that you're pointing to JS...

You're the second person to point that way. I went to the Qt forum to ask "is Qt/C++ a good choice for my projet" and someone replied "have you considered html/JS"...

And I replied:
"Thanks for your reply. I have spent some time exploring the html/javascript route... but gosh does that langage and it's clever multiple programming paradigms feel annoyingly confusing. I feel totally clueless about building an efficient and elegant software architecture in JS and there are so many people out there publishing tons and tons of very poorly written js code, it feels like a jungle to try and find something inspiring to learn from. I have yet to find inspiring learning sources for JS. I even purchased a few books and read some online o'reilly books... but even in the books, the coding does not feel elegant and efficient at all...

So unless I find some inspiring learning material, I'd much rather stay far away from JS...

Lua is a pleasure to read and write, it's elegant and readable, the only other language that felt as elegant to write was Ruby...
So I'd rather spend my learning time exploring stuff based on C/C++ (or maybe go back to Ruby? or possibly Python?)

Moreover, I'd like to play my game mostly offline... so I can cut off all external stimulation and concentrate on the music..."

Your message makes me reconsider. Am I being stupid or stubborn? What do you think Nikki, have you found a good way to write readable js? Can you recommend a place to help learn how to efficiently organise my code in JS?

[Nikki]

Your message makes me reconsider. Am I being stupid or stubborn? What do you think Nikki, have you found a good way to write readable js? Can you recommend a place to help learn how to efficiently organise my code in JS?

Oof these are hard questions, I do feel its important that the language you are working with resonates with you in a good way, on the other hand, to me personally JS and lua are very similar languages; both prototype based, dynamic scripting languages.

JS has much more history and changes behind it offcourse, I understand looking up documentation for a new learner can be a daunting task, I learned JS in almost a different era and the language is much more evolved nowadays.
Javascript the good parts was a nice read, I don't know if everything in there is still as relevant today.
And also, the way you described your problem in the original post makes me think you have a good grasp on the problem, you just need to look up individual parts how to make them, sometimes the urge to read books to learn doesn't help you much then.

Anyway the good news, that p5 I linked to has very clear tutorials and a nice and clean online editor, and is really aimed at creative people wanting to use code for creative endeavours, the organising of code is a discussion on it's own, but because p5 has almost all the functionality you need out of the box, I kinda feel this isn't going to be the enormous undertaking it would otherwise be, meaning you don't need to organise that many parts, it's already done. you just have a setup and draw callback function (like love2d's load and draw)

Well, I think I would maybe play around a little bit with it and see, worst case scenario you play around with it a few days to learn it's not your cup of tea. Oh and you can run the whole thing locally too and turn of your wifi or something

[zorg]

...

> Could it be that Löve is not the best choice of langage for what I'm trying to do?
People might say it's geared towards games, and that's not wrong, but it's still possible to do most things with it... i mean, others have also made tools with it, some made soft synths as well; i'm also working on some stuff, including a more or less complete composing software, so it's definitely possible.

> That's where I'm hoping Peter Billam's set of MIDI lua libraries might come in handy
It could be a possibility, although the one i saw on that page does offline conversion, not realtime... it might still be fine for your use case though, if it can be either used through löve, or called as an external application.

> later in your reply, you said you "tried to" write one? You gave up?
I put it on hold, since i couldn't figure out why it was parsing some things wrong... not a good idea for me to suffer through that kind of annoyance for too long.

>That sounds like a nice idea How far along are you? A soft synth is what turns the midi info into audio, right? Are you going to write your own?
As i mentioned before, the midi file parser is on hold for now; and for synths, i have too many projects and random files with stuff i want to implement, it's hard to combine all of that into one thing. But yes, i'm writing my own things, with info from elsewhere of course; different synthesis types and waveforms and effects and whatnot.
Also, technically speaking, a synth just turns whatever data it supports, be it midi or otherwise, and makes audio from it, either to your speakers, or writing it directly to disk.

>Hmmm... simply outputing a frequency will result in audio that won't sound very musical, will it?
You'll still hear the correct pitches, but yes, if everything is just a sine wave, it'll get annoying real fast; that's actually the problem with Löve's own handling of midi, if you don't have timidity... and i'd wager most people don't, hence one of my goals being to create a lua-side parser and synth (and sampler) library that could work cross-platform to play back midi with consistent instruments... at least if using the default General Midi ones. (although at that point, i could make my own format that combines the midi file with some instrument definitions as well, to make it more versatile, and still be portable... this probably already exists though, people just don't know/forgot about it, like pat files or such; most of this was DOS and Amiga era stuff anyway, which is considered ancient.)

> And down the line, I'm hoping to add some chord recognition, that will take polyphonic support...
Technically, i should have said that i expected your input midi files to only use one channel / one instrument; polyphony within those constraints is possible and still relatively simple considering the program you want to write as a test.

> Again that sounds like trying hard not to use an existing synth? Why would I want to do that?
Because none exists as far as i know, in the context of Löve applications / libraries... that said, i do recall 1 or 2, but i'm not sure if the people did write them with löve, or if they moved to something else. In any case, this one is really simple, and i'd recommend to do this. (re: A simpler way to play sounds would be to have one sound file of one instrument, say a piano note or a guitar string being strung, then you could load that into löve and play it back with whatever pitch it needs to be.)

> So if I understand you correctly, any virtual MIDI synth will handle those instruments (using a soundfont or not) and I don't really need to worry about using the soundfont, the synth will handle it if it needs it?
Nope, depends on the synth what sound it produces and whether it cares about midi instruments or not; even using a soundfont doesn't guarantee it'll have all sounds that the GM (general midi) standard defines... and even if it did, the quality can and will differ... you can check out some youtube vids that play the OST of transport tycoon deluxe through multiple midi-compatible devices to hear the difference (and yes, they all play back the same midi files)

https://www.youtube.com/watch?v=EmSEfKAYyNo using Windows GeneralMidi sounds.
https://www.youtube.com/watch?v=JTi6LG8aeK8 using the AdLib sound card (has a yamaha chip, midi instruments are purely synthesized)
https://www.youtube.com/watch?v=ol6Wv4bn2Lc using the Gravis Ultrasound sound card (it plays back samples, sounding subjectively better)
https://www.youtube.com/watch?v=Y2n1tIaMy5E using a roland MT-32 hardware unit (hybrid synth+sampler, not GM compliant, instruments will sound different)
https://www.youtube.com/watch?v=jSJu8u3QOmk using a roland SC-88 hardware unit (hybrid synth+sampler, but people mostly used the high-quality samples instead of the cut-down synthesizer it had)

> I'm curious, why did you choose not to use MIDI then?
A few reasons; midi is more limited in terms of how much control you have over the individual voices compared to a tracker module; also, the format was simpler to implement compared to midi, and it also comes with its own sound files, meaning less work since i didn't need to write a synth for that.


...btw, if it was up to me, i'd not use anything related to web or whatever that would need a browser to run, that feels cheap to me and i'm averse to have whatever browser i have running for a local tool...

[Nikki]

I threw together a little sketch
It has midi playing from a file (starts after 1 sec) and you can hook up a midi keyboard and play.
It doesn't do any of the real gameplay

[zorg]

I threw together a little sketch
It has midi playing from a file (starts after 1 sec) and you can hook up a midi keyboard and play.
It doesn't do any of the real gameplay

On my end, the sound crackles (might be underflowing) and it's just sine waves instead of instruments...and not all cases might support webmidi, as you put that error in there.