The Case for MIDI Music in Games

[@ComputerLabHighjinks]

Hello everyone! Thanks for checking out this video on MIDI music in games. I've gotten quite a few comments taking issue with the way I discussed SNES music, so I'd like to append what I said in the video. At one point I claim that SNES games stored their music "in the SPC format", which is perhaps questionably-worded, but it's still a stretch to call this false. The .spc files you'll find on Nintendo fansites are a dump of an emulator's RAM data that captures the notes being sequenced. Those are user-generated files based on running the final, compiled release cartridge, which is obviously a different set of information than the developers would have used. However, the larger issue with my phrasing is the implication that SNES games stored their data in discrete files, which is incorrect. According to the official NOA SNES Development Manual (section 3-7-9 or page 176), note sequencing information could be written directly onto the game's code, and applied to "voices" much like the MIDI format. Developers could either work with Nintendo's own audio tools or create their own playback engine. The music isn't stored in a separate file, which is why fans have resorted to capturing RAM dumps through an emulator rather than just grabbing the information from the ROM (which you can *somewhat* do with a few N64 ROMs). As well, a few people have taken issue with my apparent dismissal of dynamic audio in modern games. There are absolutely some effective examples of devs having used rendered/flattened/bounced audio stems from a DAW and using intricate event triggers to sequence the music based on gameplay events. While this technique has led to fantastic game audio, the purpose of my video is to describe the benefits of rendering all the audio directly in a game's engine, by writing the notes into some variant of the MIDI format and using virtual instruments that run on the end-user's hardware (UE5's MIDI plugin and sine wave generator being an interesting example). It's not the way the industry currently works, but hey, that's why I made such a long-winded video full of suggestions. Anyway, thanks for checking out the video!

[@snesmocha]

Video games still mostly rely on sequenced audio to make the music to bounce it down. And the processing required to process large sample libraries is yeah please for the life of Christ don’t bring this back

[@Beatsbasteln]

super pedantic of them to say that. doesn't matter if the language is MIDI or something else or how exactly the various notation languages work. they all just make sure games can synthesize sound in cool ways and your point came across loud and clear

[@ssg-eggunner]

Wait I don't get it So few games put sequence data in the Sample RAM and others didn't?

[@Tailslol]

The thing is,midi…was never in consoles.midi use external proprietary sound database (Yamaha ,Roland …) .games in the other hand was loading those database from the game and ram. Making console music closer to tracker music… this is why console and pc music sound so different. It you want music close to the snes,the sega cd or n64 on pc you need a amiga pc,or a Gus sound card. And for ff7 the game actually uses sequenced music pretty much all the time since the game use the cd player to load backgrounds and video files. The only red book audio the game use is when it read a video and one winged angel…

[@Cr4z3d]

@@snesmocha It could still use soundfonts, which are very much in the realm of possibility for modern PCs. Either that, or Unreal style Tracker music.

[@athosworld]

I have an obsession of downloading videogame music as MIDI and playing it back and hearing each instrument separated.

[@chanotv1025]

how does one do this?👀

[@polfloe10]

@@chanotv1025 I also must know this

[@ssg-eggunner]

Oscilloscope Deconstructions are Sick

[@ebanl9531]

You're not alone my friend.

[@ThePhobosAmphitheater]

​@@chanotv1025 You can do this in any kind of MIDI-making program. Anvil Studio has worked really well for me. It allows you to mute and isolate individual tracks in a MIDI. It's easy to use and free to download!

[@eddiebreeg3885]

As a musician and game dev I have to point out a few things. We haven't stopped using sequencing in games, far from it! We use it all the time, tools similar to what UE5 now does have been around for a while, the standard begin wwise, FMOD being another popular option. They allows us to implement all sorts of events and logic to react to the game in real time, much like what midi would allow you to do. The thing to keep in mind about these wonderful VST libraries is that they're HUGE. The reason they sound so good is because they encapsulate samples for different instruments, notes, intensities, playing style and so on... which adds up to be usually in the tens or even hundreds of gigabytes. At that point it makes much more sense to combine the sequencing capabilities of the tools I just mentioned with pre recorded audio. You could achieve anything you want, it's just a matter of how far you're willing to push it really.

[@ywenp]

I'd be interested in your feedback given your experience. Are the MIDI capabilities of tools like Wwise, FMOD or Elias frequently used? Because from what I've seen by looking into them, they don't seem to fully take advantage of MIDI & live synthesis/sampling. The big advantage of MIDI is that notes can be generated or edited on the fly, depending on game events and data (which goes into algorithmic composition, and thus quite farther than a dynamic layering and sequencing of pre-rendered audio stems, which seems to be one of the main use cases of FMOD for instance). Yet what these audio middlewares do in terms of MIDI is just playback. To be fair, this already presents some value, I can notably think of two reasons: - better control over playback speed (change the BPM without having to do pitch shifting) - smoother transitions (eg. let a note release or a reverb tail flow over a transition between two clips, instead of having to crossfade) but besides that, if you are planning on only using layered "static" clips, then indeed for all the reasons you gave I don't see the point of going through the trouble of using live MIDI instead of pre-rendered audio in a game nowadays, even if that means pre-rendering the same MIDI clip several times with various instruments. (Though sample libraries' size might not be a problem for every game: not every game soundtrack needs a full-fledged realistic-sounding orchestra, a lot of recent indie games just rely on pretty SNES-y sounds anyway and still get a great mileage out of them) Am I missing something regarding the MIDI sequencing capabilities of the tools you are mentioning? (I don't know anything about UE5 for instance)

[@eddiebreeg3885]

You're right when noting middlewares don't take full advantage of what MIDI has to offer. The old games running on Atari and what not didn't either! Wwise and FMOD do allow you to create and set parameters for the events you trigger, so in a sense if would be perfectly possible to create virtual instruments and generate music dynamically, similarly to what Ocarina Of Time did, but in practice if you're looking for a fully fledged instrument that would prove difficult, for a very minor benefit. That would mean creating a full sampler or synthesizer inside your game, and MIDI wouldn't solve that problem: you'd still be left with all the work ahead of you, as MIDI doesn't have a sound engine, as opposed to the tools I mentioned. Which is perfectly fine, because as you noted no game needs a full virtual orchestra, so usually the use of stems is perfectly acceptable, and the tools at your disposal will allow you to go very far. It's worth noting that wwise does have synth plugins you can use, so procedural sound generation is totally possible. Then again, it's all a matter of what you need. And also performance, because as you can imagine your CPU is also busy running the game as a whole, and synthesizers can become very computation heavy if you're not careful, so even with the hardware we have today, using stems wouldn't be a bad idea unless you REALLY need that kind of granularity. In my experience, very few games do. Think of a game like Celeste: chip tune, 8-bit style soundtrack. Seems simple enough, but there's a lot going on in Lena Raine's music, and generating all of that on the fly would be un reasonable, not to mention useless because the game doesn't do anything music wise that requires that kind of control.

[@dominicstocker5144]

There are physically simulated instruments now that don’t take up a lot of space at all though

[@eddiebreeg3885]

What you gain in memory usage you lose in processing. If you really need a full physical modelling engine for your game and you can afford it, then that's great. Most games do not, and it would simply be a waste of performance.

[@ssg-eggunner]

What usually takes up more storage? Sequenced music split into multiple stems? Or a Sequencer Framework along with Complex Orchestra Sequence Data?

[@DE23]

5:50 You are MASSIVELY underselling the prevalence of sampling in cartridge games. Several NES games used sampling, for example Kirby's Adventure and Super Mario Bros 3 used sampled drums to aid the Noise channel's percussion, and basically every Genesis game used the DAC channel to play percussion samples

[@ComputerLabHighjinks]

You’re correct about those games, but I’m not sure this particular distinction is all that meaningful. Sampled drums and midi drums are the exact same thing, it’s just incredibly short clips of audio being stored as different notes, and then the software playing them back in a sequence. MIDI samples always have come from somewhere, unless they’re just raw square waves.

[@sam_64]

@@ComputerLabHighjinks not for consoles with poor sampling capabilities like the MegaDrive or NES. A lot of games on the MegaDrive DID use sampled drums, but an equal amount used noise, or FM drums. Also I don't like how you referred to most sequenced music as MIDI in the video lol. I get that you did that out of convenience but it annoys the crap out of me as someone interested in sequenced music.

[@SproutyPottedPlant]

Mega Drive*

[@ssg-eggunner]

I'm gonna give few extra examples There was Batman: Joker Returns, Mr Gimmick & Hebereke from Sunsoft that used DPCM for a Better and More Complex Bass And there was also Earthworm Jim & Toy Story which used Software Mixing on DAC

[@Memelord9001]

Uh there are plenty of songs on the NES that sampled more than "just drums." People added bass lines with the channel as well. There are a lot of people that also still do really cool things with samples in Famistudio, famitracker, etc. You can still do this today, and also keep things within the constraints. So while vocal hits would not be the best on NES but limitations breed innovation

[@Emolga2225]

i don't know if you knew this, but the entire Plants VS Zombies soundtrack is a sequenced tracker module. The file contains the entire soundtrack, and only takes up 2.12 MB.

[@Kuba-xw1mw]

It sounds like that on the DS version, but I'm not sure about the moblie/PC version

[@Emolga2225]

@@Kuba-xw1mw its true for the PC version! It's a file called mainmusic.mo3, the game is all sequenced music

[@RandomHomoSapiens]

@@Kuba-xw1mw It actually is a tracked module on PC, but on mobile, I'm not sure it is. I've extracted and messed around with the file on OpenMPT some time ago by myself, tho I don't remember the name of the file.

[@sam_64]

@@RandomHomoSapiens probably XM or MOD

[@Maplefoxx-vl2ew]

ogg audio format can get very small too, i can put an entire orchestra track into like a few mb and still have super good quality sound. That's what fmod renders the audio into for the Audio build files for your games.

[@amp4105]

Im a producer and im confused, everyone uses "midi" in music now, we all use ableton, fl studio, bitwig etc etc... so why are people acting like midi isnt the norm? Are they just talking about the way games USE the audio?

[@feriante777]

Note sequences is not equal to midi in all cases. Seems that the author of the video is confusing midi with mod music made with sound trackers. This is why people complain.

[@feriante777]

FL studio doesn't use midi directly using native generators like samplers and synths.

[@64_three]

I think they mean the game playing back the actual module/ project file/ .mid sequence in real time, not just as like a audio recording

[@Eichro]

Music made for games uses MIDI very often, but the games themselves only play the resulting recordings.

[@64_three]

@@Eichro also HEAVILY depends on what was playing your mid files, It could drastically different from midi card to midi card (maybe even module like an sc 55)

[@BlazonStone]

MIDI is just data. It has no inherent sound. People think "MIDI" is just "retro sound". That is wrong. MIDI can trigger samples that are as lowbit or real life realitic as you want.

[@brianshurtleff3734]

If you want another neat example of sequenced music being used for gameplay: I worked on a few games for the Nintendo DS (which still tended to use sequenced music in order to fit things onto small cartridge size limitations) and for one such project we were doing a music game, where the gameplay needed to sync perfectly to the music as every level was a song that the players were interacting with and playing along to. We had realized the music format the game used supported audio FX that the game engine itself didn't support when playing back the audio... but the game could still detect that those FX were being triggered even if they didn't actually do anything when the audio was played. So, we set up the game's code to detect those audio FX triggers to trigger stuff in the game's code instead, and thus could use the music editing tools as a level editor and literally script gameplay directly into the music files itself-- the music gameplay of the level always stayed in sync with the music because it was effectively scripted as inaudible parts of the music.

[@sabo-vf3xj]

@3:15 This is incorrect. The SNES does not store it's music in .spc format. Developers made music on the SNES by programming custom engines for audio sample playback. SPC files are just partial captures of the contents of the SNES ARAM given a container for playback on modern programs. The SNES is incapable of processing these files unless some external program or hardware decodes them back into usable data. Also, in general, pre-2000's consoles did NOT used MIDI files for music playback. As mentioned before, they stored audio engines, which communicated internally with the audio chips and data within the cartridge or disc. While it is *technically* possible to play MIDI on old systems, you need both external hardware and software to do so (sometimes not even official), as no old console counts with the necessary program or hardware to process such files internally. Not to mention, MIDI file sizes far surpassed the available memory at the times. MIDI files for gaming were mostly a PC thing, as they do have the programs and memory to process them.

[@charlesgreenberg6956]

though techically correct i feel like calling all the pre 2000s consoles sequence methods "midi" is a pretty darn good short hand.

[@goatsoup]

@@charlesgreenberg6956 if MIDIs had never been used in video game hardware, i would just default to midi but like op says- they have. I just call it 'sequenced music' or 'sequenced audio'.

[@sabo-vf3xj]

@@SianaGearz Pretty much. MIDI was indeed used using the authoring process to facilitate music sequencing with the console, but it should be noted that it always needed an external device to do so. There's a distinction between using external devices to send playback data from MIDI into the console, and the console being actually able to process MIDI data on it's own. One curious case of this would be Castlevania Chronicles for the PS1. The original release on the X68000 did support Roland MIDI modules, thanks to the PC platform it was released on. The PSX release contains no MIDI data due to the console being unable to communicate with a MIDI device (without a third-party converter), and therefore the game emulates the Roland MIDI tracks through it's own custom audio engine (along with the new redbook music).

[@boptillyouflop]

@@charlesgreenberg6956 On 8bit consoles, space and CPU cycles were too short for MIDI per-se. Games had weird custom text-base input formats, some based on BASIC music functions (MML, typically used by Japanese game companies), some based on assembly language (most British tooling). This is also true on SNES (all the music and sfx had to squeeze into 64kb on a 8bit sub-cpu), NEO GEO (lots of MML), and sometimes true on Genesis (ex: Streets of Rage is MML, but western devs often used GEMS which is MIDI-based). Obviously there's also the whole family Tracker-based music systems which comes from the Amiga but showed up in a lot of PC and GBA games, and is rather incompatible with MIDI just because it does everything differently.

[@charlesgreenberg6956]

@@boptillyouflop I’m aware of the details it’s just when your explaining a broad concept like sequenced music on a basic level it’s abit clumsy to specify what the nes is doing compared to the n64 when what matters for explanation is that the sound is being generated at run time and can thus be altered at run time Midi is a good understandable short hand for music being generated at run time even if we’re glossing over some system details

[@robertosswald5896]

You forgot to mention Amiga which almost exclusively used MOD files which combine samples and playback information. In fact it survived for a long time between the platforms, and even some more recent games used MODs.

[@Roxor128]

For a specific example, I'll nominate 2019's Ion Fury, which uses Fast Tracker 2 XM files.

[@KrulliKlikk]

I remember being a kid and finding it fascinating that when Mario mounted Yoshi, the song would have additional bongos on top. I work as a game composer today haha.

[@ayinstrumentals7731]

Just curious what kind of education did you need to get into that field?

[@Purpbatboi]

8:31 THAT'S INCORRECT! FINAL FANTASY 7 DOES USE SEQUENCED MUSIC! Tho STREAMED audio was quite popular on the PSX. The PSX could also do SEQUENCED music. FF7 use sequenced music because of the sheer amount of tracks.

[@blast_processing6577]

Not only that, but Squaresoft continued to use sequenced music into the PS2 era due to capacity concerns.

[@chestertonic]

Was about to say the same thing. I think one of the re-releases of FF7 used recordings from the original PSX release.

[@FLYNN_TAGGART]

I remember Mario Galaxy being a big deal for having a "fully orchestrated" soundtrack, only to find later that only a handful of songs are digitally streamed, let alone recorded by a real orchestra. It's crazy that you can just extract dozens of instrument samples from that game with little effort.

[@saricubra2867]

They did that to cheap production costs.

[@Heboyi]

Sequenced music, while rarer than before, is definitely still used in modern games (really only by nintendo, but that's obvious). For example, very WarioWare game has sequenced music, even the ones on the switch. This is so the speed ups can be done properly without changing the pitch of the music. These songs can range from very obviously sequenced like the Mario 64 slide minigame music from Move It, to practically unnoticeable like Penny's Theme from Get It Together. That's just fully sequenced songs though. Pretty much any time there's sound effects harmonizing with the music in modern games there is usually some sort of sequence data following behind.

[@manoelBneto]

The reason games mostly stopped using "MIDI" (aka:sequenced music) is because you need exponentially more CPU and memory resources to match the quality of a pre-recorded piece. Yes, most non-live music is authored using "MIDI", but they use humongous sound fonts which can go into gigabytes and audio filters which make the Nintendo Switch CPU cry. Ironically, music in modern games is usually far more dynamic than when it was sequenced on the fly. Things like several variations of the same track which are played in sync and transition into each other like in Hades and tracks that are cut into several pieces that can switched around seamlessly like in Doom Eternal. Also, 32-bit consoles were great at sequenced music. For example, Chrono Cross' amazing sound track is fully sequenced by the PS1 sound hardware.

[@JH-pe3ro]

This was definitely true historically: not only was it about competition for compute resources, but "consumer audio" on Windows and Android was, for a long time, exposed with some shoddy, high-latency APIs that inhibited using those platforms like a professional DAW. The industry was not investing in the "audio guy"(and studios of yesteryear usually wanted one "audio guy" doing everything). The argument for not doing "live" sequencing now has diminished because we really have a lot of hardware to play with, applying it all to graphical fidelity isn't doing what it used to, and there's more interest now in getting a low latency experience. If we imagined designing a hobby game console around cheap microcontrollers, as is being explored in the retro space, it's reasonable to think of adding a single board audio device like a Daisy Seed as the audio processor. The BOM is lower than what old sound cards used to cost, and there's no operating system getting in the way on those little MCUs.

[@manoelBneto]

@@JH-pe3ro it still feels like a waste of CPU time mixing and filtering several audio voices to produce music that could be done using a single stream, when you could be using all those channels for sound FX and environmental effects instead.

[@boptillyouflop]

@@manoelBneto Well, decoding MP3 or OGG eats as much cpu as mixing something like 32 channels of music. It's not as much a cpu-use problem as an integration problem.

[@charlesgreenberg6956]

Although I’d agree that is the main reason why midi (sequenced music) died out it is kinda hell to try to recreate even the systems from the n64 using n64 samples in wwise generally speaking wwise’s midi implementation is kinda lacking. Wwise as an audio tool is fucking awesome and I’m very grateful I’m getting into the industry now where it and fmod exist but if I was to try to do what banjo kazooie does. I COULD do it but it would be much harder then if I had the tools in Wwise to have midi play at run time The reason being is that audio files don’t gracefully end. If you have reverb baked into your track you have to bounce the score a bar at a time. And even then that limits your transitions to a bar at a time (if you cut off the reverb it clicks and sounds generally bad) Where as with midi you can cut and change and go up key and do all that because your generating your music in real time. If Wwise had a more mature midi implementation we could be doing some really cool stuff with it (considering that the inbuilt Wwise reverbs are already used on switch and arnt heavy on cpu and some games do use the basic Wwise midi implementation ) I’m sorry for the super long reply this has been my fucking life as I’ve gone through school and learnt Wwise shit bothers me abit. The rest of the program great but damn if the midi implementation isn’t abit half baked

[@charlesgreenberg6956]

@@boptillyouflop the 32 channels of music isn’t the problem. If you were using “realistic” sound packs like the strings he mentioned in the video you would be wielding super high resolution strings with multiple velocities,expressions etc you would have to have afew gigs of ram ready just to load everything. That string pack is like 40 fucking gigs aka the size of a moderate triple a title

[@PKSuperStar256]

One example of sequenced music that really is pushed to crazy extents that I really like is a lot of the level music in Wario Land 4. Each transformation (or certain statuses) Wario can go through can change up the music by slowing it down or speeding it up, raising or lowering the pitch, and/or adding varying speeds of vibrato to some of the instruments depending on what is happening to him. The impressive use of sequenced music extends beyond this too. All of the sounds and voice clips are also used in many untraditional ways. Some of Wario's voice clips sometimes pitch bend for things like laughing (which only consists one sample of him going "Ha" repeated with the pitch sliding down for each), some of them are split up to make various different phrases (Like "Here I go!", "Here we go", "Wow, Go!", etc.), and some just do something weird like giving them stutter effects (like the infamous "H-H-H-Hurry Up!"). The sound effects in general often seem somewhat random at times as they use sequences borrowing from several samples stored in the game, as well as strange uses of the original Game Boy sound generator built into the GBA's sound system. Of course, one last thing I'd like to mention is that there are a few tracks in the game that actually have lyrics, a rarity for cartridge-based games (the only other I can remember is Klonoa Heroes's main theme, Sign of Hero). The title theme uses various stock vocals from CD libraries, while the theme for Palm Tree Paradise (a.k.a. Medamayaki) and the English and Japanese versions of the ending theme have completely original lyrics made for them. For Medamayaki, almost every syllable was chopped up and compressed to fit into the game, so they could play in a sequence of the music file. However, for the latter two, they still play in sequences, but the lyrics are divided into short phrases instead. There's also an English-only reprise of the ending theme that seems to only play in the bad ending when you don't any treasures after beating all the bosses, but it's also divided into phrases. Either way, sequenced music is great for the adaptivity of it as well as the simple fun of seeing what you can do with it. I've been fascinated with both that and streamed music just to see what was used to create each and it's nice to know what they are.

[@baggagelizard]

Just from barely researching, I found that Final Fantasy VII on PS1 is not audio files, it is indeed sequenced, but it uses a different standard from midi, AKAO. There is a lot of misinformation in this video, I don't know what your sources are, I would like to see them. No one ever really stopped using midi/tracked/sequenced audio, though many games use dynamic audio and sequencing that isn't the midi standard. Midi 1.0 has lots of limitations such as only allowing 16 channels, 7 bit values (0-127) for all parameters. There's lots of other misinformation in this video. There is so much fascinating information out there about sequenced and dynamic audio, but very little of it is in this video. I fear that if the algorithm takes hold of this video it may in fact cause more people to be less-informed of what you're trying to vouch for in this video. I would suggest you look into the MIDI 2.0 Standard and Wwise by audiokinetic.

[@charlesgreenberg6956]

Thank you so fucking much I thought I was going insane with that ff7 thing Also fmod is cool too :) that’s what pizza tower uses along with a lot of cool indie games!

[@tuc5987]

MIDI 2.0 is brand new and barely used anywhere yet, no need for anyone to look into that for such a video.

[@simki1531]

good chance his sources are AI since a lot of the stuff he mentions is extremely wrong

[@ywenp]

@baggagelizard He does warn right at the beginning of the video that he is using the term MIDI loosely for "sequenced music tracks in the broader sense". When he says "MIDI should be back in games" he is actually saying "more games should start again to rely on sequenced, live synthesized/sampled audio rather than pre-rendered audio", the former having indeed undoubtedly declined a lot since the N64 era. Also even if current tools like FMOD can deal with MIDI data and do live sampling via soundfonts, it doesn't mean that a lot of games actually use these features (virtually no FMOD tutorial here on youtube touches on them) > I would suggest you look into the MIDI 2.0 Standard and Wwise by audiokinetic. I don't think Wwise supports MIDI 2.0 (very few software/hardware support MIDI 2.0). Also most of MIDI 2.0 features make little sense in the context of a game, they are about easier interconnection between different hardware devices. > Midi 1.0 has lots of limitations such as only allowing 16 channels, 7 bit values (0-127) for all parameters. Higher resolutions for MIDI parameters have been around as MIDI 1.0 extensions for a while (NRPN, 14bit CCs, etc). The 16 channel limitation is irrelevant in the context of modern MIDI software which don't use one single MIDI bus (eg. in any DAW, each track is its own MIDI bus, which means you actually have 16 channels _per instrument_). MIDI limitations are not really a problem, but again MIDI itself is not really the point anyway.

[@blast_processing6577]

The PC port of FF7 from 1998 used MIDI files and a SoundFont so I'd be willing to bet the AKAO sequence format used by Square on PS1 games was probably similar enough for them to convert between the two formats with ease.

[@goatsoup]

"MIDI"/Sequenced audio is still very much part of game audio to this day, it's just now a much smaller part of the pie because it's not really needed as much as it was back then. Game audio has evolved to sport so many different kinds of ways to do dynamic audio. Also, your idea of the weaknesses of streamed audio would've made sense in the 1990's. Back then, very early CD-ROM hardware was very slow, even some cases where looping a track wasn't even feasible (most notable PC ENGINE/TG16 comes to mind), things like increasing tempo, dynamic layers, anything like that was just not possible (or not easy at least) then. But now? Streamed audio is just as malleable, if even more than sequenced music ever was, and game developers are 100% taking advantage of the processing power of newer hardware with sound. In a sense, games are still at times basically sequenced! just now with dynamic layers, sections, with realtime effects (which you touched upon) and much more .. which unfortunately, doesn't get enough love/coverage, it's really a shame wwise didn't make it in this video despite being such a popular middleware tool, that *can* do sequenced music and tons of dynamic game audio stuff and there's a ton of examples of games that use it- I think that's the 'new tool' you were kind of yearning for but weren't aware existed ^^ I think it's really unfortunate that you've phrased that musicians aren't open to adopting new tools when with wwise (and other audio middleware), I can think of countless musicians and people working in game audio that have been at the forefront, been interviewed, done GDCs showcasing some really adaptive, dynamic, and just cool. audio tech to audiences, even some still using MIDI/sequenced audio as well! Also a lot of those examples you brought for MIDI are.. still done in games today! Mario Odyssey has internal sequence data for every major area song to make sure sound effects harmonize with the music/helps the game keep track of that, and then obviously the town tune is still in newer AC games. New Super Mario Bros. on Wii/3DS/Switch also stores game sequences in the files to make sure the game responds to the "bah" sounds. Modern game audio tools can easily fade songs in and out, speed up music, even internal midi/sequence-based system working under a streamed piece of audio would totally work for trying to do mother 3's audio too! Yooka Laylee, the spiritual successor to banjo kazooie could also achieve the effect in BK without needing midi at all! All of that is the beauty of modern game audio, we can achieve the same things as these old games, but without sacrificing audio quality or being too costly on the game! So to put it again- Game audio today isn't just midi or streamed- it's both, and so much more. Game audio is a lot more complex than it was 25 years ago and we have so many more options and tools now, sequenced audio is just one of them! ^^ Minor notes- MIDI, the protocol, not sequenced music, was made for musicians/music hardware so it is inherently always been from its inception, tied to music production and is very different from sequenced game audio. It's not 'quietly used' like your description says haha And trying to do sample library playback in a video game is just incredibly unfeasible and expensive without optimization, speaking from experience and most notoriously associated with orchestral libraries! I appreciate you flashing hollywood orchestra there but I suggest you check the "system requirements" on that page.. and that's just for PLAYING the music in a DAW, not considering an entire game running beside it! In general- The MIDI Protocol, sample libraries and such were made and optimized for a music producer/musician's use. Game audio, both in 1990, and today, are optimized well- for a video game!

[@ssg-eggunner]

Has anyone ever thought of attempting to convert the sequence data in mario odyssey and convert it to a module file or something

[@goatsoup]

​@@ssg-eggunner im unsure, but the sequence data from what i heard wouldn't be of like the whole song, it'd just have basically like the chords/the necessary stuff needed.

[@marydumais9251]

As someone who makes music.. MIDI hasn't gone anywhere.

[@andywest5773]

Your choice of SPC files as a comparison with MIDI is an unfortunate one. They do not operate on the same principal at all. MIDI files contain note sequence data. SPC files are actually RAM dumps of code and data that run on the Sony SPC700 chip. They're so different that accurate SPC-to-MIDI conversion is practically impossible (although I suppose with some type of AI technique you might be able to do it).

[@PlasticCogLiquid]

It's totally possible. I used to convert SPC files to Impulse Tracker format so I could rip the instruments from them with the loop points intact. This was back around 2005.

[@Guacamole1000]

alright Sheldon Cooper

[@techguy348]

@@Guacamole1000 bazinga

[@suitandtieguy]

​@@PlasticCogLiquid that's awesome. What tool did you use?

[@atp19xx]

@@suitandtieguy Can't say what tool he used, but if I ever needed notation or just midi conversions of spc-files, I used spctool

[@starerik]

Super Mario Galaxy is proof that it’s very much possible to sync MIDI to audio files. Everytime you use a launch star a harp arpeggio plays, and it’s always harmonizing with the current chord that’s playing. Same goes for those blue-colored platforms that you have to turn yellow, the sound is always in-tune. And that was in 2007. Just wish more developers made that same effort.

[@clydesapere1977]

Fantastic example!! Gamecube - Wii era had such great event / timing unique audio. My personal favorite example would be Twilight Princess triggering different parts of the boss music depending on battle phases and advantageous moments. And the orchestration on this and Mario Galaxy sound amazing.

[@asherburdick6319]

These techniques are used in modern games all the time, often using audio middleware like FMOD or WWise. For a modern example, if you listen to the sound that plays in Mario Odyssey when you travel through a wire, you'll hear the arpeggio sound changes with the chords being played in the soundtrack. These are really fun things to do with dynamic music, but midi is not particularly necessary for any of them (other than maybe the cute band in Conker's Bad Fur Day.) I think it's hard in most cases to justify the performance cost of using modern software instruments over just rendering out the parts in layers and manipulating those. (Edit: didn't see the pinned comment. I still think the performance cost is hard to justify unless you have something you really just can't achieve without building software instruments into the game)

[@charlesgreenberg6956]

Fmod mentioned

[@saricubra2867]

"re-adopt MIDI technology" The problem isn't MIDI itself, it's the sound generation. Realistic orchestral sounds take hundreds of GB of space, you need an absurd amount of RAM for the playback for an orchestral piece, if the music uses synthesizers, the bottleneck would be the CPU. The majority of people don't have beefy music production PCs, trackers existed because of technical limitations. Even now, using MIDI for games in general is just impractical.

[@Pirateyware]

Good video! Though I wanted to note, 11:57 it's not that complicated. Going underwater in Banjo-Kazooie simply switches to a harp arrangement of the current level theme. Nothing fancy has been done to it, the harp sample just sounds that way. Also, I'm pretty sure the Xbox version still uses MIDI, and is even incorrectly emulated, most notably any time the theremin instrument appears (e.g. the beginning of Mad Monster Mansion) completely lacking the vibrato it had in the N64 version. The organ always playing the same note is just a bug.

[@idadood2278]

The case for midi files is Bug Fables' ost. That's the entire case. Thats all it needs.

[@BlackTomorrowMusic]

I'm not much of a gamer, but as a musician, I'd love to see MIDI implemented back into games. I remember playing Doom 2 once when I accidentally left one of my keyboards plugged in and turned on. The game automatically sent the music to the instrument, and the whole experience was transformed.

[@minebrandon95264]

Wait, how come Trackers/Modules never got mentioned once?

[@2K8Si]

I was thinking the same thing... 🤔

[@tommj4365]

This is common for folks who overuse the term MIDI

[@j7ndominica051]

The people who feel nostalgic about old games today seem to be mostly form America and played with consoles instead of personal computers.

[@mdjey2]

@@tommj4365 So don't some trackers use MIDI? What does Renoise uses then? What does Polyend tracker uses?

[@tommj4365]

@@mdjey2 some modern trackers may use midi, as an optional feature, but typically trackers don't rely on midi for anything

[@BADC0FFEE]

this video is so full of inaccuracies that it's actually upsetting

[@AbAb-th5qe]

This video mixes up midi with chiptunes on the nes. Chiptunes are tied to a particular sound chip and store instructions for it directly. Nes chiptunes always sound like that. Midi is instructions for generic instruments that can sound different based on the soundfont in use and aren't subject to the limitations of something like a square wave generator.

[@Kaytsey]

8:13 the FF7 soundtrack is entirely sequenced on PS1 too, actually.

[@cherrystarscollide]

None of these old consoles used the MIDI protocol.The MIDI protocol was used (and is still used) for musical instruments and sometimes on PC, but NEVER on consoles, which relied on either sample based (SNES/PS1) or synth based (NES/Genesis) sequenced music.

[@mdjey2]

Both sample based and synth music is composed by MIDI.

[@cherrystarscollide]

@@mdjey2 no, you are confusing sequencing with a protocol :) MIDI is a specific protocol and not used in old consoles. :)

[@mdjey2]

@@cherrystarscollide No I am not confusing. You are making argument that whole music is just composed for video games, which is not. Midi is the standard for sequencing synthesiser and sample music.

[@DoomKid]

@@mdjey2 MIDI is a specific form of sequenced music, which is indeed not used by things like the SNES or NES.. However, converting sequencer data from one format to another is usually trivial. The uploader used MIDI as shorthand for all sequenced music, which is incorrect, but a fairly common colloquialism too..

[@cemstrumental]

That's why 3:02...

[@MacUser2-il2cx]

Midi is also used in RPG Maker games, DOOM Wads, SRB2, and other fanmade games.

[@willia_music]

Wwise (one of the most popular middleware for music composers working with game engines) makes it easy to work with midi. Hoping that indie devs take advantage of this too

[@BADC0FFEE]

using a NES game, that uses chip music, as an example of midi is really strange, would have worked better with a PC DOS game that actually used midi...

[@alexgrunde6682]

There is one problem I can see with the pure MIDI based implementation with VSTs. While there are plenty of good free ones out there, and the ones built into some game engines, most of them have a license fee attached to them. You want to use the VST to make music in your DAW, you have to pay. Problem is, if you want to have that VST built into the game file, you’d have to pay for a copy of the license for every copy of the game sold. And it’s pretty self-evident how infeasible that is. So if the composer has a particular VST they like to use, then they have to record it to audio.

[@goatsoup]

I know of a somewhat modern game that has dynamic audio, midi sequenced stuff, and streamed tunes for space/storage reasons that actually use their own studio recorded sample library the audio director mentioned licensing issues using samples from say east west or something by having raw samples from those companies in a game. the game has live music tracks, so they were able to get the band to record samples for midi sequencing, and it managed to work out actually pretty well.

[@alexgrunde6682]

@@goatsoup Yeah that’s probably the best approach, sample the professional VSTs or live musicians and then use the in-engine midi sequencer to dynamically play back the samples.

[@goatsoup]

@alexgrunde6682 i mean in those cases that was done for memory/storage limitations, i imagine most games would just stream it.

[@jeremyseay]

The CPU power and storage needed is way beyond what we would need to play a simple MP3 file now. Yes, sample libraries like East West can sound convincing, but these complex DAWs that are used to create this music require beefy machines with huge amounts of RAM and fast storage. The environmental examples like you see with Banjo Kazooie can easily be accomplished with audio crossfading, without needing to rely on MIDI.

[@saricubra2867]

Pianoteq synthesizer sounds exactly the same as a real piano and doesn't reach 100MB in size, but you need an UFO CPU to run it. I basically don't use samplers, my CPU can generate a realistic synthesized orchestra, it's a luxury that a lot don't have.

[@MisterMunkki]

Yeah about every game today has complex systems to make the music follow dynamically what's happening, it really has nothing to do with midi at all

[@AmaroqStarwind]

I prefer tracker music, such as what Unreal and Deus Ex used. The composer gets to choose their own instruments, and the listener will always hear *those* instruments regardless of what sound device they have.

[@codahighland]

That's how all sequenced music works. It's not just trackers.

[@AmaroqStarwind]

@@codahighland Technically, MIDI is sequenced...

[@codahighland]

@@AmaroqStarwind Yes, exactly, that's my point.

[@codahighland]

@@AmaroqStarwind To be more specific, I suppose I should have said that you CAN do that with any sequenced music, and I should have clarified that the target hardware defines what instruments the composer can choose (e.g. the NES's channels). It was really only a fairly narrow part of history where people were playing back MIDI music on arbitrary hardware without composer-supplied samples.

[@GreenRequiem]

I've been excited to watch this for a while--I thought this was a really nice breakdown of MIDI's history and uses in gaming!! One aspect of this "the case for MIDI" video that I expected to be brought up was how MIDI soundtracks in old games sort of naturally made the soundscape very **cohesive** in a way that I think many modern game soundtracks can sometimes struggle with. Nowadays, we can find games all over the place with dozens of musicians and literally hundreds of tracks for a game. As nice as these soundtracks *sound* in the moment, I think that the player hearing many different composers who implement tracks with 6 different string libraries / 4 different flute libraries / 12 different piano libraries etc etc can make the "sound" of the game hard to remember in a way that old MIDI soundtracks with 1 piano source, 1 string source, 1 flute source didn't. This is super subjective ofc, but this is one reason why I'm a big fan of the older MIDI-based console soundsource music: it can help to make a multi-composer game sound more unified^

[@SoundFontGuy]

Hey I love your video, but I just want you to know that the original PSX release of Final Fantasy 7 does in fact use a soundfont (not the technical term, but you understand). The entirety of the Final Fantasy 7 OST on PlayStation is under 1MB! That said, fantastic video! I hope that game developers start using MIDI again!

[@ComputerLabHighjinks]

Thanks for the comment! I may have to look more into the FF7 situation, perhaps I'll do some sort of follow-up video. The nature of One Winged Angel's vocals as apparently just being one extremely long midi note are quite interesting. Also, I love your series on OOT with the Chrono Cross soundfont! Really great stuff.

[@SoundFontGuy]

@@ComputerLabHighjinks Thanks for the compliment, and the thoughtful response! Yes, the vocals in One Winged Angel are just a few samples triggered by midi notes! I realize people are correcting you a lot in the comments, sorry to be a contributor to that bombardment. I'd love to see a follow-up video, Keep doing what you're doing!

[@MidoriMizuno]

You have implied that "MIDI once sounded bad and now sounds good due to high quality VSTs". Well, that's pretty much a half-truth and either deliberately omitting a lot of important information or not being aware about it. MIDI isn't and never was a protocol solely used within software environments on general-purpose computers and game consoles. It is completely device-agnostic and initiallly it was developed to connect together sequencers/midi controlling keyboards with physical hardware - a lot of professional music synths from the 80s and the 90s had MIDI in/out support and a lot of music from that time had been sequenced using MIDI. Atari ST computers used in professional studios, controlling a wall of rack-mount pro-grade synths weren't uncommon. A lot of mainstream, 80s and 90s music used this technology already. For example Michael Cretu, the composer of songs performed by Sandra programmed a lot of his compositions using an Atari ST computer.

[@Tailslol]

The thing is,midi…was never in consoles.midi use external proprietary sound database (Yamaha ,Roland …) .games in the other hand was loading those database from the game and ram. Making console music closer to tracker music… this is why console and pc music sound so different. It you want music close to the snes,the sega cd or n64 on pc you need a amiga pc,or a Gus sound card. And for ff7 the game actually uses sequenced music pretty much all the time since the game use the cd player to load backgrounds and video files. The only red book audio the game use is when it read a video and one winged angel…

[@EpicureMammon]

Lucasarts iMuse system is a great argument for MIDI. I was spacing off at work the other day and thinking about how well it worked in TIE Fighter to change music smoothly and dynamically with what was going on in the game (even down to a friendly ship being destroyed). Also, I love the FFVII PC score for Yamaha XG. I have to admit that I amassed a small arsenal of Yamaha MIDI devices just to hear the music across different levels of hardware. MIDI was just fun on PCs! Before the early days of 3D cards, it was the one thing that could make a game seem very different from computer to computer.

[@ColonelMidi]

monkey island 2's woodtick theme system is still one of my favourite pieces of dynamic music. these people really layed the ground for the future. BUT systems like these are perfectly possible on mainstream audio solutions without MIDI or MIDI-like interfaces. the limiting factor is not the tech or reluctancy to use other tools, but money, communications and the low priority soundtrack often gets inside productions.

[@mikosoft]

iMUSE was used to great effect in all SCUMM LucasArts adventure games where it created an incredible cinematic feeling. Especially Indiana Jones and the Fate of Atlantis with its very atmospheric soundtrack that seamlessly blends musical cues one to another. People rank Monkey Island as the greatest Lucas adventure game, I strongly disagree, I find Fate of Atlantis has much better writing and has a much more epic feel.

[@Alzter0]

It's not MIDI music - it's sequenced music.

[@NaraSherko]

Hey I remember you in scratch

[@kiwirocket64]

You know, I gotta say one of my favorite parts of the midi music is just the fact that I can take my favorite songs and I can just pull them apart. I can listen to different parts of the song separately and I think it’s really fun to do.

[@jexboxx]

this channel is a hidden gem of videos, im gonna binge them all.

[@Maplefoxx-vl2ew]

hi there, we still make music in midi format and then put it in games after being rendered to audio.. it's still midi triggering samples here on our end , i'm a game composer lolol so technically even new games are using "midi music" it's just not pulling sounds out of a chip in a cartridge, it's rendered to audio. put into fmod or wwise then turned into a build file which works with the code of your game, there is no way we are going back to the old way.. i mean you can if your a hobbyist i guess. It would be very pointless. I can still have the same sounds from those chips on my computer right now. In fact i do have them. Impact Soundworks sells Super Audio Cart and also Insidious, the C64 one. I even have a Gameboy sampled into sofware from another company.

[@SoundSnakeStudio]

Inspiring video, great examples of how midi was superior in many ways, that unreal plugin looks very useful! Subscribed!

[@xeode]

was always impressed with how all the diddy kong racing zones had their theme music crossfade as you went around and was basically a differently instrumented version of the same song with different stylistic bits added in. was always shocked at how well it was done and would end up getting sidetracked by messing with 'playing it' like an instrument lol

[@charlesgreenberg6956]

I really loved this video! Im currently majoring in video game music and it honestly really surprises me how little midi implementation there is in audio middleware. wwise has it but there's a lot it cannot do (changing tempo being a huge one) that said ALOT of what you talked about with midi here can be and IS done with audio in games with wwise. The underwater example is super doable!!! you would have both versions of the song playing at the same time. the underwater version at -96 (basically no sound) at the above at 0 (standard volume) and then set all the water to be a trigger to fade out the above water and fade in the bellow. you can even do things withlow pass filters, reverb etc. Ive even seen people do things like that scarecrow song with audio simply by essentially making a sampler out of triggers (though midi would and should be used there) Theres ATON of rly cool interactive music things still happening and its really really cool! I think my favorite thing right now is how some games will change melody and stong structure on a dime depending on whats going on. I know vampyr does it with its main investigation theme.

[@user-ft4jo8ev1v]

Man, I really have a hard time describing how much I love this channel! Your videos hit that perfect nostalgic spot, while still keeping everything relevant. This is my new favorite "comfort" viewing channel and I love it. Greetings from Sweden!

[@ajpink5880]

If I'm not wrong, the GameCube, Wii, and Wii U all still used midi-type stuff for some games, so at least for Nintendo, it depends on the game but they're still doing it (idk about switch, don't own one myself)

[@starerik]

Yeah, all first party Nintendo games (Intelligent Systems excluded) used MIDI for the in-game music on the GameCube. Some Wii games as well, and of course GBA, DS and 3DS (the system stuff, Mii Plaza, Mii games etc.)

[@CheesecakeMilitia]

One great example of the advantages and disadvantages of MIDI implementation is Guitar Hero vs Rock Band animations. Guitar Hero 3 and onwards tended to use a mix of motion capture and manual animation on its character models when they're performing songs. The pinnacle of which is Bohemian Rhapsody in GH Warriors of Rock - which beautifully recreates Queen's original music video with handcrafted staging and lighting angles. And I remember reading that one song took over a month of dev time - Guitar Hero songs were incredibly laborious to produce. And that results in a lot of shortcuts elsewhere - the camera hardly ever zooms in on what the guitar player is actually playing (because it's not that accurate), and you'll see certain canned animations like Slash dancing in a circle multiple times. And if you export a song to another game or get DLC, it's likely that song will have gimped animations due to filesize limits. Rock Band by contrast controls almost all animations with a midi track - everything from where a guitarist is fretting to venue lighting and camera work. And the end result can certainly look janky or robotic at times, but the quality is way more consistent and Harmonix had a cheaper and easier time publishing DLC tracks - to the point where over 4000 songs were officially released in Rock Band's lifespan compared to several hundred Guitar Hero songs produced by Neversoft. Anyone who's delved into editing rhythm game tracks can tell you how much easier a MIDI standard is to work with than some awful custom file format solution. Beat Saber of all games uses freaking JSON text files to describe note positions, which is hell if you want to map a song with any BPM changes (which is pretty much any song that's played by real musicians not using a click track) or if your song goes longer than 10 minutes (since the note positions use floating point values, which become less accurate the further you get from the origin). It's so painful seeing devs reinvent the wheel when the MIDI standard has existed since 1983 and is STILL on v1.0 because it's so well designed.

[@SpringySpring04]

One of the reasons I really like Touhou music is because the creator, ZUN, often releases each soundtrack separately as MIDI files. In the early Windows generation of Touhou games, between Touhou 6 and Touhou 9, there was literally just MIDI files sitting in the game's folder. I've always loved that ZUN did that, because it makes things easier if we want to analyze the soundtrack by opening the MIDI in a DAW. Not only that, but ZUN also released the soundtrack from the PC98 games (Touhou 1 to Touhou 5), which were originally composed with only classic FM synthesis due to the limitations of the PC9800 computers, but he released these songs in a Windows-era styled instrumentation AND in MIDI form. Touhou is great for MIDI music! I'm not sure if it's using MIDI, but there is a rhythm and action focused game called Hi-Fi Rush that, like Mother 3's combat system, requires you to do combat to the beat of the song, though it's a bit more complex. You can do combos by combining different 1/4 notes, 1/2 notes, and 1/4 rests. It's also just one of the most fresh gaming experiences I've seen in recent years, with a good story, good comedy and good music. I highly recommend!

[@chrisfratz]

There's two "evolutions" to MIDI that I really find fascinating. Tracker music which started on the Amiga, and Microsoft direct music which was an extension to DirectX allowing developers to sequence music with their own custom samples. Or in the case of Tron 2.0 (a game which as far as I can tell does in fact use direct music), it serves as the backbone for its adaptive music feature. That one is a doozy of a case where each track in the game is split into several smaller audio tracks that get played back together in a sequence to form the whole song. The reason they do that is so that when you say get into a fight the game can easily stop the music and then switch into the combat music, and vice versa, once you're out of comment it'll switch back to the ambient track. Now it went in game, it's Great. I love that each level has an ambient and combat variation of the music, the pain comes from the fact that the game soundtrack was never released officially, and the only recording of it on KZbin sounds really bad where the first set of tracks are only in mono for some reason, and then the rest have some really bad peeking in the recording which makes me think somebody just used the audio output on their PC into a recording device and didn't set the levels properly, so I would just prefer to go straight to the source and extract the music from the game which is possible. It's just that assembling each track by hand is tedious because I can't just Play some on a DAW timeline right next to each other because each track has an extra fade out part that gets played and it's baked into the audio. So I need to time it properly by ear. I'm going to stop ranting about the music to Tron 2.0 now. Tracker music started on the Amiga (Like I said earlier) and can sound amazing. Basically the Amiga had four audio channels but they could play back digital sound. But then trackers evolved when moving over to the PC and in my opinion culminated with games like Unreal, unreal tournament 99, and Deus Ex. All of which used tracker music using the UMX format because they were all games that were built with the original Unreal Engine.

[@chrisfratz]

I will say I've tried piecing together a couple tracks from Tron 2.0 and the results are amazing. But I don't know if I could do the whole soundtrack just because of how much music there is. In fact the only two tracks I really did were the menu theme, and the ambient track to the first level.

[@chrisfratz]

Also if you want another example of something like what Conker's Bad Fur Day does in the menu (sequencing animation to the music data), check out Animusic. It may not be video game related, but it's an amazing CGI video that was released on DVD and VHS in 2001 if I remember correctly. It also got a sequel in 2006 and it's also really good. People have uploaded the whole thing on KZbin, and I recommend watching it.

[@graysonmusic.]

I really liked unreal tournament as a kid. cool comment

[@ComputerLabHighjinks]

That's really interesting! Making a soundtrack from a bunch of shorter audio clips arranged in a row is a solid option, I'd imagine it just takes a lot of work to accomplish (AFAIK Breath of the Wild was done this way? Solid music cues in that game). Tron 2.0 has been on my list for a while, I love the movies and moderately enjoyed the Evolution game on 360/PS3, as well as the Kingdom Hearts world, so I'm always down for more Tron gaming content. I had no idea the early Unreal Engine games used a variant of MIDI, or that DirectX included that kind of feature set. I'll have to look at this for future videos! I'm hoping to do some videos on UT99 since that game and its custom level community was so robust.

[@chrisfratz]

@@ComputerLabHighjinks tracker music is its own beast but it's still rock the spirit of sequenced music, it's just that the person making the music can use whatever samples they want for the sound. Leading to a lot of Amiga music having a very techno and club style sound. Especially in the demo scene.

[@mantalayer]

I've been a composer, sound designer, and tech sound designer for over 10 years. Most stuff here is absolutely spot on, but meta sounds in UE just isn't ready for most audio applications yet. Wwise, for all of its draw backs, is still the gold standard. It even allows you to set up MIDI and make the music dynamic, and combine MIDI with normal audio playback. Check out the GDC talks for Peggle 2 from Becky and Guy.

[@ProdChunkkz]

i’m a multi genre music producer currently learning game composing who recently got in contact with a game dev to do work and i mostly use just midi for almost all my tracks. i love the creative simplicity of it plus i work with orchestra a lot and i can not afford to hire an orchestra to play my music for me. then u have my beats which is also always midi. my electronic music. always midi. it’s only my guitar i use for some rock that’s not midi

[@SampelMSM]

A modern game that uses midis is My singing Monsters Itll have a long midi track just for a monster's idle and playing anim, If theres no note being pressed, it'll do the idle, if theres a note being pressed, itll do the playing anim

[@robin_redacted]

the things you claimed being only possible with sequenced music are still done with streamed music today. mario galaxy had a different mix for underwater (beach bowl galaxy) and botw whole soundtrack is all streamed and insanely responsive in such an immersive way, people don’t even realize. hi fi rush has the combat system intertwined with the streamed music to a super high degree. there are cases for sequenced music in games - but these were not them. having a game handle on the fly key changes, a rubato tempo that subtly shifts with the player with no loss of sample rate are both things i’d be super interesting to see. i’m fairly certain mario galaxy’s monkey ball-esque levels actually use sequenced music just so they could get that super dynamic tempo that responds to player speed.

[@robin_redacted]

the other thing that would be fun to see on pc specifically is having you game composed in standard midi and allow for users to either change the soundfont being used, or the ability to use external midi devices like the roland MT and SC-55

[@jamnnjelly3102]

Agreed, I wish more games used it. WarioWare is a franchise that always uses sequenced music due to the nature of the microgames and how they speed up

[@Domarius64]

Banjo Kazooie (and tooie, which introduced tempo transitions) is probably the best argument for "MIDI" (or tracker music) in games, and probably the best example of it, its certainly my fav example. Also its no mystery FF just recorded their MIDI to the disc, its dev time, it would've been quicker to do that than make a custom midi playback for their game if they didnt already have one from previous games in the same studio. The extreme example of this would become COD being 200gb, just wasting space to save dev time by avoiding space optimisation.

[@tkc1129]

I agree. Anothee good example of MIDI tech in games was iMUSE, which built up songs dynamically out of smaller elements. There is a new MIDI 2.0 standard that allows for things like vibrato, but as typical, it is only used by music producers, not games. The sound libraries required to make MIDI sound good are stipidly expensive. For MIDI to be used in games nowadays, there would need to be modern, standardized libraries that come free with platforms like consoles or Windows. I think more simulation of notes could be helpful in bringing cost and memory usage down.

[@SandmanDP]

This video needed to be 10% about MIDI and 90% about tracker audio.

[@ComputerLabHighjinks]

Hoping to make a follow-up video at some point! Although, in fairness, I feel like the title matches the subject. Instrument stems and shorter clips have tremendous utility, but this video is more focused on generating audio from within a game’s engine.

[@blast_processing6577]

​​@@ComputerLabHighjinks : "...this video is more focused on generating audio from a game's engine." Wavetables, transwaves, and some other technologies used in professional instruments are essentially specialized samplers, so if you want to "generate audio from a game's engine" you could absolutely get a lot more out of music trackers provided certain features were implemented. *Edit:* Also, music trackers like Deflemask demonstrate that even when sticking to traditional feature sets there's still a lot that can be done with a music tracker.

[@lumiere_eleve]

8:15 Music of Final Fantasy 7 was composed with Yamaha's XG MIDI standard in mind that was incompatible with game consoles. If the game music in the console sounds almost the same as what you would hear from an XG synth, that's exactly why. XG not only offers more instruments (melodic voices or sound effects) and more drum kits than General MIDI, but also offers audio effects (reverb, chrous, delay...) that are simply non-existent in General MIDI (revision one). (I know this because I develop something that's compatible with multiple MIDI standards XD)

[@AstronautLoveTriangle]

Sunvox is a music tracker with a powerful synthesis/audio engine, and there is a library available to use its sound engine in other software. I'm not a programmer, but I THINK it can be used to make audio/music in a game that reacts dynamically to in-game events.

[@Halfbit_0]

Depends on whether their playback engine is either open source/okay legally to use in your game and gives access to individual tracks, note data etc. I feel like the easiest way would be to use midi files and samples/soundfonts and maybe write or use someone's implementations of the most common effects like reverb and delay. I'm only familiar with Milkytracker, but trackers are more limited in the granularity of note lengths and seems like a pain if you want to work with odd time signatures.

[@Starlit-Music]

If "MIDI" does become a big thing in games again I really hope it's in the form of more modern standards, such as MPE, or even OSC. I also find the idea of a game having it's own built in version of something like Vital as a modern version of sound fonts a bit amusing.

[@MustacheMerlin]

So. You can kinda just do all of these with pre-recorded audio perfectly well. The main advantage of midi remains file size... unless you're using those super realistic VSTs you were talking about where the digital instrument is 1TB of samples. For one, you can totally just provide the sheet music in a midi like format alongside the audio track to get all those "game responds to the beat" party tricks you were showing off, without having to actually synthesize the music on the fly. Half the time you don't even need that, a "this is the tempo" annotation would be plenty for a lot of effects. The banjo kazooie thing where the audio swaps out instruments underwater is also very simple to do and even common in modern games - you just record two audio files and switch which one you're playing. A great example of that is the way Nier Automata seamlessly swaps to a chiptune rendition of the soundtrack whenever you start the hacking minigame. It's even simpler if all you want is the "low pass filter" sound - now we can just have the game run the audio through an actual low pass filter. The hand pushing harmonic piano keys is a little tricky, but still doable. Again you can provide a midi file as metadata alongside pre-recorded audio. Or you could annotate the track with key/chord information. A very similar problem that I've seen solved multiple times (Gris, Transistor, It Takes Two) has a much, much simpler solution - I'm talking about making a 'press X to hum' button. You want the character to hum and harmonize with the soundtrack playing when the player presses a button. Gris just has you hum a little chunk of sound that does the music theory to fit in with any part of the soundtrack in the area you can use it. It Takes Two tried to be super fancy and make a crazy modular synth thing using midi data to choose a note in the right key and chord based on what was playing... until they scrapped that and ultimately used the solution I'm about to talk about. Transistor (and It Takes Two in the version they shipped) made the hum button super easily. They want the character to hum and harmonize with the current soundtrack. So... they just recorded someone humming along to the whole track, and brought that in as a separate audio track. When you press the hum button, the humming layer of whatever song is playing fades in. Sooo. With that in mind, how would I actually make the piano effect you showed off? I'd compose a piano layer into my soundtrack. And all it would be is a little annotation saying "Play this note at this part", no fancy music theory aware programming, just check the midi file for the current note and play that. So the piano itself would be kinda a super simplified pseudo midi, but the music track itself would probably still just be a normal audio file. If you wanna make an Animal Well type game and stuff it all into 30 MB then I'd seriously consider midi. But otherwise, there's actually not that playing midi files on the fly offers that I can't easily do with regular audio, and less CPU power besides.

[@Ikatxu]

The problem with using MIDI with modern virtual instruments in video games is that they take up a lot of processing power and RAM. I produce music using a gaming laptop and there is a point in every project where I will have to start freezing tracks to audio in order for the computer to handle it. As video games are generally also quite heavy developers won't want to use the majority of available cpu on background music. Then there is also the issue of disk space. While midi itself does not take up a lot of disk space, some virtual instruments do. For example Hollywood Strings mentioned in this video takes up around 300GB of disk space alone(and you'd probably want more instruments in the game than just strings). So while these amazing virtual instruments exist, in realtity a video game would have to use significantly less amazing ones, and the quality of music would suffer as a result

[@EuphoricPentagram]

Midi and programable music as a whole is something very intriguing And it’s something im definitely planning on using in my next game, like I could change keys based on biome, have it play slower at night And use different instruments to mimic sounds of the world, like string for wind, or tambourines for magic

[@charlesgreenberg6956]

Look into fmod :) might be something you can use for that

[@Roxor128]

Might want to look into tracker modules for your music formats (there are quite a few of them). Many of them can have sub-songs which the game can jump between. Unreal does it, as does Deus Ex (both use the same engine, so it's not really surprising). Unreal tends to have a quiet sub-song, an action one, and one or two transition bits. Deus Ex seems to typically have four or five in most of its music files. Apogee's Stargunner (1996) has 4 or 5 sub-songs in its level music tracks. One for the main level, two for different stages of the boss, and a "level complete" one seems to be pretty typical.

[@TreR90]

Rick Astley will be on repeat in my head tonight

[@GalekC]

the file size segment's bgm is reminding me so much of the "Thank You" ED from bleach now i can't get that song out of my head lmao

[@ThePhobosAmphitheater]

This is really cool! I was just thinking about this very subject the other day. If there was one more game that deserves recognition for using MIDI music, it would have be Doom. It spawned a whole community where MIDIs are about as prevalent as ever there. Those composers have made some astounding pieces of music over the years that have raised the bar for how to get the most mileage out of the same universal 128 virtual instruments. Anyhow, great video man!

[@tommj4365]

Your use of the term MIDI was getting me upset, but then you clarified you're using it loosely. Breath of fresh air, lol... A few other technical issues aside, it's a good video and I agree with the overall message.

[@Alpha17x]

1. Great video, Really. 2. Sharing this not to slide in like Milhouse, but because I find it so interesting; Final Fantasy 7 actually used composition data (MIDI) and samples. The music is stored in 'PSF' files on the discs. They are a container format containing the samples for that specific composition's 'MIDI' data. The MP3 file size of the entire soundtrack is 654mb (somewhat dependent on compression, of course. The filesize of the entire sound track in PSF format (what the discs had) is 834KB. Even One-Winged Angel is like this, with the vocals being their own sample file. I used to think the same thing you do, then I got into doing crazy stuff like ripping assets off of console game discs and such to learn how they were structured, made, connected, etc. 3. With the high quality samples that now exist, can be licensed, created, bought, etc. Now would be an amazing time to use MIDI in games, especially for something like Dynamic Audio/Music. I'm actually surprised that companies like Native Instruments haven't gotten on board with things like this. Imagine something like the Kontakt Player (plays sample libraries for MIDIs, sounds like real music) integrated into a game; Same great sounding music, lower file -size, more room for cool things.

[@TextilisMusic]

I don't use midi files for the music in my game, but I do use midi modules(Roland SC-88). It's a vst, but I just love making music from the sounds of my childhood. It's also really cool getting a midi file from a game like final fantasy and having the Roland vst play it exactly how it sounds in game, but inside FL Studio. Great Video!

[@Javeton]

None of you deserve midi. Dragon Quest 11 delivered with midi and people shit and pissed their pants.

[@unyu-cyberstorm64]

Quick Time still supports MIDI

[@mybachhertzbaud3074]

The use of MIDI is what got me into computers and games. The first pieces of hardware I got in 1990 was a Roland LAPC I and a Soundblaster. Today, I pump a lot of game music through my various synthesizers. Too much fun.😁🎶🎹🎶Play On

[@thoreaukilbourne3911]

Very cool video, I’m a big fan of MIDI based music and I wish more games would use it, if only to capture the vibe it has. One small thing I don’t think anyone has mentioned (and it’s not very important) is that the PS1 actually had onboard reverb, so it wouldn’t have taken any cpu time to use it. I only say this cause I think it’s interesting, not to detract from the video. I wonder if any games used it as an effect when the player went under water or into a cave.

[@CarletonTorpin]

When I want to hear Earthbound's music, only MIDI will suffice.

[@EpicureMammon]

Yeah! Realistic music with cartoonish games is very jarring and weird to me.

[@andywest5773]

Really? EarthBound has some of the most beautiful and masterfully-written music of any video game. Why would you want to listen to an inexact copy with a random sound font? So much of the charm is in the sample data.

[@CarletonTorpin]

@@andywest5773 I think you're right. I think I was mistaken. Thank you!

[@jlewwis1995]

​@@andywest5773you do realize the original songs were midis themselves right, they just used a custom soundfont... I'm pretty sure you can just find the soundfont online and load it up into any half decent music player, then get exact rips of the midis, then you can listen to it without having to put up with the SNES overly aggressive lowpass filter. Or I guess you could just download an .snsf rip of the soundtrack and play it in a music player that can play that format and just turn the interpolation off, i guess that would technically work too...

[@arn3107]

​@@andywest5773because it will sound good anyway

[@LocalAitch]

I'm excited to see what people do with the Harmonix beatmatching tech that's now in the latest versions of UE. It can basically take the best parts of sequenced MIDI (which drives the technology at the base), and recorded audio, and blend them together.

[@Diddz]

some games used them to help the game engine control animations and hits to keep them synced to the dynamic music regardless of the player's timing on hits ( wind waker's dynamic music layering comes to mind) - this was on top of the game using sequenced music

[@tortoiselover7215]

I feel like having the computer perform music in real time, especially if it uses a lot of effects, large sample libraries or complex synthesizers, it would take way too much computing power to do that.

[@Roxor128]

I dunno. I think you could go pretty far if you budgeted 20-30% of your CPU usage for audio. There's a game called FRACT OSC that uses nothing but procedural audio. You can even play around with the synthesiser outside of the game's puzzles.

[@SilverwingedBat]

Yeah. Using the term "MIDI" nowadays is a bit hard because for example Final Fantasy VII Remake's OST was composed on virtual instruments with MIDI and not a live orchestra but sounds incredibly real, that's just the level virtual instruments are at nowadays so when people say "MIDI music" as a way to describe old sounding music it's kind of wrong.

[@BlazonStone]

MIDI is just data, it has no sound. MIDI can trigger samples ranging from blip-blops to real life orchestral, drum or any other instrument samples that you would have hard time knowing it was recorded live or sequenced

[@user-ft4jo8ev1v]

This might be a long shot, but I would absolutely LOVE a video about some of the games available for the Mac in the late 90's. I specifically think of the Pangea games, like Nanosaur or Bugdom. That era of gaming seems to be completely forgotten nowadays.

[@ComputerLabHighjinks]

Thank you for the comment! Glad to hear you're enjoying the channel. Oddly enough, I grew up using an iMac G3 and only found out about the included Bugdom 2 game quite recently. It was quite a surprise to find a full 3D platformer hidden in a random folder! I'll add these to my list for sure.

[@RaposaCadela]

5:06 "What strenght! But don't forget there are many guys like you, all over the world."

[@RB939393]

Missed opportunity to not talk about System Shock 1. The soundtrack remixed on the fly based on so many variables

[@Aranimda]

Not going to happen. Game studios do not want players who have crappy sound cards to have to listen to crappy music.

[@ComputerLabHighjinks]

In terms of game audio, the sound card really only serves as a digital-to-analog-converter (DAC) and some power management for things like grounding and line noise. Modern sound cards don't create the sounds using virtual instruments built into the cards themselves, which was a feature we saw in the DOS (and early Windows) era. It's basically all handled in CPU now.

[@saricubra2867]

@@ComputerLabHighjinks CPUs back then in the 90s struggled with MP3 playback in the first place 😂. They were so slow and trash, RAM and storage was so expensive and slow AF for audio as well (unless you used recorded music from CD streaming). Nowadays with a current CPU, you can have a lot of instances of accurate emulations of a Roland Jupiter 8 analog synthesizer and others. Still, audio/music production is still the most demanding thing that you can do on a computer, that hasn't changed in decades.

[@napdogs]

I'm so glad that it isn't just me who is like this. I recently tried to find a midi collection website like the ones of old to get that feeling again but they are all behind pay walls now. Ridiculous for what was everywhere only 10 years ago

[@blast_processing6577]

I think a new tracker music format, like an *updated* Fast Tracker 2 or Impulse Tracker music file format, would be far more useful than MIDI in the gaming industry, provided certain features were implemented. There's no reason the tracker program itself couldn't accept MIDI as an input though.

[@Roxor128]

A few of them actually did. I just checked in DOSBox and Fast Tracker 2 does.

[@sobanoodles4286]

honestly I came into this video expecting you to be referring to game using that soundfont 16-bit style for music again, which I've seen be referred to as midi because its only playable like that. But instead I'm leaving this fully convinced that you're not actually sure what you're talking about for starters, midi hasn't gone anywhere, heck you even said yourself that some vsts using midi sound just like the real thing. where do you think those sounds come from when you hear an indie game's soundtrack? For example, Hollow Knight's ost is actually just vsts, just used really, really well, which is why it was a big deal when it was announced that silksong's ost WAS going to be live orchestral recordings second of all, with the increased power of vsts, comes at the cost of storage and cpu. You think modern triple A games are massive now? try fitting a ton of sample libraries into it. Also while some vsts are very light on cpu, others are very heavy. There's no way you could run them AND an actual game at the same time unless you had an incredibly powerful computer third of all, most of the effects you said needed midi can still be done by splitting instrument tracks, its called dynamic audio, and it's used a decent amount. Again, hollow knight is an example, having heavy percussion kick in when you enter combat, turning a normally calm soundtrack into one thats more intense or even frantic forth of all (or third of all part 2), the game play effects that "need midi" to work, could just be coded in otherwise. like the timing in mother 3 could 100% exist in a modern game that uses audio files, just needs the timing coded in and finally, while yes you can theoretically make any instrument sound using a synth and effects, most people aren't at or going to learn that level of sound design when they're trying to make a video game ost look, I don't mean to sound so negative, but as a composer myself, this video screams "I don't actually know much about what I'm talking about, but I'm going to act like a know a lot", even in ways like talking like protools is the only software of its type when its not, its a type of program called a DAW (Digital Audio Workstation) and there are many out there, like ableton, fl studio, lmms, reaper, etc, some of which are arguable much better then protools (sense protools is held back by needing to have outdated code so old recording studio equipment that need it can function) I will say, at least, that the quality of the editing, pacing, etc in the video itself is good, so good job there

[@ComputerLabHighjinks]

Alright, I'll bite. You're making some interesting assumptions about my level of expertise with audio editing/mixing, which I suppose is fine, but it's worth keeping in mind that this video is meant for a general-interest audience. I'm not going to dive into my favourite Waves plugins and Pro Tools bus workflows for a video that's under 20 minutes and meant to serve as a general overview of what MIDI-like tools can accomplish. It's not a gear unboxing or software tutorial, it's a comparison of different workflows in the abstract. Can someone manually plug note information into a game to replicate the Mother 3 stuff? Sure, but they probably won't. Deadlines are getting tighter and budgets are getting more rigid, and most companies aren't going to pay for a Nintendo-esque full year of QC. And I maintain that a simple crossfade between bounced audio tracks is less powerful than what can be accomplished with true real-time instrumentation. I work with stems on a regular basis at my day job, and even on the video production side of things, they're rigid and unhelpful. I would similarly argue that Blackmagic's push toward integrating post-sound into the NLE is more powerful than just sending .wav files between different teams. Much of what has become "industry standard" is based on what helps management maintain control over their employees, rather than being the most optimized workflow in the abstract. To your other point, Pro Tools certainly isn't the only DAW, but I was using it to illustrate how musicians are often held back by legacy gear. However, in my experience working in film post-production and having colleagues in music production and even game audio, I can safely say that Pro Tools is the only DAW that'll land someone consistent paid work in any of these industries. It's nothing to do with Pro Tools' abilities, it's just the nature of labour and capital in a marketplace where decisions aren't made by the right people. In terms of better DAWs, Reaper will hopefully get there in the next decade, fingers crossed.

[@Xbox4Ada]

I believe another good example of the sequencing being used for something other than exclusively music is fortnite's festival gamemode. It uses midi sequencing to control the animations for the instruments and I assume all the effects happening on stage.

[@CarletonTorpin]

For the first second of this video, I thought I'd clicked on A "You Can Beat Videogames" video. :D

[@ComputerLabHighjinks]

Oh WOW, just looked up that channel and I see what you're talking about. Nearly identical opening shot! That's eery.

[@TheOneFreakservo]

If the argument is that Midis should make a comeback, you haven't been listening. Head toward the homebrew or 'Aftermarket' of retro games. Check for Genesis games like Demons of Asteborg or ZPF, or Neo Geo titles like Project Neon. And if you think they should come to modern consoles or Steam, many usually find their way to those digital storefronts. Listen to the new generation of Midi.

[@Beatboxbob]

I am aware that Final Fantasy 7 was used as a test-bed for next generation console technology. It's why differnt environments used completely different character models, they combined 3d models on 2d backgrounds and why there's quite so many crazy glitches possible. So my best guess would be that the plan was always to use orchestral recordings (particularly for that final number with the choral parts), but that the orchestral recordings took too long, or too much money. So the initial release used audio recordings of the sequenced tracks because it would be a completely different use of technology to playback the sequence. Also, later on it did use orchestral recordings for everything.

[@diggus88]

You brought up actual recorded audio in the 16-bit era but didn't mention Magical Pop'n! That was almost certainly the most extensively voice-acted game of the era, and a wonderful, cute, very replayable metroidvania to boot for the SNES/SuperFamicom. I'd be surprised if any other game from that generation used more recorded audio.

[@1ucasvb]

As you pointed out, old games didn't use "MIDI", they used sequenced music. It's not really accurate to use "MIDI in the broader sense", as it undersells what the music engines and audio hardware was doing in these old games and consoles. It also invites unflattering comparisons between old .mid files and chiptunes, which is something the chiptune community is kind of sick of.

[@IsmayaMelasrana]

I tend to extract the midi from old java phone games and change the instrument to a modern vst. It giving a new fresh take on the tracks, bc most of the composition is still good even after a decade. I think having midi inside the game is almost feel like giving music source code for free, pretty sure the game company didn't want that. encrypt might help but people eventually break those protection. While recorded music even when it stolen or decrypted, they don't have the full source code of the composition.

[@LeonardoOuteiro]

Great video! I actually owe a big part of my career as a music composer to midi - and that is my main occupation! I scored plenty of films, series and theater plays all with midi and VST's. It is a great tool that empowers composers that otherwise would not have a chance in their life to hear their music played by an orchestra! You're also right about the conservatism. I've heard some musicians, very snobishly, call the use of midi and samples as being dishonest.

[@captainsoda6590]

I highly recommend looking up Lucasart's iMuse system used in Tie Fighter and Dark Forces from PC DOS. Dynamic music changes based on situation with seemless transitions. Great stuff! Love my Roland SC55.

[@atalhlla]

Just to add to this, I recently learned that the (one of the?) Genesis/SMS music system had a mechanism for that dynamic game engine interaction called mailboxes. It apparently wasn’t used that often, but it did exist! That said of course, you could still always make your own driver to implement whatever dynamic behavior you want.