MIDI & Music in the Browser

Welcome to Syntax. On this episode, we're gonna be talking about the Wes Audio API and the WebMIDI API, what these things are, why they matter, and maybe some interesting things about them that you might not know. Because me, personally, when I think of Wes audio and web MIDI, I'm thinking of maybe playing sounds or, accepting input from MIDI, but you can do a lot with these things. And there's a handful of really great community libraries out there to help you make both music, but also just control things with MIDI messages. So we're gonna be talking in this episode about some really interesting audio technical, concepts such as sampling. We're gonna be talking about MIDI in general, the types of messages that are sent, how it's sent, what it means, how we can interact with that in the browser. And then, ultimately, I'll be showing you some really neat demos of stuff we can do. In fact, my desk Play us in, Scott, will you? Play us in. Okay.

No. Play us in. Play us in. Is that Yeah. Sounds awesome.

Okay. I can't hear that, so I don't know what it sounds like in my monitor right now. But I'll I'll be definitely show showing you some of this in just a little bit. I'm building something fun with this, so I thought it'd be a good opportunity to take a time and do an explainer.

But, hey, if you're trying out new things in your site, maybe you're pushing it live, and maybe that new thing is broken. Maybe it takes down your site. Maybe you're getting a ton of issues on one specific thing, or perhaps I think that I have covered all the use cases that I thought I should cover in this, and it's not working. So you're gonna want to use software like Sanity at century.i0/syntax, Sanity it two months for free. This software allows you to keep track of all of the errors, bugs, issues, anything that's happening in your site, fix performance issues, find out what went wrong. You even get a session replay so you can see a video of what happened. Let's say the user went to click on a keyboard on a website and it played the wrong note. I could figure that out with Sanity. So check it out at sentry.io.

Wes, my man, what's happening today, man? How are you feeling?

Pretty good. Pretty good. Excited to talk about Web Audio. I've dabbled in it, I don't know, three or four times. Yeah. There's this, app called BeBot, which is on the iPhone. I've had it for probably, like, fifteen, Node, fifteen Yarn. And it's generated sound, and it it goes up and down the scales. And I don't really know a whole lot about music and how all of that works, so I I went and built that at one point. Oh, did you? Cool.

Each small CSS and JavaScript, and it was kind of a fun learning experience.

Yeah. So what that on Bebop, I believe it's the vertical if you're holding it sideways, the vertical is the pitch or it's the the the tone or something. One of them is the pitch and one of them the tone. It's kind of like a a theremin in in in an axis of of how you're doing that. Yeah. I have a lot of experience with this stuff. I I did a lot of music programming when I was in college. In fact, I took several years of a music programming class where I was doing, like, complex Yarn off chains to generate, music, generate songs, like, you know, automatically generated unique songs every time. We were doing things using, physical hardware triggers to trigger music. I I built a a synthesizer out of a projector, actually, Wes. That was, like, one of my coolest projects. What I did is I projected this is actually a really technical, like, interesting way that I solve this. I projected color beams of different color onto the wall. So let's say there's, like, six different colors, red, whatever whatever.

And I projected that, and I had that image saved of what that should look like. And then I had a video camera pointed at that and using, a or a Max MSP, I was able to overlay the two. So I had what the video was showing of the projector plus what it should look like. Mhmm. And I was able to do a if one of these color beams suddenly has a a difference of enough threshold, than fire a MIDI note. And that was sent to a controller to play a sound at a certain tone. So you could run through this thing, and it would sound like you're dragging your hand through a keyboard. It was probably one of the cooler projects that I put together. So That would be a fun Canvas project if anyone's looking for something JS take a video and then

figure out you probably just use the x y coordinates of the person's hand, or you could you could overlay it if something is breaking up a specific color.

Yeah. Well, let me tell you. I even did a, theremin, which you don't know. Theremin is is used by, like, radio waves, I believe. And you put your hand into it, and it detects where your hand JS. You go.

Yeah. That's really cool. I made one of those with my webcam, and I did video tracking. So I would hold up a little color square and where the dim where I put it and how big the color square got, I had, like, a three d, dimensions of being able to tweak both the the audio, the quality, the timbre, all that sort of stuff. So I was able to make noises with my webcam just like that. So long story short, I've been I've been experimenting with just tech and music, the intersection, for a long time. And and, actually, since getting into web stuff, a lot of that kind of fell off for me for no reason in particular. I don't I just became less interested. But more recently, I've been getting back into some of this stuff, and the Wes Audio APIs that ESLint are really fantastic for a number of reasons.

I had used the Web Audio API to do audio synthesis. I've done media recorder.

I've done kind of, like, things here and there, but I haven't really dove into it until, somewhat recently here. So we can start off first and foremost. Let's talk about the WebMIDI API because I think the WebMIDI API is is fairly underrated amongst folks. In fact, even people who know what MIDI is and understand MIDI might not understand what you can do with MIDI in the browser. So first and foremost, MIDI stands for music instrument digital interface.

It's the way that electronic music instruments can, interface with devices, computers, those types of things.

And when you have a MIDI device, it is sending messages.

It's sending messages to whatever the device is and the device, the digital audio workstation, you know, you have Logic, Ableton, those types of things Yeah. Receive those MIDI messages.

And those MIDI messages, unlike Node audio form, like, when you send a waveform, that is Yeah. Audio.

When you send a MIDI message, it's just data. It and it's not even data of how it sounds.

It's a few messages. And, typically, those messages will look like, play this note, play it at this velocity, and Scott, or, you know, note on this note, this velocity.

So it's just a description of the notes

and how long and and how high they should play them. Right? Correct. And there's all kinds of things that people can use MIDI for beyond notes. Like, I have a MIDI keyboard here. And Yeah. You can do, like, pitch bend and modulation, which is which is little touchpads.

Those are all sent via MIDI. Or, even, like, being able to change volume with these knobs, that that message is sent via MIDI as well. But MIDI can send you know, it's more than just, like, you think about, like, a key press on your keyboard.

You you get the on and off, but you don't get velocity. You know? You don't get how hard you pushed it, which that's a lot of interesting information. Right?

Yeah. Because BetterTouchTool, like the keyboard shortcut and everything tool that I use, they have full support for a MIDI controller. And I always thought, like, oh, that's kinda interesting. People are hooking up keyboard shortcuts to their piano.

And it it makes a lot of sense because you can you can store a lot more information in a

a MIDI payload, I guess it's called. You're gonna go into it's three bytes. Right? Yes. So most MIDI messages in the typical sense you're playing a keyboard are three bytes. It it can also be two or one depending. It can be more than that, I've heard. I don't really, if you're, like, a midi expert out there, in general, when you're working especially with, like, a keyboard, it's gonna be three bytes. The status byte is eight bits, and then the data bits our data bytes are seven bits each. So the first message, the first number that comes in is an eight bit byte of the status. That status typically means if you break it down into actual binary, the first digit is telling MIDI that it's a status byte.

The next three are telling it if it's a note on or a note off. Like, you're either pressing it or you're telling it to stop. Right? Mhmm. And then the next one next four are just simply what channel it's on. So that's the status because you can send many messages on many different channels and receive them on multiple different channels as well. That allows you to What is a channel? So a channel is exactly kind of like what you think about it. It's kind of like a a pipe. So if I'm sending this message down pipe number Node, in my controller or wherever I'm receiving that message, I can look in pipe one, and I can know that this is coming for this specific thing, I e, like, you have this thing. Right? It has drum pads, and it has a keyboard. I can put this on one channel, and when I send a message here, it could send that in channel one. And then the MIDI receiver can listen and say, anything that's on channel one, that's a keyboard. And then I could send these on channel two. Anything that's on channel two, is going somewhere else. Maybe it's a drum. Maybe it's modulating something. Maybe it's changing.

Oh, okay. I see. I see. Yeah. Yeah. Yeah. It's like just putting things in different you go here, you go here, you go here type of way. So the second byte is a data byte, and that's a seven bit. And by seven bits, it goes zero to one twenty seven.

So this is the note. So when you're playing and I hit middle c, it sends a note on, which is zero x 90. And then the second byte is going to be 60 for middle c because it's zero through one twenty seven. And then the last one is the velocity, zero to one twenty seven. And that's also a seven bit byte.

Can I tell you how hilarious this is that we both got into writing binary, within the last seven days of our entire career? So I recently did a video on controlling my Roomba with JavaScript.

I hooked up, like, a serial port to it. There's three different, like, motors on it. There's a spinner motor on the side. There's the actual vacuum motor that sucks, and then there's the main brushes underneath that spin. And then also two of those brushes can spin in reverse.

And in order to send the kind of state of of where you want the motors to be on, right, because there's five different things, there's a million different combinations. Scott a million different combinations, but whatever, the permutation of that is.

And in order to send that combination, you send it over in in binary. Right? And I thought I thought it was so hilarious that, like, I have never actually written raw binary or or bits and bytes like that before. Yeah. And and here we are doing it in the same week.

It it is funny because I was like, I I mentioned I've been doing quite a bit with this. And at first, I was trying to do everything with the bits and bytes. I was trying to, like, really get into the raw MIDI. I was like, yeah. I'm just gonna use MIDI straight up, and I'm gonna be very, like, content with that. But the more and more I got into actually, playing notes and playing scales and wanting to do, like, more real, audio stuff with it, I I realized, like, oh, man. I I really don't wanna be getting into that. I want what I wanna be doing is just worrying more about the audio stuff. So, actually, I'll I'll share some libraries in a little bit that that help with that sort of thing because there is, like, music based MIDI libraries or somebody's already figured it all out. You can say play this Node, and it will just play rather than having to worry about the binary. But, yeah, I I got into it quite a bit. So, basically, what happens, like, you play a keyboard, it sends a note on that note in the velocity.

And then the moment you release the key, it sends another message, which is note off. And, if you wanna see this in action, I'll actually, share my screen here if you're watching on video. And for those of you who aren't watching, we we'll just basically tell you a little bit about what's coming in here. So, basically, what I'm doing in this example here is I'm getting access to my MIDI inputs, and then I'm setting a function to the on MIDI message.

This message, this function just receives a message. You can see that anytime I hit a key and I know if you're just listening to this, we'll describe it, but you're getting a it's a straight up JavaScript event that gives you some information.

Inside of that information, is a data array.

It's all kinds of stuff that you have the array buffer here.

You have what type it is. It's a MIDI message.

You can come in here and see exactly, like, what exactly it's coming in from. So for instance, we the device itself JS, this is a Samsung it's a crappy MIDI keyboard.

So it tells you, hey. This is coming in from a Samsung Samsung MIDI keyboard, and the input message itself comes in as message data.

And that gives you the command, the Node, and velocity inside of the message data.

That's cool. How does it know what the the keyboard is? I I'm assuming the the hardware sends along, like, an identifier byte, and then the library looks that

up? Oh, I'm sorry. This is not done with the library. This is done with the the web the MIDI API itself. So my understanding even through a library. Yeah. This is just the device itself sending that information to the the browser.

Because you'll see here in my my other example that you can check to find all MIDI input and output devices on your your device just by giving the browser access to it. I see. I see. The so the browser itself has a list of similar to how you can get all webcams and microphones, you can also get all MIDI devices. Okay. Yeah. So I have all my MIDI devices, all all that stuff. Either way, that's what type of message it's sending. Now in my more advanced example, I did use this library called WebMIDI JS, and it's super good. I'm gonna tell you that right now. If you're doing anything with this stuff, the WebMIDI JS is well worth using instead of potentially trying to do it raw yourself because, man, the dealing with the the raw information is kind of tough, especially if you're doing music stuff with this. So with this library, I'm able to do a more standard, like, add event listener. So I'm listening for a note on and note off and control change, these types of events. And this time, when I hit an, key, I get a lot of information with each Node, like, the attack is the velocity, the release, the octave. That's cool. Yeah. So it gives you more information.

Like, it's giving you the name of the the note itself because this library is taken it's like, we're gonna decode the MIDI messages into things that make sense. So if you're actually doing music with it, you'll wanna use midi. Js,

or WebMidi. Js, I believe. And, man, what a cool library for that. And the velocity or the attack, as they call it here, that's how fast you hit the key.

So I imagine somebody playing piano. And if you hit it very quickly,

it would sound different than if it was you pressed it slowly. Right? So, yeah, I just smacked it, and it gave me a one. Now I'm gonna lightly press it. As fast as possible.

Yeah. I lightly Wes it, and it's giving me a zero a point zero zero seven eight. So, yeah, if you're playing lightly or you wanna you you can do all kinds of stuff. And, you know, it's a more advanced digital audio ways of working with MIDI Wes is that they're actually checking that velocity and even playing a different audio sample based on that velocity. Like, you have a snare drum, and that velocity, if it's below, you know, point five, it's like a little a recording of a tap of a snare drum. And then you hit it more. It actually plays a different audio file, like a snap. You know? Because oftentimes, what we're doing with MIDI is we're using these to control and play waveforms or generate synth. You're generating synthesizers or something like that. But, typically, you know, if I hit this and I'm doing a drum kit, it's usually just a waveform of a drum being triggered in the background. So the MIDI message JS, like, play this note, whatever. It's like, fire this MIDI or fire this waveform. Yeah. That's cool.

Yeah. So MIDI, really neat. Right? You can do so much with it. What I did not realize about the MIDI API is that it doesn't just allow you to receive MIDI because that's the way I always perceived it. Right? You have MIDI devices. The web browser can accept those MIDI messages. You can do something with it. But the browser can also output MIDI messages, which I found to be really super neat. So for outputting MIDI messages, your browser, just like the inputs, has a set of default or your your computer has a set of MIDI outputs.

And by default, it could be any number one of these things. You might not have any, to be honest, if you don't have any of these things set up. So for instance, I have my audio converter, which is my digital audio interface. That shows up here. I have the keyboard. It actually shows up as a MIDI output as well.

And then I have this I a c driver bus. So you can do this in Windows or Mac. I don't know the Windows, way to do this. But in in Mac, you go to audio MIDI setup.

You go to window, and you open up MIDI Studio.

MIDI Studio gives you access to, like, an interface. You click on IAC driver. You enable that thing. And then what that does is it essentially sets up a MIDI bus on your computer that you can send audio to. So it creates a MIDI device on your computer.

And by doing that, that now shows up in browser as a as a place I can send the MIDI. So I'm sending this MIDI to this bus.

And then in another application, for instance, a digital audio workstation, I use Ableton Live a lot, you can then see the one that you created and you can use it here. So for instance, I built this keyboard synth. So if you're watching along, you can see this. Otherwise, you'll just be able to hear it. I built a keyboard synth and allows me to do programmatic music. So what's cool about this this synth here that I built is it's just sending MIDI messages. Right? But these MIDI messages are actually able to do a lot more than just, like, a straight keyboard.

So Oh, man. Like a normal keyboard, you play the Node.

But what if you could play chords? So I have it set up. Anytime you play the a key, it plays an a chord, and that's sending three messages, MIDI messages. So if I'm playing, and recording in Ableton and then oh, shoot.

This is the hard part because you have the the browser active.

So let's start the recording. I hit the a key.

Yeah. Ableton receives those messages and records them right here as data.

This is this is blowing my mind because MIDI is a way to get data in and out of the browser. It's it's almost like a WebSocket, isn't it? Yes. Yeah. You you could use MIDI as a way to pipe data out of the browser, and then you can use like, you could have you could probably have, like, a Node app that listens on, for on, like, a local MIDI that that MIDI source that you just created. So you're sending it from the browser over MIDI, and then you can pipe it into could, obviously, could be a music app like you're doing here, but it could be, like, BetterTouchTool, which is, like, custom keyboard events.

Or it could be, like, a Node app that you're sending data over. It's probably the the data is limited. Right? You have three bytes.

Yeah.

But you could. Yeah. And what's cool about it is you can augment it. So, for instance, while I'm hitting this key down, I I'm saying, oh, play the a key, but also play, the the d key, also play the g. Right?

Yeah.

But you can do some math on that. So I built this, what what I'm calling a locked chord view.

So now if I play the second row of my keyboard, it's always going to give me, like, a chord within that scale.

So if I select a c major scale, every chord I play automatically is going to remain in the c major scale because I'm I'm doing that math wise. And so I added some some keyboard controls of this where I can, I can change which key it is? And you can change major or minor.

Also, in the normal chord view, you can modulate them. So I have a second row where I hit a a chord, and then I can make it minor just by hitting a two key. And then I hit the three key, it makes it a seven, a major seven, minor seven, diminished.

Yeah. So you could modulate it. You can do all kinds of stuff with math just by changing the midi notes on the input. And then furthermore, you can use things like Ableton Live to Augmentative for further, where I'm piping in an arpeggiator.

And let's change this to locked chord mode.

And now it's playing a chord into Ableton.

Ableton is spreading the notes out.

Beautiful.

I'm so inspired right Node, this music.

You're just messing around with it. Yeah.

That's this is this is actually fascinating to me because it it's like one of these overlapping things. You know? Like, there's web development, but then there's also music, but then there's also math, which JS,

figuring all this stuff out. It's that's really cool. Yeah. It's it's neat. You Node, I think the coolest thing about it for me is that now I'm able to build, like, a little exploration engine where I can, like, I wanna say I wanna build a a song in d minor, and then I don't have to, like, memorize my hand positions because I I took piano. I I was a music major. I know how to play all the scales. But, like, sometimes I'm just like, oh, what was that scale again? Oh, you know, I I'm in a weird, key or or whatever. I I don't, you know, play it often. My hands aren't used to that. Now I can just be like, hey. Give me the the fourth chord in a a d d major scale so pretty cool.

So, yeah, a Scott of neat stuff going on here. Overall, I I think there's a lot of just opportunity here, and I'm I'm gonna take this a little bit further and try to build something like Truly that I can mess around with. I I even thought about building the arpeggiate arpeggiator directly in the browser, and then I was thinking about it. I was like, man, Ableton does that way better than me trying to write my own arpeggiator because then I gotta write a a metronome and a clock, to keep it all, like, in sync and stuff. Mhmm. I don't wanna get into any of that. So that's WebMidi.

Man, now now you have me going down downloading a ringtone Mhmm. Because ringtones are MIDI. Right? They they were. Well, yeah. They were. Yeah. Yeah. Twenty years ago. But I always wondered about them. Like, how how come you could just download ringtones back in the day? And it was because it was just a file with a description.

So it was like, a, like, a one KB file with TypeScript of how to play them, and then the phone itself is the one that actually has to turn it into to music, right, which is why it sounded so

so funny. Yeah. Yeah. It uses the same audio engine that it they use it when you push any of the keys. Like, beep boop boop boop. It's using that same audio engine to to recreate the songs. That's MIDI, man. It's it's so cool because it's small. MIDI is really small where audio and waveforms are really big. So it's very transportable. It's fast to send. It's fast to do anything with. So, one of the many cool benefits of MIDI. Let's talk about the Wes Audio API though because this is the Web MIDI API. The Web audio API can be used for a ton of interesting stuff too. For instance, you can generate sounds using an oscillator.

An oscillator is kind of what it sounds like. It creates a waveform. Right? So you're creating, like, a sine wave, and then you're controlling the Hertz. What is that? The frequency? You're controlling the frequency of the the hertz, and that changes the pitch. So the velocity is how big the waveform is, and the pitch is how, tight or wide the waveform is. And if it's a sine wave, it's gonna sound one way. If it's a square wave, it's gonna sound a more more buzzy, sawtooth, those types of things. There's all different types of waveforms you can do for that. But you can generate audio straight up in the browser using an oscillator, and I'll show you an example of that in just a second.

You can also, bring in sound via WAV files. You can bring in sound via the WebRTC media stream.

A lot of neat stuff here.

Yeah. So one one little demo I built a while ago when I was looking into the the Web Audio API was creating a dialer, with your phone. So, you know, when you when you hit the numbers on a phone, it goes.

So it turns out that all of those sounds are and I I initially stumbled upon this when I was, watching a documentary on phone freaking Yeah. Phone freaking. How how, who was the guy that did Kevin Mitnick.

Kevin Mitnick. How he But also Steve Wozniak did a bunch of phone freaking. Yeah. You know? Yeah. They basically they blew a whistle they got in Captain Crunch, and that whistle made the same frequency that would allow you to bypass these, like, phone companies and whatnot. I thought it was kinda interesting. So I, like, I looked into it, and I was like, you can make any audio you want in in the browser with the the Web Audio API. And I I went into the Firefox OS code Bos. So for those who are, a little bit younger, probably about ten years ago, Firefox launched a operating system that was meant to run on your phone. Yeah. And it was written entirely in JavaScript. And it was I was at the launch of it, and they gave us they gave us, like, a Android phone that had been flashed with Firefox Bos. And it was really interesting. And what was cool was that you can you still can. You can go into the code Bos for Firefox OS, and you can see how they implemented all the, like, native apps. And one of the native apps was the dialer, literally dialing stuff. So I I went into it, and they have an array of the frequencies that are needed to play. Because when you press a button on a dialer, it actually plays two frequencies at once. And every single number has a different frequency, and that's how that data can be sent over the phone. And I just, like I I reimplemented it in, I don't know, maybe twenty, thirty lines of code, of of JavaScript. It's just creating two oscillators, and then setting the the frequency of them Wes you press down. And then when you let go, you you turn them off. Yeah. So I don't know if you'll be able to hear this because it's

in the browser. Maybe turn on my maybe that will help. So, basically, for those of you watching, I have some code on the screen. For those of you listening, I'll describe it. You create a new audio context with window dot audio context, then you create a new oscillator with audio context dot create oscillator. So that creates essentially your oscillator. You set what type it is. Is it a sine wave? So you say sine. Then you can set the frequency, and you set a value at a current time. And this is a straight up hertz number. So, like, four forty is a four. It's the first a key or it's like the middle a key. So you go to middle c and you move over to a. That's a four forty.

And then you connect it to a destination like your speakers.

You hit start, and then you can stop it. So I I have this code that you might be looking at. I'm just gonna hit enter on this. And I don't know if you could hear that. It was super loud in my headphones. No. Blew my ear out. Either way, it played a sound tone. We'll have this code in the show notes. You can copy and paste it. Throw it in your console. Just hit enter, and you'll get a sine wave. And you can see really quickly what in, like, six, seven lines of Node, you're able to generate a sound using a, a sine wave directly in your browser with, like, no effort. And this can be used to do a lot of stuff, to create full synthesizers. That was one of my first projects Wes to create a a keyboard that played audio synth, just based on whatever, you know, key I'm hitting on the keyboard, similar to what the project I'm showing you in this this, podcast, but, like, for generating synths instead of sending MIDI somewhere else. And that's really great. I think it's really neat. There's actually a really great library that can take care of this as well called Node JS.

Tone JS is super good. It handles, like just like WebMIDI JS handles a lot of the nitty gritty stuff. This really just makes it super easy.

Play this note, c four. You know? You Node, play it now. Start out. Like, it's way a nicer API than having to get into some of the nitty gritty of this all, but definitely a a cool thing to mess around with if you're interested in actually generating audio synthesis.

Like like, even another use case, if you're not into music, sometimes in your app, you need to just play a little sound. Yeah. Like, for example, if something finishes and you need to play, like, a little boop or or something is incorrect, like, if somebody tabs away and you need to play an audio alert that something happened, sometimes it's easier to simply just write, what, seven lines of JavaScript to create that little boop and play it Vercel having to load a whole wave element and then, like, load it up, play. Because the thing about the audio API and playing audio in the browser is you have to latch onto a click event, before you can actually play audio.

Mhmm. So you gotta, like, wait for somebody to, like, click something. And the first time they click, you gotta set it up.

And then you you can play it later on. So Why is that mess? Nice. Do you wanna explain to the audience why that is? Well, you can't just autoplay audio, on websites because it's it's the same with video. Right? Yeah. Auto playing, those can be extremely annoying, so you have to do it. There's a whole maybe we should do a little show on that, like, on what makes you allowed to autoplay, because some some websites can. And there's, like, a algorithm that goes into whether you're allowed to autoplay audio or not. It's honestly kind of a mysterious algorithm.

So yeah. No. Super interesting.

It's just math. But Yeah. Well, no. It that's you know what? Wes, sometimes math is mysterious.

Yes. That's true. Yeah. What's interesting is that the browser also takes care of, like, sampling audio. So those of you who don't know too much about how digital audio works. So let's say you have a record, like a a vinyl record, and you're playing that record. That is actually playing a straight continuous waveform. Right? That's playing just straight up why why it's called analog audio. Right? It's playing just a straight up waveform.

Where the difference between analog and digital, and it's some of the reason why people say they like their records better than their m p threes, is because when you have an m p three or you have a digital audio file, kind of inherently what it's doing is it's sampling the audio.

So every x amount of milliseconds, it's taking just a single chunk of that waveform. So you could imagine, like, a smooth waveform suddenly becomes like stepped blocks. Right? And the amount of stepped blocks, how Sanity you chop that chop that up we're we're, you know, we're chopping carrots here. How finally you chop that up is what's called the sampling rate. So the higher the sampling rate, the more close it is to analog audio and the better it's going to sound. But, also, the the larger this this audio form is gonna be, the more data it is. Right? Because you're sending way more data every single time you have smaller chunks.

So that's a a big reason why people say digital audio doesn't feel as good as Not just analog audio because analog audio is not losing any information, where digital, you have to chop it up. So, like, if you zoom in close enough, it's always going to be stepped. It's always gonna be chopped instead of a smooth continuous waveform.

So if you ever hear sampling rate, that's what that's doing. You can control sampling rate in the browser in an audio buffer. So if you're sending audio over, you know, what, like, over WebRTC, you're sending audio to somebody and you want it to send less data, what do you do? You lower the sampling rate. The the audio JS gonna sound tinier. It's gonna sound more hollow. It's losing some of that information, but it's going to be much smaller. Right? So that's one of the reasons why, you know, you're when you record or or we do this stuff, we wanna record locally, to get the best sampling rate instead of whatever we have to send over the network for Wes here, or vice versa.

Yeah. So, yeah, you can you can control sampling rate directly within the audio buffer just by changing the sampling rate. Pretty neat. I just found another project from Phil Miller. He's the,

dev behind Domino, d m n o, doing, like, some, like, secret stuff. Mhmm. I just randomly found a project by him, which was WebRTC and using webMIDI.j s. So being able to play your MIDI things, but with friends.

Mhmm. And instead of sending audio to friends, you're sending Yes. MIDI data, and then you're just reproducing it on the person's machine. So, like, that would be

that's not a lot of overhead at all. I was thinking about doing that same thing. So that's interesting to know that somebody else would want that.

Yeah. I know. I was thinking about, like, how you could have a multiplayer MIDI workstation or even you're recording MIDI in the browser, multiplayer, or something like that could be really super interesting. I'm gonna check this project out. That looks really super neat. I should have actually looked before we recorded this. So that was just like, oh, I wanna make that. No. Sometimes you have an idea, and it's like, I don't even wanna

look if somebody's done this because I'm so excited about learning and and doing it JS I don't wanna be, like, someone already did that, or,

I I don't wanna see how they did it because I I love the process of trying to figure out how to do it myself. Yeah. That that's actually, like, a lot of what inspired this keyboard that I made for myself because I got an ad. I got served an ad for a synth created by or a MIDI controller. I don't even know if it has synthesis on it. I think it's a MIDI controller.

From Tame Impala. There's a band, Tame Impala. They re they created their own idea generator synth. I don't even know what they're calling it. Regardless, it's sold out. They had, like, a limited number of them. And it does exactly what mine does, where you can hold one button to do keys and hold another to modulate or play, arpeggios inside of a chord or or strum them or do all kinds of stuff.

And I went to look at it. I was like, oh, that sounds neat. I would love to own that. And then it there's like, oh, we're we had a limited amount, and it's sold out. I'm like, you know what? I'm a build in JavaScript. So I had I had a buddy of mine message me on Instagram, and he was like, hey. Did did you get inspired by the Tame Impala Sanity do this? And I was like, oh, shh. You Node? Yes. I I did. That's definitely what I'm doing. I'm recreating it.

One more one more neat fact. Sorry. I keep just interjecting on the show. Node. Neat facts. I love it. But this is my app where I built the the dialer.

Node. And it it wasn't working earlier.

And I was just debugging it a second, and it says, like, an an audio context is not allowed to start Yeah.

Because it has to be attached to a gesture. Right? We just talked about that you have to click first. But guess what is considered a gesture on Safari, on iOS?

Could it be, like, the accelerometer?

Node.

I'm not. The mute button on the side.

I I couldn't figure out. I'm like, it was working. Now it's not. And so I turned you just I had my my mute button off, and I I turned it off, and then I turned it back on again. And then that was enough to consider it a gesture.

Mhmm. And and now Safari lets you play the audio, which is awful.

You know what doesn't need a gesture is the MIDI API. So that means, like, you hit a key, and it would still register the MIDI ESLint. And that what that considers it a gesture, so maybe you could start audio synthesis then. I don't know. I haven't even tried that. Yeah. You have to you have to wait for the first

user input, whether that's, like, hitting enter or I'm curious what the gestures are considered now. Yeah.

That's a browser level implementation. But, clearly, flicking the mute button on the side of an iPhone 13 Pro Max is enough of a gesture to allow it.

Yeah. I I guess that gets us into the last little bit here, which would be the media recorder. Now we're not gonna spend a ton of time, but media recorder, as far as Wes Audio APIs go, is really great for a lot of reasons. Like, potentially, you could be, just building a simple recorder where it's connecting to your audio source, your microphone, you're talking into it. You can record that. You basically, you create a a new recorder and you pass in data into chunks. And then once you're done, once you stop the recording, you can then create audio directly in the browser.

And it's pretty amazing. Like, I can essentially record audio directly in the browser, save it as a a file, and then download that file directly, entirely in the browser without any additional software just using the media recorder API. So, man, so much cool stuff in this these APIs that I feel like are fairly underutilized, whether that is the media one, the, audio API.

So much cool stuff here. And that's not even getting into, like, WebRTC or any of that cool stuff that you can do there. So a lot of interesting things you can do. If you're a music person or just curious about any of this stuff, spend some time just diving into the MDN docs. We'll have a link in the show notes or fire up Node. Js or Wes MIDI. Js and experiment with this stuff because this is the type of thing that when I work on it gets me super inspired to make fun things rather than, oh, I'm just pushing pixels on a website. No. This is like, I'm creating music. I'm making something that has actual sound. I'm I'm, you know, something that I can show my kids. Hey. Push this key. How does this sound? Or now push this minor note. How does that sound? Like, I could use these as educational tools.

So very interesting.

If you're feeling at the very least burnt out on coding, putting buttons on screens and clicking them and updating databases, try hook up a a freaking keyboard or a a Roomba to your browser.

That's really fun. And, it will ignite not to say that I'm burnt out. A lot of people thought that on my video. But, I always recommend people to do creative coding and go outside of your your regular button clicking

because it makes you so excited about web development in general. Yeah. Imagine this. Right? Here's some, here's some freebie project ideas. Like, we we talked about sending and receiving messages or data or audio or stuff like that or generating. You can use that information in all sorts of cool ways. Like, what if you took the mini message, I hit a specific note, and that note controls the background color on the website. That's a fun little project right there. I hit this key. It changes to this color. And you could just, what, clamp the the notes on a scale to the range of RGB colors and, you know, have some fun really just doing something expressive or or creative. Or, I got a a midi drum set. I'm gonna go downstairs. I'm gonna hook the midi drum set in, and I'm gonna have elements on my page in large in size based on how hard I hit the drum. That sounds like a ton of fun to make some of this stuff. So if you have any MIDI devices around, you can experiment with these. Otherwise, the audio synthesis API I have this, like, crappy little MIDI controller that's probably, like, $60. You can get them for super cheap. I got this. It's a simple little MIDI slider off of Etsy for, like, $20.30 bucks. You can get all kinds of cool stuff, out there to control MIDI and use that just for for fun projects. Who cares? Build some art. You can just make art.

Yeah. You're allowed to just make art.

Awesome. That was awesome. I I can't believe how much, like, how much of a well these APIs are sometimes. Right? It's just one or two little APIs in the browser, and it's just it goes all the way down, and it has such history as well.

Yeah. Totally. Right? So cool. Let me know what you think. If you're building anything like this, let us know. I will make my synth available or my my midi controller my in browser midi controller available. But, just to let you know, it does not generate sound on its own. You need to pipe it into a digital audio workstation, at least right now. I don't feel like I don't feel like doing audio synthesis ESLint is, like, what I want to use it for because whatever VST inside of, Ableton or whatever I have is gonna sound way better than anything that I could make myself with, you know, the web audio API. So either way, I'll make that available. We'll have a link in the show notes if you wanna check it out. That code will be open source too if you wanna see exactly how I did it. As always, let's get into sick picks and shameless plugs, Wes. What do you got for sick picks and the shameless plugs?

Yeah. I'm going to, sick pick a little microcontroller. This is the one that I use to program my Roomba.

Actually, this is one that I'm playing with a little bit of these, LED, arrays right now.

But these things are extremely fun to work on. They're a little microcontroller that has Bluetooth and Wi Fi built right into it.

You can power them via USB. You can power them via several other ways.

They're $5, which is unbelievable.

Yeah. There's lots of projects out there for home automation. There's lots of projects out there for, LED.

WLED is is one that runs on it. And Oh, what? It's just a it's a really fun thing. Like, if you're trying to, like, tinker and get into electronics for, like, probably for, like, $10, 5 dollars for the controller, $5 for some wires and whatnot, you can start building some some pretty fun stuff with it.

I'm very interested in this WLED.

Are you doing stuff with, like, pushing visuals to an LED panel? That's what I'm working on right now. So I've used WLED

for the mirror in our gym. So when I built the mirror in our gym, I put LEDs behind the entire thing, and then it does it can do, like, cool effects and, like, kinda backlight the mirror. And then my on air Mhmm.

There we go. That is a sign that says on air, and I just gutted it and put a ESP 32 and some LED strips in it, and I can control that however I want. But now I'm yeah. Now I'm dipping into these LED arrays That's what I want right there. I'm trying to figure I'm, like, how sick would that be if we built a website where you could just draw on the website and it would, in real time, paint to the LEDs? Yeah. So that's my idea, at least. There it's a lot of LEDs, though. So, like, the powering them is is a bit of a thing I gotta figure out yet because it's it's not just, like it's it's a lot of power that these things take. If you have them all at full Cook, you need, like, like, a massive power supply or or several of them. So I gotta gotta dip into that. Yeah.

Wow. Yeah. That's interesting, man. I'm fascinated about that stuff. Let's,

let me update or give me an update on that once you're you're cooking a little bit more. I'm I'm super interested. Yeah. You should just just order some order a couple of these and, at least, like, get it running so you can kinda get in the headspace of it because it's I bet you can probably get it up and running in, like, twenty three minutes. You flash the microcontroller in the browser.

Like, you don't even have to download, like, an IDE or, like, install Python or or any of that. You literally visit a website, plug it into USB, and it flashes it from the browser, which is it's using the Wes Serial API. It's unbelievable.

Man. Cool. Man, you do the cool stuff sometimes.

Awesome. So my sick pick is going to be I have after, what, many, many years working in computing on computers, I have finally given in and gotten a mouse. That's right. Alright. I never work with mice ever. I I'm a trackpad guy. You Node? I'm super trackpaddy.

We're Costco. My trackpad. Yeah. We're Costco. But you know what? When I broke my finger, the, the trackpad was giving me all kinds of RSI. I'm like, now's the time. I'm gonna get a mouse. I got the Logitech MX Master, the S 3 3 s three s.

Three s. Yeah. That's what I have as well.

And I gotta say, the coolest thing about this mouse, besides the fact that it's just nice and comfortable and all that, the software that Logitech made for this is crazy good. Yeah. It's actually pretty good. You think you gotta get into, you know, some third party to do some crazy stuff, but I didn't realize this, Wes. You can set every single button on this thing to respond to gestures.

So every single button you could hold and move the mouse in a direction, and it would trigger a different piece of functionality. So I have, like You can do it per app as well. Per app as well. I I have it. So, like, if I hold essentially, the there's the back and forward buttons are typically what they are. I have my forward button. If I hold that and hit to the right, it does paste. If I hit to the left, it does copy. And I've been just copying and pasting that way, and it feels so natural. It's crazy how natural it feels once you get used to all of, like, your own predefined setup. So, yeah, I'm impressed I'm impressed with the mouse, but I'm equally impressed with the software that this thing came with. With. So, shout out to Logitech for making something that's actually extendable and, like, customizable out of the box. If you're a tinkerer, this thing's a tinkerer's dream. I I just have been going nuts with customizing this thing. So I know I'm super late. Stuff. Yeah. Very late to the Sanity. GitHub Vercel. They're, like like, half going after, like, the gamer market, but but, like, as soon as I ordered this thing,

every single gamer I I did a TikTok on it, and every single gamer comes out of the woodwork being, like, oh, it's not fast enough refresh rate.

And, like, all this, like I I find it lags quite a bit when I'm killing noobs in my game. And it's, like, I'm not gaming with this thing. I'm clicking on buttons and and, like, highlighting text. And it's it's a totally different world when you're, like, targeting, like, web developers who don't necessarily they care more about ergonomics and productivity than they do, like, refresh rate or something like that. Neat stuff. Logitech.

Shout out. Shameless plugs. Head on over to YouTube. CJ released a a crazy good video on the most starred projects in JavaScript in '24.

Man, what a a banger of a video. So check that out. We'll link it available in the description of the show.

It's really fascinating because it's not just like, here's a list of the top 15 most starred projects.

It actually breaks down, like, why. It gets into what they did and why they had the year they had or why people cared about it this specifically in the last Yarn. And it gives you a lot of context to where we are today in 2025.

So well worth your time.

I learned a lot from it, so shout out to CJ for putting that together.

Sweet. It was very good video. Alright. Thanks everybody so much for tuning in, and we will catch you later.