I like the idea of both the game influencing the music and the music influencing the game.
Like, you have a directed graph of music, with audio on the edges, and the current game state determining which edge one takes out of a node. The game influences the music there.
But then you also have an “event track” on the edges that can be used to do cosmetic things like flash lights in the game or whatnot in time with the music, or, as with rhythm games, require the player to do things in time with the music. There, the music influences the game.
Yeah the newer music-based games are pretty cool. Maybe as stuff like OpenRGB becomes more standard we could also see that tied into game/music events more as well
Maybe as stuff like OpenRGB becomes more standard we could also see that tied into game/music events more as well
That’s possible, though I should note that control of external lighting has been around for a while, and we haven’t really seen it become a standard thing in games. I think that the problem is less lack of a protocol and more that similar configurations aren’t commonly available or that people don’t feel that there’s a great way to incorporate it into games. And it’s not even specific to lighting, but to devices other than the monitor and speakers that provide some kind of sensory input to people. After all these years, the norm is still:
You can see what’s on the display.
You can hear what the speakers play.
For the lifetime of the personal computer, maybe fifty or sixty years at this point, the only real change in the norm has been for fidelity improvements to what’s on the display and fidelity improvements to what’s playing though speakers (well, headphones more-frequently these days). And people have tried a lot of things.
I’d guess that keyboard backlighting is one of the more-common things, and it doesn’t get used much.
RGB LEDs on the PS5 controller are standard, but I don’t own a PS5, so I dunno how commonly they’re made use of.
DMX512 is a standard for hooking up stage lighting devices like strobes and spotlights and colored lighting and smoke machines and stuff – you’d use it to drive stuff like live performances, lightshows at clubs, stuff that runs in time with music like this. USB-to-DMX512 transceivers are pretty affordable, and Linux can handle them.
Multiple monitors. That’s seen some uptake in flight sims (though I think that VR headsets will displace that), but outside of that, doesn’t seem to have really caught on. And it’s a (moderately) expensive way of doing things.
You can get external small displays, of the sort that Matrix Orbital sells. Logitech tried incorporating them into “gaming keyboards” with integrated miniature displays for a while; it looks like they dropped that in 2013, with the G19s. There have been a few games that have had uptake. It did not take off.
Various forms of 3D displays coupled with glasses did not do very well.
VR headsets. We’ll see where this goes. I remember an earlier VR headset craze back towards 2000 that didn’t seem to take off, and I’m not sold that the current one is going to become the norm either. Though at least in terms of sensory input to your eyes, that can (given sufficient brightness, contrast, resolution, etc) replace all the other visual stuff here. If I had to bet on one external sensory device becoming the norm in the next decade, I’d bet on VR headsets, and I’m honestly not that bullish on even them. Maybe if they become cheaper and higher-resolution. I will say that I think that they pair well with laptops, since they address the limited screen size well.
Force feedback – not simply rumble motors that use offset-weight motors, which is what I think a lot of people think of when they think of force feedback – doesn’t seem to me to be very healthy. Used to be that there were joysticks that simulated the resistance in pre-fly-by-wire aircraft, like the Microsoft Sidewinder Force Feedback Pro, but those seem to have fallen out of fashion. There are still some steering wheels that provide resistance in response to driving. The PS5 controller has, as I understand it, voice-coil actuators and mechanical resistance on the triggers. But in general, I feel like force feedback has been something where people have talked about potential for a long time, but which hasn’t really become the norm.
Rez famously came with a vibrator; these days, buttplug.io is a FOSS layer that can drive a bunch of sexual stimulation hardware (not, despite the name, just buttplugs). That’s had some game uptake, but it’s pretty limited. People have been working on computer-controlled sex devices since the mid-1970s, and it still hasn’t become the norm.
Temperature. I know that Disney’s done some things like have water jets that spray at people for their rides. I imagine that someone could rig something up with a computer-controlled heater/air conditioner, make a cave in-game feel cool and clammy. Kind of hard to both set up in arbitrary places and get a common experience across people. There have been people playing with using thermocouples for VR.
Scent. Again, I recall reading that Disney worked on incorporating this into Fantasia presentations many decades back. I suppose that one could have some kind of device that kept a bank of essential oils and then could mist a given set, though I think that how long it lingered would depend a lot on the surrounding environment. Maybe if you had some kind of vent by the nose or something that passed air directly by the face. There are apparently companies trying to sell devices to do this sort of thing.
Motion simulators exist, can be used for stuff like flight sims, but I think that the price point is probably going to remain too high for them to really take off for the mass market.
I mean, in theory, I could imagine someone writing some kind of software layer that cleverly abstracts all these sorts of systems, has games drive events. But in practice, I’ve been watching external devices that aim to provide a new source of sensory input come out for a long time, and they have pretty depressingly and consistently failed to take over the market.
I think that realistically, in terms of having many games providing sensory input to people, it’s gonna be the display and audio, and maybe VR headsets for the foreseeable future.
I’ve been starting to see it available on a bit more hardware these days, mostly peripherals etc that have blinky lights, but I’m hoping that it’ll gain more traction over time. Why reinvent the wheel, after all?
Or one step further, let the music slightly alter the timing of non-cosmetic events. E.g. player throws grenade, game sends approximate explosion time to the music player, which transitions to an appropriate build up and sends the exact explosion time back to the game. Explosion visuals, SFX and music all line up perfectly.
I liked the idea of DirectMusic. Gameplay influenced music scores are an under-used concept
I like the idea of both the game influencing the music and the music influencing the game.
Like, you have a directed graph of music, with audio on the edges, and the current game state determining which edge one takes out of a node. The game influences the music there.
But then you also have an “event track” on the edges that can be used to do cosmetic things like flash lights in the game or whatnot in time with the music, or, as with rhythm games, require the player to do things in time with the music. There, the music influences the game.
Yeah the newer music-based games are pretty cool. Maybe as stuff like OpenRGB becomes more standard we could also see that tied into game/music events more as well
That’s possible, though I should note that control of external lighting has been around for a while, and we haven’t really seen it become a standard thing in games. I think that the problem is less lack of a protocol and more that similar configurations aren’t commonly available or that people don’t feel that there’s a great way to incorporate it into games. And it’s not even specific to lighting, but to devices other than the monitor and speakers that provide some kind of sensory input to people. After all these years, the norm is still:
You can see what’s on the display.
You can hear what the speakers play.
For the lifetime of the personal computer, maybe fifty or sixty years at this point, the only real change in the norm has been for fidelity improvements to what’s on the display and fidelity improvements to what’s playing though speakers (well, headphones more-frequently these days). And people have tried a lot of things.
I’d guess that keyboard backlighting is one of the more-common things, and it doesn’t get used much.
Monitor bias lighting has been around for a while.
RGB LEDs on the PS5 controller are standard, but I don’t own a PS5, so I dunno how commonly they’re made use of.
DMX512 is a standard for hooking up stage lighting devices like strobes and spotlights and colored lighting and smoke machines and stuff – you’d use it to drive stuff like live performances, lightshows at clubs, stuff that runs in time with music like this. USB-to-DMX512 transceivers are pretty affordable, and Linux can handle them.
Multiple monitors. That’s seen some uptake in flight sims (though I think that VR headsets will displace that), but outside of that, doesn’t seem to have really caught on. And it’s a (moderately) expensive way of doing things.
You can get external small displays, of the sort that Matrix Orbital sells. Logitech tried incorporating them into “gaming keyboards” with integrated miniature displays for a while; it looks like they dropped that in 2013, with the G19s. There have been a few games that have had uptake. It did not take off.
Various forms of 3D displays coupled with glasses did not do very well.
VR headsets. We’ll see where this goes. I remember an earlier VR headset craze back towards 2000 that didn’t seem to take off, and I’m not sold that the current one is going to become the norm either. Though at least in terms of sensory input to your eyes, that can (given sufficient brightness, contrast, resolution, etc) replace all the other visual stuff here. If I had to bet on one external sensory device becoming the norm in the next decade, I’d bet on VR headsets, and I’m honestly not that bullish on even them. Maybe if they become cheaper and higher-resolution. I will say that I think that they pair well with laptops, since they address the limited screen size well.
Technically, there have been some changes in speakers that have met with some level of uptake, but none have become the norm. You have things like chairs with backrest-mounted subwoofers or surround sound.
Tactile feedback touchpads.
Force feedback – not simply rumble motors that use offset-weight motors, which is what I think a lot of people think of when they think of force feedback – doesn’t seem to me to be very healthy. Used to be that there were joysticks that simulated the resistance in pre-fly-by-wire aircraft, like the Microsoft Sidewinder Force Feedback Pro, but those seem to have fallen out of fashion. There are still some steering wheels that provide resistance in response to driving. The PS5 controller has, as I understand it, voice-coil actuators and mechanical resistance on the triggers. But in general, I feel like force feedback has been something where people have talked about potential for a long time, but which hasn’t really become the norm.
Rez famously came with a vibrator; these days, buttplug.io is a FOSS layer that can drive a bunch of sexual stimulation hardware (not, despite the name, just buttplugs). That’s had some game uptake, but it’s pretty limited. People have been working on computer-controlled sex devices since the mid-1970s, and it still hasn’t become the norm.
[continued in child]
[continued from parent]
Temperature. I know that Disney’s done some things like have water jets that spray at people for their rides. I imagine that someone could rig something up with a computer-controlled heater/air conditioner, make a cave in-game feel cool and clammy. Kind of hard to both set up in arbitrary places and get a common experience across people. There have been people playing with using thermocouples for VR.
Scent. Again, I recall reading that Disney worked on incorporating this into Fantasia presentations many decades back. I suppose that one could have some kind of device that kept a bank of essential oils and then could mist a given set, though I think that how long it lingered would depend a lot on the surrounding environment. Maybe if you had some kind of vent by the nose or something that passed air directly by the face. There are apparently companies trying to sell devices to do this sort of thing.
Motion simulators exist, can be used for stuff like flight sims, but I think that the price point is probably going to remain too high for them to really take off for the mass market.
I mean, in theory, I could imagine someone writing some kind of software layer that cleverly abstracts all these sorts of systems, has games drive events. But in practice, I’ve been watching external devices that aim to provide a new source of sensory input come out for a long time, and they have pretty depressingly and consistently failed to take over the market.
I think that realistically, in terms of having many games providing sensory input to people, it’s gonna be the display and audio, and maybe VR headsets for the foreseeable future.
Here is an alternative Piped link(s):
this
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source, check me out at GitHub.
I’ve been starting to see it available on a bit more hardware these days, mostly peripherals etc that have blinky lights, but I’m hoping that it’ll gain more traction over time. Why reinvent the wheel, after all?
Or one step further, let the music slightly alter the timing of non-cosmetic events. E.g. player throws grenade, game sends approximate explosion time to the music player, which transitions to an appropriate build up and sends the exact explosion time back to the game. Explosion visuals, SFX and music all line up perfectly.
Adaptive music is a niche I happily fill with my games ;)
EA used to do this even before Microsoft!
They have their own music system called “Pathfinder” which controls music interactively in little chunks.
I believe it was first used in NFS 2, which came out before DirectMusic.
If you’ve heard the pursuit music in NFSMW 2005 or Carbon’s Canyon music, that’s Pathfinder! It was also used in Medal of Honor and Red Alert 3.
It was kinda a thing even in the early SNES days, at least in a basic way Mario world would add a bit of a drumming back-beat when you jumped on Yoshi
Come to think of it, it’s surprising that’s not more prevalent today.
I’ve run into it in a few games, but it’s often subtle.