This could be amazing for accessibility. Say that the only available input is voice.
If a player could say "down-b, dash, okay dodge and shoot while I think, see if we can get in some shields" to where it really matters what and when the user is saying things as opposed to "okay win this match please", the player and the opponent could both have a sense of a fair fight.
Like, compared to having a STT model that adds so much latency to where there's no shot, this could be actually useful.
My only question is, it seems like Reyn1/2 are recieving the same voice inputs? They're each doing the "same" thing for every command, just right/left, up/down, taunt/dash dance. Like, it's not as if one is shooting while the other is edge-hopping or something.
There are two instances where the commands differ (up-b/drill him, and I forget the other one), but I wonder if that's the model choosing different commands based on the single input since all of the others are seemingly mirrored.
We did an exploration into collaborative agents in Super Smash Bros Melee.
We wanted the ability to talk to and coordinate with it in real time.
We used a combination of imitation learning (we labeled thousands of hours of data using english language predicates) and RL to create an action policy for doubles in Melee.
If a player could say "down-b, dash, okay dodge and shoot while I think, see if we can get in some shields" to where it really matters what and when the user is saying things as opposed to "okay win this match please", the player and the opponent could both have a sense of a fair fight.
Like, compared to having a STT model that adds so much latency to where there's no shot, this could be actually useful.
My only question is, it seems like Reyn1/2 are recieving the same voice inputs? They're each doing the "same" thing for every command, just right/left, up/down, taunt/dash dance. Like, it's not as if one is shooting while the other is edge-hopping or something.
There are two instances where the commands differ (up-b/drill him, and I forget the other one), but I wonder if that's the model choosing different commands based on the single input since all of the others are seemingly mirrored.
We wanted the ability to talk to and coordinate with it in real time.
We used a combination of imitation learning (we labeled thousands of hours of data using english language predicates) and RL to create an action policy for doubles in Melee.
Full write up (and model weights) on our website: https://www.frisson-labs.com/reyn
Built upon our previous work creating a competitive Melee agent: https://www.frisson-labs.com/faynt