ManiacalSpyBot - Proof that I need to be stopped

So near the end of 2020 as I was continuing to figure out what to do after graduating in the middle of a pandemic I decided to work on a "small" project that would keep my skills from getting rusty as I searched for jobs.

I decided I wanted to start in MonoGame, since it makes you do a lot of things yourself I figured it would help keep more skills sharp than if I just worked with prefabs in Unity or blueprints in Unreal. The game itself wasn't anything groundbreaking, but it gave me a chance to practice my skills. Using what I learned from my math classes to be able to calculate the points for a triangle and calculate the overlap myself, pulling from my OOP classes to ensure that I kept the code clean, and drawing experience from my graphics classes to write logic for vertex buffers for colors to gradually change as opposed to just flipping to a new color. All of my classes were coming together to cultivate this:

https://video.wixstatic.com/video/d7e15c_d957521be52e4d7786dad9b49cbdea65/720p/mp4/file.mp4

Now that I had clearly revolutionized all modern gaming for the rest of time, I thought about what I wanted to do next in this project. As I was cleaning up some code in this project I was looking at the player controller. I made sure to have an interface between the controller class and the input handler, and as I was cleaning up commented out code I joked to myself that since I did that I could really turn anything into a viable controller for this game by switching out the input handler, even something like a Twitch chat.

Even a Twitch chat.

So naturally, having piqued my own curiosity, I looked into how I would go about including Twitch integration. I searched the internet and found a python library called twitchio that would let me create a bot to read the chat of a Twitch stream and I could execute code based on the content of the message. I wanted to give myself an extra challenge by not taking the simpler way out of sending key inputs through libraries like PyWinAuto, since that would still rely on the basic input handler. So I faced the issue of how I would be able to make my python script be able to interact with my game written in C# without going through keyboard inputs. The answer I decided on came once I found out about C# processes.

Without getting too into the specifics, when I pushed a specific key while the game was running, the player controller would instantiate an input handler that created a process that ran the chat reader's python script, and redirected the standard output into a stringbuilder, so all the python script had to do is print out specific keywords when it encountered specific chat messages. Then the input handler would read the printed output from the stringbuilder and output the same flags as the standard input handler, so the controller itself would only have to handle those flags, regardless of their source. I'd include footage but unfortunately I don't have any available to me.

After getting used to the python library for the chat reader, I thought I'd expand on it a bit more, as I had recently begun streaming again and I thought the idea of chat integration would be entertaining. So I broke out the chat reader from the simple game, and so begins the story of ManiacalSpyBot.

As I expanded on the bot, I wanted it to be more than just a basic script that silently read chat messages, I wanted it to be an actual chat bot. This meant I had to learn how to do more than just send a basic message into chat, which means I needed to learn how OAuth tokens worked. A few weeks of reading went by and I was able to allow the bot to do things like edit my stream title through Twitch's API, I had learned how to make the bot write to a SQL database so that users can do things like save quotes out of context for others to call up later. I had also gone with the approach of sending keycodes to my computer when I enabled chat integration, which involved me learning how to send specifically DirectX keycodes, since a large number of games wouldn't accept usual keycode inputs sent by the pyhton script.

Within a few months I had a python script that was able to connect to a web API, call a library I found to convert python objects to C types to send keycode inputs as structs directly through user32.dll. And on top of that it was also able to read/write to a SQL database for sake of keeping track of various things like quotes, how long a user has followed the stream, and so on.

But I wasn't done yet.

I had streamed a few games using the bot, including a playthrough of Pokemon Emerald where I let the chat control me every time I walked away to use the restroom or drink some water. But I knew that all the integration was essentially the equivalent of handing twitch chat a keyboard and letting them push buttons on it like a monkey on a typewriter. But I wasn't satisfied with that. I wanted to give the chat the ability to do things that someone standing next to me who could just reach my space bar couldn't do. At the time I was streaming Valheim, an Indie came that had recently come out and had many people in the game modding community, so I started to look into what else I could do.

Quickly I ran into a similar issue I had before but with a new twist. Valheim was made in Unity, and my bot was still written in python, however, since I had flushed out the bot to be more than just a chat parser, it wouldn't work to just use the same process trick I used in the past. I looked into ways to call C# functions directly from python, but those ideas were quickly dismissed due to the fact that the game and the bot could run completely independently of each other. A few weeks passed as I researched ideas while starting a new job working in an IT position that was heavily involved in SQL, something I had never used before creating ManiacalSpyBot, so I can credit my current job to that. Eventually though I found my solution, and that was sockets.

The issue I was running into was that I couldn't have one program own and control the other like I had done in the past with processes, and the two programs were written in different languages, sockets solved both of these issues. I wrote a small mod for Valheim that made it so when the game started it opened a local socket on my machine. Then I turned to the bot and added a special command that only I could call that would create an object in python that connected to that socket. So now the two programs had a point where they could communicate, next came the language issue.

The bot continued to act as it normally did, only now, if the special input handler that connected to the socket was active, the bot would pass all chat messages to that handler as well. The handler would then parse the message and compare it to a list of keywords it had, which was populated based on the name of the game included in the initial chat command. And if the handler found a match, it would create a JSON object that included the user that sent the keyword, as well as the name of the corresponding command that the game itself would handle. Then, on the other end of the socket, the game mod I created would parse the JSON object sent, and based on the command included in the object, would call a corresponding function through the game's mod API. So I was free to do whatever I wanted through the modding API and could assign it to a keyword in the chat. This could clearly only lead to good things.

Okay, so maybe giving chat a "spawn skeletons" option wasn't my best idea. Neither was letting them set me on fire on demand. Excuse the giant pause button, I don't have any old stream footage anymore so this screenshot is all I have.

Regardless, since the interaction is handled through JSON objects, that meant that if I decided to rewrite my bot in another language, I wouldn't have to rewrite my Valheim mod. More importantly though is that, since the bot always created a JSON object, it didn't matter what the receiving end of the socket was coded in, as long as it allowed me to open a socket, and the language supported converting objects to JSON. So without changing anything in my bot other than creating a file to store the keywords in, I could add in-depth chat interaction to any game I streamed. Below is an example with Risk Of Rain 2, note the twitch chat in the overlay in the bottom right, then the game chat in the bottom left for my benefit so I can see who did what without looking at my streaming software.

https://video.wixstatic.com/video/d7e15c_a688f2859a3e403d83b55d6dbc2a7e73/720p/mp4/file.mp4

(Excuse the bad gameplay, I was moving with my keyboard with one hand and typing the chat messages on my phone with the other)

So to recap, ManiacalSpyBot is a Python script I've made that connects to a web API to read Twitch chat messages and update stream information. It also connects to a SQL database stored on my computer, and parses through strings to dynamically load python modules containing game input data, which then either create C-type structs containing DirectX keycodes and/or virtual mouse inputs (I glossed over that bit earlier), or creates a JSON object to be sent over a web socket to be read and converted into a (potentially) different language by another program on the other end of the socket.

So to repeat myself: I need to be stopped. Because I know I won't stop doing this on my own.