RuneStonks

Well, I finally did it--I snapped. And by "snapped," I mean that I attempted to make a neural network that predicted RuneScape 3 Grand Exchange (RS3 GE) item (or "stock") prices.

Okay, that's great, you may say, but why?

Well, I'm glad you asked. As of late, I've noticed that the predictive power of neural networks has taken the world by storm. Haven't you noticed this too? Anyway, in creating this project, I merely wanted to learn more about neural networks, and, in particular, models that analyze sequences and time-based phenomena (like stock market prices).

So, yes, this is just for learning purposes, though I do hope that some of what I'm doing here is useful, or at least inspiring to people who may have more time and expertise (and compute) than I do. Theoretically, if one could predict the prices of items in the RS3 GE with accuracy, well, then...that would be something! One imagines that this type of thing could be done on different exchanges...perhaps ones that involve actual money...Who knows, though? One also imagines that there are a lot of people who would like to know, for instance, just how much they can expect to pay for an abyssal whip tomorrow in the RS3 GE and stop there, completely happy, fulfilled and unperturbed.

Wherever you fall on the gamer spectrum (full disclosure: I've never played RS3 at length, and I haven't played RS Classic at length in about 15 years, give or take), my adventure and meandering prose may interest you.

So, yes, I wanted to build a neural network that would predict RS3 "stock" prices--or the prices of items sold on the GE at a given day. For those who don't know what the GE is, it is a market introduced in the game of RuneScape where players can buy and sell items at market prices, much like the New York Stock Exchange, except far less...material. No, I'm not taking a shot at gamers. I'm just saying that all of this is purely digital and that the "money" you spend or gain in the GE is in-game currency, and therefore...well, you get the idea.

But yes, the basic idea is that you buy and sell items for in-game currency at prices determined by supply-and-demand forces. Great! Glad we could clear that one up.

And how did I start this adventure? With ChatGPT, of course! More precisely, I used ChatGPT o1-preview (the latest, bleeding-edge model) to help me when I got stuck.

And I definitely got stuck...

For instance, I had no earthly idea what I was doing. So I just asked it how to do what I wanted to do. Novel concept, eh?

ChatGPT was very helpful. It provided an outline of what I needed to do, and, when pressed, provided some code for me. With a few adjustments, I was able to create a working model in very little time. I would emphasize, though, the word "working," in opposition, perhaps, to the word "effective."

Yes, I'll come out with it: It's not a very good model. But the experiment still yielded interesting results, and I learned a ton. If you're rolling your eyes at me for wasting your time right now, you may want to know that, actually, the model I prompted into existence predicted some prices very well. (Furthermore, what it really came down to is the amount of time/compute on my hands...I'm a busy many with things to do. I can't train a complex model on my MacBook Pro 16GB all day and night. We'll get to the potential fixes for the model's "flaws," though, if you will but bear with me.)

So, yes, with minor adjustments, I was able to copy/paste a sophisticated lady--I mean model--into a Jupyter notebook and train it on a Kaggle dataset of RuneScape 3 Grand Exchange data.

If you don't know where Jupyter is or how to Kaggle, it's fine. These are just tools with funny names. It hardly matters if you're acquainted with them.

Anywho, my model--a sequential model--was trained over 100 epochs. A sequential model is a neural network that works well with time-sequenced categories, like item prices or stock prices, that evolve and change over periods of time. Training epochs are aptly named: The amount of training epochs you subject your neural network to can be thought of as the amount of times that it tries to accomplish a task and trains itself to do better next time. Think of it like shooting baskets in basketball. You don't become a pro overnight. You have to try again and again, from many different angles, correcting posture and many other variables. An epoch is like a basketball shot--it really doesn't do to just try once!

So, yes, 100 training epochs. And this was the kicker--had I done more training epochs, I believe I could have reduced my percentage of error and trained a better model, but I didn't have time. But here are some neat things that I learned...

First of all, I was skeptical about the use of neurons. To my untrained brain, it seemed evident that there should be one neuron in each layer per item in exchange. This seemed logical because I was thinking of neurons as data points. But my thinking was misguided. I knew I was wrong--who ever heard of implementing 5,000-neuron per layer for a simple task like this? Preposterous!

So I asked ChatGPT, as I always do. It had a very good explanation, as it always does. It said that my task was like predicting the next part in a story; I didn't need a neuron for each character. Neurons aren't characters. Rather, they're helpers--helpers who, when they organize together, can help predict things. Sure, the more complex a task, the more neurons you will need, but it's more about finding the sweet spot in the balance of neurons vs complexity of your task. So, no, thousands and thousands of neurons were unnecessary. Maybe just a few were needed.

And that's something awesome that I learned!

Okay, so what else? What about predicting the Grand Exchange?

Well, as I said, I had mixed results. With a mere 100-epoch training regimen, I emerged with a training data loss value of 0.0015 and a test data loss value of 0.0215, both on a scale between 0 and 1. What this loss value measures is the difference between the predictions the model makes and the actual values that it tests against (yes, we divided the dataset into training and testing values just for this purpose, as one always does).

Furthermore, after analyzing the RMSE values for each item (that is, the Root Mean Squared Error), I found that, actually, some items came pretty close to their predicted test values. Others, though...well...not at all. For the uninitiated (like I was before I Googled the term), RMSE is a measurement of the predictive accuracy for each item (or stock). A lower RMSE is a good thing. A higher one...generally indicates that you're a noob like me and don't know what you're doing yet.

Again, I attribute this to the planning of the model. If I had run more training epochs, surely this would have closed the gap even more, right? It may have to do with the type of model that I used, as well. I'm learning that a lot of factors go into sequential prediction. There are really a ton of different ways to look at problems like this!

Surprisingly, I noticed that some items had a RMSE value of basically zero. But this turned out to be because the price never fluctuated--like ever. So, I decided to look at something more interesting: The bottom 50 nonzero RMSE values.

A screenshot of my notebook page showing the top 30 nonzero RMSE values

At the lowest, I found that "2/3 strawberry cheesecake" had an RMSE value of 0.000053! Not too shabby! At the bottom of the list, #50 was "Woodcutting potion (1)," which had an RMSE of 0.031587. Again, not just awful, though I bet someone reading this could do a lot better...

Okay, so apart for all the technical jargon, I managed to whip up a pretty neat little model, with the ever-so-valuable help of ChatGPT. If you want to check it out and see what it's predictions are like, I have provided the Jupyter notebook on GitHub. All you need to do is follow the instructions, and you too can predict the prices of RuneScape 3 Grand Exchange items! Yes, why shouldn't it be open-source?

Thanks for reading! Let me know what you think on in the comments or on X.com!

Thinking about Invention