War Era Stats

Pretty cool game. Even though I'm only like level 6 and have made little to no progress so far, I still think it's something nice to play without having too much stress. It's also kinda crazy that I'm stopping my slumber of posting projects on here for a game I started playing less than 48 hours ago... but here we are.

What is War Era?

Forgive me if I'm writing something stupid here for the people that have actually been playing this game for the past half a year. From my experience so far, War Era is a game where countries fight over territory to boost production of certain goods. The higher the boost, the more companies settle in your country, the more money, etc. Next to that, the game features a full economy where you can work, start your own businesses, and trade items needed for progression.

In the end it's all about money (just like in real life). Money can buy better equipment, food, armor, bullets & weapons. Money can also be used to set bounties, offering other players from outside your country to fight for your cause in return for cold hard cash.

The game has a really nice look and feel to it, and it's satisfying to perform certain actions in the game (props to the devs for this).

As a new player, the start of the game is a bit slow though. The lack of knowledge and resources gives you the feeling of being "lost" and trying to find ways to fill those holes. For me, the solution was to start working on this project.

Data source

The first idea that came up to me was: "hey, the website is for sure using something to send data to the client. I can read that and scrape the website myself". I already had devtools open, skimming the requests my browser was making. Around that time I also asked if there was a Discord. Turns out there was. When I joined, I saw the "api" channel... no way this game also gives a public API..

Even though each one of these says "POST" in front of them, they're all GET requests. Under the hood it's running a TRPC server, which is quite interesting. I would love to know why the devs chose to use RPC over simply a REST server, or even a websocket connection for what it's worth. The company I currently work for is moving all RPC services into the trash right now..

The interesting part of their API (which might be a feature from TRPC, I haven't actually checked) is that you can make "batch requests". So if an API endpoint is for example /trpc/country.getAllCountries, you could also combine that request with a different function like item prices, like so: /trpc/country.getAllCountries,itemTrading.getPrices?batch=1. In that case you would retrieve all the countries and the prices of all the items tradable.

This isn't documented anywhere, so I did not use this in this application. I found out about this by the skimming I did before.

Data scraping

So, for the past two weeks I've been running my own n8n instance. It seems interesting. It's mostly promoted right now with the buzzword "AI", but it's lots more than that, it's a low-code development tool. Personally, I've had my fair share of low-code development and I always feel like it removes the fun of the actual coding part. However, it does help you quickly set up data streams. Oh, and it allows you to set up "code blocks" along the way, so I usually can't stop myself from just doing everything by code anyway.

Market data

For the market data, I first make a request to get item prices. This API returns an object containing an item-name-to-price mapping (see the example by scrolling up to the previous API response example image). This also gives me all the item names, which I split out and then make a separate request for each item to the tradingOrder.getTopOrders function, returning the current asks & bids for that item.

I transform this data into something that looks more like an order book, and calculate the effective prices per batch (10, 50, 100, 1000 of the item). That way, the user can see what the average ask/bid is for, for example, 100 of the item, which can differ a lot from just the most recent price, or the mid price of the first bid & ask.

Getting transactions

The transactions are the money movements in the game and these are not all the movements in the game btw! The current API, transaction.getPaginatedTransactions, does not include gear sales, which is a little sad. I did skim the actual function required to retrieve those transactions and tried to make a request to it. However, I received an unauthorized error. I'm assuming they made that function unavailable on purpose, and left it at that.

Since workflows in n8n don't keep state, I make use of a datatable with a row to keep the "latest transaction time" variable. This is used to determine which transactions are new and which ones have already been seen before.

After this, the transactions go into a switch statement, where the useful data gets extracted from each case and added to the MongoDB database. All the way at the bottom of the switch, the "fallback" case is a Telegram message node that sends me a message with the data if a new transaction type occurs that the system does not recognise (yet).

From all these transactions we also gather the user IDs. These are sent to a sub workflow that checks if the user already exists in our database. If not, a request is sent out to War Era to get the user's username & profile picture. If they are already present in the database, we check when the last update was, if it's been over 24 hours, they also get refetched from War Era.

Making the data available

I've also... done this... with n8n workflows. In my defense, I did not research what the concurrency limit is on n8n, but it's incredibly low. You should practically assume a maximum of ~20 requests a second (which is incredibly low).

I feel like the n8n workflow names speak for themselves and don't need further elaboration.

Frontend

So now, with all the data in the database and growing, and the data "available" through the APIs, it's time to make a nice UI for it. I've decided to use Svelte Kit. It's basically just simple HTML, CSS & JS/TS with some additional features. AND it's really performant.

The project uses SSR, so the first visit will always be rendered server-side, containing the exact HTML. After that, pages get rendered in your browser with the data provided by the Svelte Kit server.

As I said before, the APIs created with n8n workflows aren't really performing well. For this reason I added caching on the Svelte Kit server side. The cache would invalidate after 1 minute, and be fully "unavailable" if the data was older than 3 minutes.

Eventually I noticed this wasn't what I wanted. This would mean that if a user visited the page, and if the data was older than 3 minutes, it would take literal seconds before the page loaded.

So instead, I used one of the features of Svelte Kit. The "streaming with promises" feature, to be precise. It allows you to make a call on the server side which returns a promise, and then "transfer" that promise to the frontend.

Under the hood this works pretty neat. Instead of just giving an HTTP response and that's it, the connection stays open. When the promise resolves on the server side, the server adds the data to the end of the response and then closes the connection. That last little push gives the last update. This is also why you might see the loading bar still "loading" while the page is already showing you data. In that case, the data is stale, and the server is working towards bringing you the newest data.

The only problem is the /items page on the website. That page needs so much data to render properly that it makes about 50 API calls to the n8n workflows. Half of the time that even fails. I've upped the caching of the items page to 30 minutes since it's more of an overview page than anything. But yeah, this issue would need to be fixed later on by simply moving away from n8n.

Next steps

There are some low priority things on my bucket list:

• Remove n8n as the "API". The Svelte Kit project is more than capable of just connecting to the MongoDB server by itself.
• Do scraping through a custom made program in, for example, Go.

These would allow me to simply remove n8n from the equation and the big liability it is.

But FOR NOW, I just want to simply enjoy this game.