The War Robots celebrates its 10-year anniversary this April! And to this day, we continue developing and supporting it, not only releasing new features for our players, but also improving it technically.
In this article, we’ll discuss our many years of experience in the technical development of this large project. But first, here’s a snapshot of the project as it stands:
Hundreds of thousands of DAU (Daily Active Users)
Hundreds of millions of installations
Tens of thousands of simultaneous 12-person matches
Available on four major platforms (iOS, Android, Steam, Amazon)
Tens of thousands of supported devices, including Android and PC
Hundreds of servers
Approximately 20 client developers, 9 server developers, a cross-project development team of 8 people, and 3 DevOps
About 1.5 million lines of client-side code
Using Unity starting from version 4 in 2014 and currently using version 2022 LTS in 2024.
To maintain the functionality of this kind of project and ensure further high-quality development, working only on immediate product tasks is not enough; it’s also key to improve its technical condition, simplify development, and automate processes – including those related to the creation of new content. Further, we must constantly adapt to the changing market of available user devices.
The text is based on interviews with Pavel Zinov, Head of Development Department at Pixonic (MY.GAMES), and Dmitry Chetverikov, Lead Developer of War Robots.
The beginning
Let's go back to 2014: the War Robots project was initially created by a small team and all the technical solutions fit into the paradigm of rapid development and delivery of new features to the market. At this time, the company didn’t have large resources to host dedicated servers, and thus, War Robots entered the market with a network multiplayer based on P2P logic.
Photon Cloud was used to transfer data between clients: each player command, be it moving, shooting, or any other command was sent to the Photon Cloud server using separate RPCs. Since there was no dedicated server, the game had a master client, which was responsible for the reliability of the match state for all players. At the same time, the remaining players fully processed the state of the entire game on their local clients.
As an example, there was no verification for movement — the local client moved its robot as it saw fit, this state was sent to the master client, and the master client unconditionally believed this state and simply forwarded it to other clients in the match. Each client independently kept a log of the battle, sent it to the server at the end of the match, the server processed the logs of all players, and awarded a reward and sent the results of the match to all players.
We used a special Profile Server to store information about player profiles. This consisted of many servers with a Cassandra database. Each server was a simple app server on Tomcat and clients interacted with it via HTTP.
Drawbacks of our initial approach
The approach used in the gameplay had a number of disadvantages. The team, of course, knew about these, but due to the speed of development and delivery of the final product to the market, a number of compromises had to be made.
Among these disadvantages, first of all, was the quality of the master client connection. If the client had a bad network, then all the players in a match experienced lag. And, if the master client was not running on a very powerful smartphone, then, due to the high load placed on it, the game also experienced a delay in data transfer. So, in this case, in addition to the master client, other players suffered as well.
The second drawback was that this architecture was prone to problems with cheaters. Since the final state was transferred from the master client, this client could change many match parameters at its discretion, for example, counting the number of captured zones in the match.
Third, a problem exists with matchmaking functionality on Photon Cloud: the oldest player in the Photon Cloud Matchmaking room becomes the master client, and if something happens to them during matchmaking (for example, a disconnection), then group creation is negatively impacted. Related fun fact: to prevent matchmaking in Photon Cloud from closing after sending a group to a match, for some time, we even set up a simple PC in our office, and this always supported matchmaking, meaning that matchmaking with Photon Cloud could not fail.
At some point, the number of problems and support requests reached a critical mass and we started seriously thinking about evolving the technical architecture of the game – this new architecture was called Infrastructure 2.0.
The transition to Infrastructure 2.0
The main idea behind this architecture was the creation of dedicated game servers. At that time, the client’s team lacked programmers with extensive server development experience. However, the company was simultaneously working on a server-based, high-load analytics project, which we called AppMetr. The team had created a product that processed billions of messages per day and they had extensive expertise in the development, configuration, and correct architecture of server solutions. And so, some members of that team joined the work on Infrastructure 2.0.
In 2015, within a fairly short period of time, a .NET server was created, wrapped in the Photon Server SDK framework running on Windows Server. We decided to abandon Photon Cloud, keeping only the Photon Server SDK network framework in the client project, and created a Master Server for connecting clients and corresponding servers for matchmaking. Some of the logic was moved from the client to a dedicated game server. In particular, basic damage validation was introduced, as well putting match result calculations on the server.
The emergence of microservices
After successfully creating Infrastructure 2.0, we realized that we needed to continue moving towards decoupling the responsibilities of services: the result of this was the creation of microservice architecture. The first microservice created on this architecture was clans.
From there, a separate Communication service appeared, and this was responsible for transferring data between microservices. The client was taught to establish a connection with the “hangar” responsible for meta mechanics on the game server and to make API requests (a single entry point for interacting with services) using UDP over Photon Cloud. Gradually, the number of services grew and, as a result, we refactored the matchmaking microservice. With this, the news functionality, Inbox, was created.
When we created clans, it was clear that players wanted to communicate with each other in the game – they needed some sort of chat. This was originally created based on Photon Chat, but the entire chat history was erased as soon as the last client disconnected. Therefore, we converted this solution into a separate microservice based on the Cassandra database.
Infrastructure 4.0 and server locations
The new microservice architecture allowed us to conveniently manage a large number of services that were independent of each other, which meant horizontally scaling the entire system and avoiding downtimes.
In our game, updates occurred without the need to pause servers. The server profile was compatible with several versions of clients, and as for game servers, we always have a pool of servers for the old and new versions of the client, this pool gradually shifted after the release of a new version in application stores, and game servers disappeared over time from the old version. The general outline of the current infrastructure was called Infrastructure 4.0 and looked like this:
In addition to changing the architecture, we also faced a dilemma about where our servers were to be located, since they were supposed to cover players from all over the world. Initially, many microservices were located in Amazon AWS for a number of reasons, in particular, because of the flexibility this system offers in terms of scaling, as this was very important during moments of gaming traffic surge (like when it was featured in stores and to boost UA). Additionally, back then, it was difficult to find a good hosting provider in Asia that would provide good network quality and connection with the outside world.
The only downside to Amazon AWS was its high cost. Therefore, over time, many of our servers moved to our own hardware – servers that we rent in data centers around the world. Nonetheless, Amazon AWS remained an important part of the architecture, as it allowed for the development of code that was constantly changing (in particular the code for the Leagues, Clans, Chats and News services) and which was not sufficiently covered by stability tests. But, as soon as we realized that the microservice was stable, we moved it to our own facilities. Currently, all our microservices run on our hardware servers.
Device quality control and more metrics
In 2016, important changes were made to the game's rendering: the concept of Ubershaders with a single texture atlas for all mechs in battle appeared, which greatly reduced the number of draw calls and improved game performance.
It became clear that our game was being played on tens of thousands of different devices, and we wanted to provide each player with the best possible gaming conditions. Thus, we created Quality Manager, which is basically a file that analyzes a player's device and enables (or disables) certain game or rendering features. Moreover, this feature allowed us to manage these functions down to the specific device model, so we could quickly fix problems our players were experiencing.
It’s also possible to download Quality Manager settings from the server and dynamically select the quality on the device depending on the current performance. The logic is quite simple: the entire body of Quality Manager is divided into blocks, each of which is responsible for some aspect of performance. (For example, the quality of shadows or anti-aliasing.) If a user's performance suffers, the system tries to change the values inside the block and select an option that will lead to an increase in performance. We will return to the evolution of Quality Manager a little later, but at this stage its implementation was quite fast and provided the required level of control.
Work on Quality Manager was also necessary as graphics development continued. The game now features cascading dynamic shadows, implemented entirely by our graphics programmers.
Gradually, quite a lot of entities appeared in the code of the project, so we decided that it would be good to start managing their lifetime, as well as delimiting access to different functionality. This was especially important for separating the hangar and combat code, because many resources were not being cleared when the game context changed. We started exploring various IoC dependency management containers. In particular, we looked at the StrangeIoC solution. At that time, the solution seemed rather cumbersome to us, and so we implemented our own simple DI in the project. We also introduced hangar and battle contexts.
In addition, we started working on quality control of the project; FirebaseCrashlytics were integrated into the game to identify crashes, ANRs, and exceptions in the production environment.
Using our internal analytical solution called AppMetr, we created the necessary dashboards for regular monitoring of customer status and comparison of changes made during releases. Further, to improve the quality of the project and increase confidence in the changes made to the code, we began researching autotests.
The increase in the number of assets and unique bugs in the content made us think about organizing and unifying assets. This was especially true for mechs, because their build was unique for each of them; for this, we created tools in Unity, and with these, we made mech dummies with the main necessary components. This meant that the game design department could then easily edit them – and this was how we first began unifying our work with mechs.
More platforms
The project continued to grow and the team began to think about releasing it on other platforms. Also, as another point, for some mobile platforms at that time, it was important that the game supported as many native functions of the platform as possible as this allowed the game to be featured. So, we started working on an experimental version of War Robots for Apple TV and a companion app for Apple Watch.
Additionally in 2016, we launched the game on a platform that was new to us — the Amazon AppStore. In terms of technical characteristics, this platform is unique because it has a unified line of devices, like Apple, but the power of this line is at the level of low-end Androids. With this in mind, when launching on this platform, significant work was done to optimize memory use and performance, for instance, we worked with atlases, texture compression. The Amazon payment system, an analogue of Game Center for login and achievements, was integrated into the client, and therefore the flow of working with the player’s login was redone, and the first achievements were launched.
It’s also worth noting that, simultaneously, the client team developed a set of tools for the Unity Editor for the first time, and these made it possible to shoot in-game videos in the engine. This made it easier for our marketing department to work with assets, control combat, and utilize the camera to create the videos that our audience love so much.
Fighting against cheaters
Due to the absence of Unity, and in particular th
