A deep dive into Panorama: Xenyria's new in-game screen system

A look into the past

Previously, we were using item frames and maps to display some user interfaces such as the Invasion game list and the player results during the match outro. While this system works, it doesn’t work great and comes with a lot of issues.

Bandwidth issues

Let's go into the technical details: One Minecraft map has a resolution of 128 x 128 pixels. In Minecraft's protocol, every pixel is represented by an 8 bit integer. This means that sending one map to the client takes around 16kiB (16,384 bytes) of bandwidth. Doesn’t sound like a lot? Keep in mind that we've only taken one map into consideration so far.

Let's take the old Invasion room list screen as an example: Its physical size inside is 5 x 3 blocks. This effectively means we need around 15 maps which then results in a total of 16kiB x 15 = 240kiB of data that needs to be sent to the client every time the user enters the Invasion subway.

In reality, less data than this is transmitted due to the use of compression, however, this comes at the cost of using CPU resources to compress packets. While this is still acceptable when there’s just a few players on the server, the entire system becomes quite tricky once we hit higher player counts.

Wait, that looks off...

You may remember the Invasion lobby board displaying all players of your current room in the form of their heads on the board. Back in January, one of our users reported an issue with Invasion's lobby board where their player skin looked weird on the board. The reason for this was quite simple. While maps are a great way to render any shape or image imaginable it comes with a severe disadvantage: Color depth.

Usually Minecraft skins use 32 bits for representing a single pixel, but as previously mentioned every pixel on a map is represented by 8 bits. This means we can display a theoretical maximum of 256 different colors. To make matters even worse, the actual number is a bit lower since Minecraft doesn't assign a color to every theoretically available value.

Just to put into perspective how much of a difference this makes, let's just keep the following in mind: Our maps can display somewhere around 256 different colors while one pixel of your Minecraft skin can be one of 4,294,967,296 different colors. It's also important to mention that the conversion between both color systems is quite iffy.

Do I smell something burning...?

While we can represent every color that can be displayed on a map as a 32-bit color, the opposite isn't possible. In order to convert a 32-bit color value to a Minecraft map color, we have to go through all colors that can be displayed on a map and check how close it is to the color we need.

This became especially tricky for the outro screens where we displayed lots of bars and perk icons on one screen. Ironically, the outro phase of a full eight player match used way more CPU resources than the actual match logic itself. This is one of those cases where the entire system becomes very inflexible once PaintSquad reaches a higher number of concurrent players.

At this point it was quite clear that we needed a new system for efficiently displaying user interfaces inside a Minecraft world.

Panorama

With the latest features of Minecraft in mind, we started planning a new system for efficiently displaying in-world user interfaces across Xenyria. The result of this planning phase was what we now call "Panorama".

Goals and expectations

Our goal was to create a system that allows the easy creation of new in-world user interfaces while also offering the best possible developer and user experience. Another idea behind Panorama is offering an API that is so easy to understand, everyone who knows a few things about web development should be able to create user interfaces for Panorama.

Screens, Scenes and Views

To make the following parts a bit more clear I'd like to first explain the difference between a screen, a scene and a view: Once a player is in the range of a screen, a new view is created which will then render the scene that has been configured on the screen.

This way, every player has their own instance of a user interface. This is more or less required since various parameters like the user's language affect the way a scene will look.

A scene contains all components of the user interface. All components are separated into a tree-structure, similar to how the DOM works in web development.

Every screen has a set resolution in pixels as well as its size in the Minecraft world. For example, a screen with a resolution of 300 x 100 pixels and a physical size of 3 x 1 blocks results in one block being 100 x 100 pixels in size.

Everything is text

We decided to re-use one of the core concepts of our Glitter UI framework: Everywhere you can display text, you can display a user interface. The new Text Display Entities that were introduced earlier this year were everything we needed in this case. Now that we can render Glitter components at any point of a Minecraft world, we can essentially display text, buttons, images, boxes, and much more outside of an inventory.

The most important part about Text Display Entities is the fact that we can scale, rotate and move them with interpolation. The scaling part is especially important when we consider that the size of the text is dependent on the resolution and physical size of a screen.

Another JavaScript framework?

While everything we've talked about so far works great for displaying elements, we haven't talked much about logic yet. We came to the conclusion that utilizing a scripting language would be the best for our use case, but this meant we had to decide which one would best suit our needs here. After some consideration, we ultimately came to the conclusion that the best choice for us would be JavaScript. It's widely used in web development and the majority of our team already knows it, which put it in a perfect position for us.

There was just one small problem: While we already had a working JavaScript engine that could be used together with Java code, it just wouldn't be able to keep up with the scale of PaintSquad in the future. This is based on observations we made during the development of Invasion where we used it to handle various interactive map elements. It took quite some time to process some of the scripts. Besides this, there was no support for most modern JavaScript features (such as “let”, “const”, arrow functions, etc.).

We've tried out a few existing JavaScript implementations and bridges for Java but ended up finding nothing that satisfied all of our requirements. The most problematic part in this case was being able to debug scripts from our code editor via breakpoints. This is the point where we decided to take matters into our own hands.

The inception of V(ee)8

Our idea was to take an existing JavaScript engine that fits our needs and then make it accessible from Java. We quickly determined to go with Google's "V8" JavaScript Engine since it's widely used (most web browsers use V8!) and should be fast enough for anything we throw at it.

This is how our Vee8 project was born. While it's effectively just a bridge that provides access to the native functions of V8, it also comes with its own set of features such as a built-in generator for TypeScript definition files, support for debugging via Chrome Remote DevTools, creation of proxy objects for Java classes and interfaces, as well as access to Java constructors.

Vee8 is also the first project at Pixelground Labs that was developed using the concept of TDD (test-driven-development).

We're also working on replacing the old JavaScript library that was used for Invasion with Vee8. This should also make the development process of new Invasion maps a lot easier.

For the future

Vee8 and Panorama provide us with extremely flexible and future-proof solutions that will allow us to create many awesome things in the future. Even though we might not be able to have the craziest use-cases imaginable for the two systems at first, we’re sure this will allow us to create amazing experiences in the future.