The whole team (including the leadership team) felt in awkward, not the fact we got backdoor'd in V5, but the fact that we shipped a game that is really terrible.

So, what did we learn and what do we plan to do next?

Background

I am a Roblox game developer, of course. Just check my articles, they are pretty much all about Roblox development. I happen to work in a group called Frappé, employed as a developer working for the upcoming version V5 at 25/04/2020.

Development of V5 started in around 2017-2018, but it eventually got restarted due to some issues. Enters the re-V5, which is the current V5.

The UI was done, but there are a few missing parts in it. Some critical features were missing because the previous UI designer left. So I was employed to finish those missing parts and as well as finish some part of the backend.

We had a really tight deadline when I was employed to work on it. I think we had only 3-4 months left before shipping it to the public, so we have to rush finishing most of the part (horrible idea). Therefore, this leads to low-quality code and introduces a lot of bugs in the game that still remains unfixed when the game launches. V5 also uses some of the code in previous version, and previous version is a combination of code in older version and new code, which makes it a lot worse because consistency between code is lost.

We tested the game multiple times before actually releasing it, but we did not do any sort of stress-tests, which means the capability of the backend is unmeasured and is likely to be vulnerable to large player base considering it is rushed.

The problem

Okay, it's August in the same year, we released V5 to the general public. It was full of hype. The fact people waited so many years for V5 to come out, makes V5 a trending topic in the group RP category. Which also means that the player base is increased by a lot more than what we were expecting.

Initially, many acknowledged FPS drops while playing the game, most people got an average FPS of 20-30 in V5 –– not something we are expecting to happen. Then, bugs in the backend made a lot of functionalities go malfunction. The gameplay experience is affected by a lot, in fact, nothing really works in the game. We were alerted by this issue and we were completely shocked. The problem is so severe that it creates a lot of negative feedback regarding the group, and still is not completely relieved to this day.

What we learnt

We eventually closed down the game and rolled back to V4 and announced that we will be optimising the game and will be rereleased shortly this year. This promise was made but never satisfied, and I believe the reason why is...

• There are too many bugs, we simply can not afford the time to fix all the bugs completely
• The whole codebase structure is flawed
• Error handling is improperly done, especially in remotes
• Heck, the performance is so bad that we can barely work on it in Studio
• Backdoors!!! It's too painful to get rid of them

So, what have we done to prevent any of those issues above, for the future?

• Newer projects are coded with a MVC framework called Knit

  • We have now established a standard codebase structure for our games' code
  • Knit is Single-Script Architecture, modules run in sync and conflicts are less likely to occur
  • Code quality is ensured to be high
  • It is much intuitive for developers to work with

• Error-prone code is now wrapped in rpcalls (recursive pcalls), such as code interacting with Http APIs, et cetera

• Errors are properly handled with the help of promises

  • An error may not mean that code will be unavailable, we can consider using an alternative solution

• Errors are logged to our log drain (Self-hosted Sentry)

  • More easier for us to do debugging

• Newer projects are done with Rojo and done in a Git repository

  • Small patches, UI etc does not require opening Studio to finish, reducing the load time and developers can work on games with a lower-end device or tablets
  • Established a new developer policy in the coding department
    • Developers only have read access to projects they are assigned to. To publish their changes, they have to create a new pull request and ask for reviews from the head developers. Reducing the probable chance of backdoors
    • Only head developers have maintain access
    • 2-factor authentication is required
  • Coding can be done with a code editor, reduces the memory overhead compared to using Roblox Studio

• UI is now written with a declarative UI library called Fusion

  • Increases the readability and maintainability of the code
  • Components reduces duplicated code
  • States make race conditions less likely to occur

• And a lot more

What to do next

With all the changes above, we definitely can not work on the current V5 anymore. The current V5 is not done with Knit, nor is in Git, nor its UI written in Fusion either. So, we decided to recode it from scratch, completely. Start with an empty project.

Of course, this means that folks will be waiting a lot more time for a complete rerelease of V5 to come out. But what you are getting is: high-standard, high-quality, and performant code in the game. Juicing every single part of your hardware to give you the most immersive, most stable and smoothest gameplay experience possible.

We have already began working on it, and it's going really well. Our developers prefer coding in this environment compared to the traditional one in V5 and others.

The basics

• Roblox uses RakNet –– A network middleware for multiplayer games
• RakNet works with both UDP and TCP
• It has dropped support in around 2014
• At the time of that, RakNet still has quite a plenty of bugs/flaws remaining unsolved
• Of course, it is open source under the BSD licence, by Oculus (now Meta? i honestly have no idea about the “renaming”)

Introduction

A couple of years ago, I’ve took a look at RakNet’s source code and discovered quite a few vulnerabilities. Such as being able to screw RakNet up, by sending a specifically designed packet string.

At first, I didn’t take it as a serious concern in game development as, I wasn’t working on games back then! And this vulnerability was not “on-trend” in the Roblox community.

Until ~2021, where I finally worked on games and this vulnerability being used to launch attacks on specific games. So! I have decided to revisit this middleware again.

Case 1: Using RakNet

I was digging into Roblox’s RakNet infrastructure a few days ago, so don’t really expect me to have strong evidences on each points I’ve made, those are really based on my opinions and assumptions (of course, not 100% opinions)

As what I have mentioned earlier, RakNet contains a lot of bugs and flaws in its code, and the author, Kevin Jenkins, stopped working on RakNet for a few reasons. Which also means there are no chance of a patch to resolve all of those bugs and flaws being merged into the code.

Considering the source code is open source in GitHub, exploiters were able to find exploitable flaws and try to abuse it in Roblox.

A reply made by Quenty to a thread in DevForum

In fact! Quenty has made a post to a thread in DevForum in 2015, which exactly talks about a vulnerability in RakNet.

Do excuse the vulgar language used in this image

There is also a “crasher” script which abuses the flaw to screw RakNet up.

Case 2: Using custom RakNet

RakNet is open source, that is what I said above, which also means that there’s another possibility –– Roblox created a vulnerability, or they tried to patch the vulnerabilities but didn’t get it right.

from zeux.io, a personal website of a Roblox software engineer

It’s clear to tell that Roblox have made changes to the networking infrastructure. Of course, this blog post did not mention a single word regarding RakNet, but the network protocol is heavily related to RakNet.

The thread, again! But with zeux’s reply

A reply made by a DevRel regarding this issue, but there is no “update” on it…

From those two replies, and with the image excerpt from a blog post, it’s really likely Roblox has a modified version of RakNet.

Whether has the malformed packet vulnerability been patched by Roblox or not, I do not know. I am not an expert in cyber security (obviously), and I do not have the braveness to risk trying the exploit.

If it has been patched, could it be another vulnerability from RakNet? A vulnerability created by Roblox themselves? Or a new vulnerability pops up after the patch? All those can be possible, but it is uncertain.

Case 3: A vulnerability in design

Maybe it can be a vulnerability that exists in most networking middleware?

A video in the cloud, name is blurred for multiple good reasons

There is a “DDoS” panel that works for both Roblox and FiveM, and even Tom Clancy’s Rainbow Six Siege.

We know Roblox uses RakNet, but neither FiveM nor R6 does. FiveM uses ENet. And R6 got called out for using a “horribly-made” custom networking library, and got suggested RakNet as well as other libraries, apparently.

Heck, even works for Call of Duty: Modern Warfare

Obviously, this is a really extreme and uncommon case. The panel might have just supported multiple games with different exploits. Which seemed to make more sense, otherwise the whole game industry would be in nightmare already.

Other cases

• “Your game is backdoor’d/has serversides/etc” –– Unlikely, it’s too hard to believe that, when there are multiple games suffering from the same issue, from small to big, from infamous to famous.
• “They are just spamming the server IP” –– Does make sense, but spamming the server IP seemed to be really inefficient, still will consider this as a feasible case
• “They are just spamming your remotes” –– Again, not when there are multiple games suffering from the same issue, also there are rate limits on remotes
• You name it?

Conclusion

We still have no idea about the cause of those attacks, but we can safely assume that its either custom RakNet or actual DDoS/DoS.

• RakNet is open source, I have huge doubts that Roblox never modified it. A person I know, also claims that Roblox has a modified version of RakNet deployed in their infrastructure.
• You can always overload a server by DDoS/DoS, this might be because there are no mitigation policies set for attacks like this, in Roblox’s servers.

I do not have a lot experience in cybersec, so I won’t be making bold claims in areas I am not familiar with. Please! If I made a mistake in this article, do let me know.

RakNet is not bad, Roblox is not bad either. I am making this article not only because of games I have worked on, but also because of my curiosity

This article is originally published as a reply to a thread in DevForum.

I tend to get pissed off over small details, especially with UI design. It is probably a bad habit since it wastes a lot of time and really strains my eyes. However, UI is pretty much about the details, a good UI is when good UX is matched with fine details. So I kept the habit.

This one though, I don’t really think it’s me being picky, it can bring a huge difference when scaled up. It’s about applying strokes to rounded UI in Roblox.

At the moment, there are two ways to apply stroke to an UI –– either by using UIStroke, or by Duplication & ZIndex. Both has this problem, but you can’t fix it if you are using UIStroke.

The problem

Ohhhh no. Inconsistent radius!

If you have good eyesight, you will sense that there’s something wrong. The stroke and the actual layer do not share the same roundness.

But they have the same corner radius value!

They have the same corner radius value, yes. But do they have the same size? No.

Corner rounding is like putting a circle into your shape, and its size is based on the corner radius (times two). As it is bigger, the shape itself would start to look like a circle. Eventually becomes completely rounded once the radius has reached half of the shape’s height. From that being said, when the size of the shape has changed, the distance for the circle to make the shape as a circle will be changed too.

Assuming that stroke is 2px thick, that means the distance has been increased by 2px, hence making it less rounded.

The solution

We have just explained how corner rounding works with simple logic, and that “roundness” is based on the corner radius and the size of the shape.

So, how do we rectify it? Simply by reverting the distance change with Mathematics. If it has been increased by 2px, increase corner radius by 2px too.

Hence, we can come up with a formula:

BorderRadius = CornerRadius + BorderWidth

LaTeX, if you want.

Happy designing.

Whether you are UI coder or not, animating UI can be painful. Choosing the correct easing for the correct UI can take a lot of time, and usually the provided easings does not suit your UI well enough. Easings provided by Roblox in enumerations can not be modified, which can be disastrous if the easing is too exaggerated, or too weak.

It’s not only about easing. If you are animating an UI with an unpredictable value (user-defined, etc), maintaining the duration consistency for each animation can be difficult, and usually creates a hot mess in your codebase.

For one animation, it is fine, but if for multiple, uhhhh. Yeah, not a good idea.

Why bother.

Why bother creating a new tweening library simply for custom easing? And why bother creating multiple formulas just to calculate the correct animation duration?

If you are encountering either these two problems, maybe it is time for you to learn a new concept –– Spring motion

Spring-driven UI animation

Rather than defining an animation with time and easing. Define an animation with frequency and damping ratio. No matter how far away the end value is, the animation is always consistent, it scales with the distance.

With the damping ratio, you can create infinite possibilities on the animation behaviour –– whether to go overshoot, or smoothly. Solving the two problems we have mentioned in this article, completely.

How

You do not need to study physics and reinvent the wheels to create spring motions. For Roblox, there are a couple of modules that does it for you. Such as Spr from Fraktality, or Flipper from Reselim, et cetera.

Of course, it would be really hard to guess the animation behavior with only numbers. There is a visualizer for it.

Parameters view for the visualizer

In springs, the speed is faster when the frequency value is greater, where the damping ratio changes the converging behaviour, in which, value lesser than 1 would create overshoot.

That is pretty much how you implement spring motion for UI.

Conclusion

Spring motion is great for UI animations, especially if you want custom easing, or want to have a consistent animation speed.

It’s also great for other applications, such as camera panning, camera manipulation, car physics, et cetera.

Of course, while spring motion seemed like an ideal solution for animations, it’s not always. Ask yourself is there any necessity to adopt spring motion before actually doing it. Sometimes it would backfire.