The Epic Games website said at the time about the Unreal Developer Network:

Licensees receive full source code for our engine, tools and most recent game. Support is provided directly from the team that actually develops the engine […] UDN is a repository of knowledge, documentation and tutorials for the recent builds of the Unreal Engine.

On the question “is it the Unreal II Engine, the UT2003 Engine, the Unreal Warfare Engine, the Unreal Championship Engine or the Unreal Tournament Engine?”, the answer was clear:

It is neither. It is simply the “Unreal Engine”.

Of course that’s part of a communication effort, but while we see the workflow of snapshots being distributed like id Software did, it reveals a common trunk, or what we may call it today, a master branch. They already talked about a “build”. That means the engine must be repurposable without requiring from the game developer to rebuild it.

Today Unreal Engine is one of the most popular game engines, or game framework like people said at the time. Maybe that Unreal Engine was the cause of that vocabulary slip.

Epic built a community of game developers around their technologies, they provide reusable tools and editors, and they now focus on making artists and developers meet together. This is made with community in mind. Epic’s first product is its service, and the community is a service itself.

While not being free, Unreal Engine sources are made available and are meant to get contributions back from game developers. This is about software projects being alive.

Competitor: Valve and Source engine

While we can expect big game projects to use custom branches of the engine, with time it becomes an evidence that one of the first service to provide to game developers is a reusable build.

Let’s now focus on Valve. Valve got some snapshots of Quake and Quake II, and when they got it in hand they called it gold. They called the source tree “Gold source”. They released Half Life on the engine they built from it and later, the same way Epic Mega Games removed the “Mega” adjective to focus on “Epic”, Valve removed “Gold® to just focus on “Source”.

Source engine by Valve is meant to be reusable. You can install the Source 2007 SDK in Steam and install the GoldenEye: Source game. Developers don’t have to build the engine themselves. The Source engine is not a template, you don’t have to fork, build and maintain it yourself.

Valve provides tools like the Hammer level editor and the map compiler. Those tools are maintained by Valve and designed to be reusable. Mappers don’t have to rebuild the tools.

It looks to not have been that obvious on id Software side.

With Steam, Valve built a software distribution. So, they had to design their software in a way it can be distributable in a repository, they solved file duplication and feature fragmentation, they solved dependencies and made it possible for various teams to have their own release cycle while depending on each other. Outside of legal and ideological issues, maybe we can technically make Debian packages for both the Source engine and games. At least, some of the issues were fixed, Valve had to.

Valve ports their own previous games to updated versions of the engine. They don’t see old code as a dead code, and they have to solve specific problems you get when porting games. Their designs must not prevent ports from happening.

While those products are closed source we see they solved some issues we face as an open source community. Valve’s first product is its service and the community is a service itself, this is about software projects being alive.

Those services induce methodologies like Debian as a software distribution has to build methodologies.

On open source side of things: Godot

On the open source side, Godot is a free and open source ecosystem that competes with engines like Unreal, Unity or Unigine, but don’t compete with id Tech for a simple reason: Godot is meant to be reusable from the start and development is meant to get contributions back to the main branch. By design Godot is not meant to be a game by itself but the underlying technology for your games. The product is not the game, but the tools. This perspective introduces interesting problems to solve.

As a game developer, you download a pre-built version of Godot and there you go. Godot is meant with community in mind, built as a free open source and community software.

id Software engines, editors and tools were templates

Unlike the others I just quoted, id Software closed or opened codes were game templates.

When you make a game based on id Software technologies, you have to fork an existing tree and maintain it yourself.

id Software was selling source code snapshots as templates, and as a developer you had to do required modifications so it is not anymore meant for another game but yours. This is called “instantiation”.

There are some limited features to run a “total conversion” mod on existing games, but you can’t take a build of the engine and just package it with your game to make a standalone game that runs without the original one.

You have to make some required edits and set up the infrastructure to rebuild the engine yourself. This produces a “code induced mindset”.

Because the code itself is designed to be forked, when you work with it, if you don’t try to think out of the box and make special efforts to go against what the code induces to your mind, you start to think the only way to do things is to fork without contributing back. If you ask “why do I have to fork?”, some people may answer you like if you asked a silly question if they don’t simply say “it was always done this way”.

To prevent this on engine side some redesign is required.

To fix the fork mindset on the game code itself, the easier would be to clone an upstream repository and facilitate merging back and forth. Older version control systems like Subversion are also known to make collaboration more costly than forking so using tools inducing good practices like Git is a good idea.

Since id Software codes were opened because of being abandoned, outside of a few projects like ours you can’t hope for an upstream.

On the game code side it’s also possible to split parts of the code as reusable libraries. We’re not considering it for Unvanquished yet, but we ditched the existing historic user interface code for a third party one we don’t maintain ourselves, and for other parts we introduced some code generation mechanisms.

File formats meant to be forked

Let’s look at the BSP file format. BSP is the name of a technique to store spatial information for the game level as a 3D model. We will not focus on that technique but on the file with a .bsp extension that is used to store such information and other things.

You can see a BSP file as a primitive tarball: a container for multiple files the game engine loads. That can be the 3D model itself but also other things like pictures. A BSP file starts with a header with magic numbers, like the “IBSP” string for id Software games, “RBSP” for Raven games or “VBSP” for Valve games. Then a version number. After that you have to expect everything else to be undocumented and specific to every fork on earth.

There is an array for subfiles entries with their offsets and sizes, but there is no number to tell you how many subfiles there are. So, you cannot extract subfiles from other forks.

Imagine every id Tech-based game use its own tar format, and if your game needs to ship one more file, it is required you fork the tar format and create another undocumented variant.

At first the problem does not look that problematic because it’s expected from you to fork the level editor, to fork the map compiler which produces the BSP data, and to fork the raytracer to render light maps. Everything works in your closed ecosystem that never exchanges with the world.

Things become hard when you need to get out of that mindset. SomaZ from the OpenJK project is writing a Blender plugin to load BSP files and bake lightmaps using the Cycle raytracer. No one wants to fork this plugin for every game, so in the end such a plugin would have to implement your own game-specific variant if you want to use it.

When you’re a proprietary game publisher from the 90’s burning games on CD ROMs that would never be updated, you don’t care. But even if id Tech games were produced with the PC market in mind, when you’re dealing with id Software open technologies you’re applying methodologies that fit perfectly if you target console cartridges.

The Unvanquished game uses the Dæmon engine which is a fork of the XreaL engine which is a fork of the opened Quake 3 engine. To introduce new features and to solve problems, XreaL had to extend the BSP format. This engine had code to load both the new format and the original one, but because of the way formats and engine were designed you had to recompile the engine with specific defines to enable support for one format or another, without being able to support both at the same time. For Unvanquished and Dæmon we currently gave up on those enhancements to not lose compatibility with third party software and existing assets, but that means we have to workaround issues that have known fixes we don’t use yet.

Editors meant to be forked

At Quake time there was an in-house level editor named QuakeEd, at Quake 3 time the editor was named QERadiant then Q3Radiant, from then you get CoDRadiant for Call of Duty, MoHRadiant for Medal of Honor, you get it…

Like the engine, the level editor had to be forked, instantiated, edited to fit your specific game, built and maintained by the game developer. The same went with map compiler tools.

Here is my own personal list of known Radiant instances for various games. Expect a long list too for map compilers and ray tracers.

When you look outside of that id Software heritage, there is no DarkBlender, no NetBlender, no ÜberBlender… but when you’re dealing with id Software technologies, you have to. This is not only because the open code is dead, unsupported, and people don’t know how to collaborate together. The reason is it was meant to be that way: developers paying id Software at the time had to fork and do the editor maintenance and publishing work. Splash Damage had its own fork at the time, Raven had one fork per game and that’s a lot.

At some point id Software opened the source code for the editor and the map compiler of Quake 3, it was ported to GTK to be multiplatform, and made somewhat multigame. Don’t expect this multigame feature to be straightforward. It is not meant for you to work on multiple games at the same time, multigame only means you don’t have to reinstall the editor when you stop to work on a game to start to work on another.

Even with its multigame enhancement, the Radiant level editor does not expect you to edit levels for one game one day and for another game another day. This may fit the development model of commercial games, but not the contribution model of an open source volunteer community driven by passion and desire.

The editor expects you store files at only one hardcoded place, so you cannot store your files where you want. This is enough if you’re part of a team working on one game only on a shared folder provided by your company, not if you’re an enthusiast.

History also uncovered an interesting id Software behaviour: when they needed a level editor for their Quake Live game, instead of using GtkRadiant 1.5 that was maintained by the community after they opened it, they just went back to an older 1.4 version they kept in-house before republishing and reopening it again. They forked their own code from an old version, and after that people had to choose between two forks because features and games support was split. At this point it looks like id Software did not know how to deal with a community.

Here is a simple list of features that were added on Unvanquished initiative to the NetRadiant branch, which is the community fork with probably the largest list of supported games today:

• open file from command line and file manager,
  that was implemented in 2017.
• load game assets from predefined arbitrary paths,
  that was implemented in 2017.
• change the current game without having to restart the editor yourself, reload the current map with the new game in case you’ve loaded the wrong map with the wrong game,
  that was implemented in 2019, last year.

The ability to choose where to store your data and to open a file from the file manager was added in the last three years, for a software designed before year 2000. Why it took so much time is because this software is meant to be forked to target your own specific game hierarchy.

Because this is a Debian talk, I quote a special feature that was added to NetRadiant in 2018: making it possible to install the editor following the File Hierarchy Standard. When you’re expected to ship a fork of the editor with every different game you publish, making the editor able to be installed widely is the least of your concern, and it may even be stupid because all the forks will ship with related tools like map compilers that may share the same binary name but don’t behave the same.

There is already two Radiant forks shipped in Debian, DarkRadiant which is a fork of NetRadiant purposed for the Dark Mod and focusing on Doom 3 based games, and UFORadiant which is a fork of an old branch of DarkRadiant that can only edit assets for the UFO: Alien Invasion game.

On software development and child sacrifice

So, we figured out multiple things:

• id Software technology is very efficient but it’s like every part of it assumes you will fork the whole and never contribute back either mutualize development efforts,
• id Tech engines were meant to be sold and forgotten. It looks to have been designed for only one social relation: the sale,
• map formats, materials, net code and other things are not really meant to be extended, versionized, and in many cases, there are no bits to even detect the formats, while sometimes just one integer would have allowed to parse unknown data.

Almost everything in id Tech engines is designed to ship a game that will kill the previous one. New development is meant to ship a new incompatible game. The design is thought around that mindset, and then the design induces that mindset again.

Forking a software project is like giving birth to a baby, a fork is as fragile as a newborn: it cannot do anything else by itself other than dying. If you’re not able to give enough care and attention to the newborn, the baby will die. Same for forks.

The development workflow of id Software at the time looked like child sacrifice: every time a new game had to be developed, the previous child was killed to redirect resources and life support on the new one. Unlike other game engine developers like Epic or Valve, id Software never took the risk to let its child reach adolescence and become an adult.

They didn’t ship reusable engine builds, editors were not truly reusable, they did not build a community around the tools like if it was a product by itself, they did not build a software distribution system so they never had to solve the related problems, they did not build a data distribution system so they did not have to solve problems about format incompatibilities and file conflicts.

At some point in game development with id Tech based code, you’ll notice that to solve some problems you not only need to extend your engine but to fork your own game. While working on Unvanquished I may have seen this situation like 5 or 6 times.

In some way we can measure the impressive genius of John Carmack on the fact its technical skills kept id Software alive for so many years despite so many marketing mistakes, strategic misses and opportunity losses, preventing their own game communities to grow up with them.

With such a retrospective, what looks weird is to discover that because the code was dead, I mean killed, it was possible to open it. So on one hand we got free and open source code, but on the other hand this code was meant to be dead and to produce zombies.

Also, the less you get contributions back, the easier it is for your legal department to clear up everything for an open source release. So, it’s like it was more possible for id Software code source to be opened because the workflow was more closed than others. This is a bit weird to figure out.

In 2020, no one game developer is looking for an engine that requires to be forked and maintained alone. For Dæmon one objective is to make it possible for a game developer to just reuse an existing build, let’s say the one we do for Unvanquished, and package it with their own game code and assets. This is not entirely done yet, but we want it to be done.

Code induced mindset

Quake 3 engine comes with a really nice feature called PK3 VFS, it’s a special virtual file system making it really easy to extend game data. But it’s not designed for modding. This can be used for modding, but it’s designed to allow the game to receive updates coming from one source: the seller.

The design does not prevent the player’s game from being broken by any random custom file from any random server. So we extended it by creating the DPK VFS in a way it makes possible to only load a pre-defined selection of assets instead of loading everything.

That’s not enough to fix some issues and there are some other features to implement, but you’ll see the difference between designing the format to allow the seller to update the game, and to design the format while taking in account unknown random people will update your own game.

That minor change induces a new mindset. Because packages now have a base name and a standardized version string (reusing Debian’s version sorting), it becomes natural to think about packages as libraries. Artists can do textures libraries and models libraries, they can have their own release cycle and mappers can then create levels using this or that texture or model library version.

The change we did is so minor that if you just revert the file extension to the one expected by Quake 3, a 20 years old build can load the data.

But that very minor change means a lot of things for a community, each member being able to develop its own craftsmanship and relations to others, being a supplier of another one or relying on suppliers, this is called an economy.

We’ve talked about the BSP file format, it embeds some pictures that are precomputed light and shadow textures. Only one storage format was supported and only one picture size was allowed, and no bit describes the format, only the number of pictures was variable. Years ago some people had the good idea to make the engine able to load such textures from outside this file. This allowed new sizes, new picture formats including compressed one.

This simple example shows how the first design requires games to fork the format to create incompatible variants if someone needs to store higher resolution textures, while the second design allows any engine to add any format without introducing incompatibilities to the BSP file specification itself.

id Tech code followed the modern linear consumption pattern: production, consumption, trash. Our efforts at Unvanquished is to recreate a cycle. And to recreate this cycle, we have to fix the design inducing the mindset of the linear pattern.

On open source code and sharing human communities

Dave Airlie from Red Hat published a blog article on this topic while I was preparing this talk. While talking about graphics stack instead of game engines, he said:

There is a big difference between open source released and open source developed projects in terms of sustainability and community.

In theory it might be possible to produce such a stack with the benefits of open source development model, however most vendors seem to fail at this. They see open source as a release model, they develop internally and shovel the results over the fence into a github repo every X weeks, […] but they never expend the time building projects or communities around them.

He added:

I started radv because AMD had been promising the world an open source Vulkan driver for Linux […]. When it was delivered it was open source released but internally developed. There was no avenue for community participation […]. Compare this to the radv project in Mesa where it allowed Valve to contribute the ACO backend compiler and provide better results than AMD […] could ever have done, with far less investment and manpower.

There are four important things in that quote. First, it is important for companies to spend time building projects and communities around their own projects.

Second, the name of Valve was quoted again. The frontier between bad and good is not between humans but goes through each human being, same for companies and human organizations. While Epic and Valve never made their game engine free and open source, they understood what a community is. On the other hand, while id Software knew what it is to release their code as free and open source, they did not really build communities around them, even closed source communities.

When you look back it looks like companies that were using id Software technologies were more likely to accept that situation given the code was efficient, and for some of them the only social relation may have been the sale.

Rebecca Heineman, the developer of the 3DO port of Doom even said that at the time her company bought the right to port DOOM to that platform, she did not get the source code at first. This was not id Software fault and she did not have a problem getting a copy of the source on a CD by reaching John Carmack directly, but that means even when source code is bought on purpose the idea of a service seems to be optional.

Software design usually reflects the services around it. Of course we can’t expect methodologies from the DOOM era to be on par with actual ones, but when looking at later id Software source codes, they don’t seem to reflect fundamental changes on those methodologies.

The way problems are solved or features implemented usually reflects methodologies and workflows. And reflections of problems faced by communities are hard to notice in id Software code design.

The third important thing is the wording Dave uses in his blog post when talking about the “open source release model”. We can remove the “open” adjective. id Tech softwares seem to have been more built around a source release model than a sharing development effort model, even if proprietary and under non-disclosure agreement.

The fourth important thing is the cost. id Tech softwares seem to only have been designed to solve technical problems, not the social problems about getting contributions back and preventing waste of development resources on a community scale. Resources seems to have been saved at the local level of the company to the detriment of the community.

In the end, such design deeply harms the technical growth of the software itself when it has to survive one man or one company, and it is relying on volunteer contributions.

Building an ecosystem and a community as a service

How to make a project like Unvanquished and the Dæmon engine doing what the Quake 3 engine never really did on the community side and make its way through the id Software hell of the fork mindset?

First, that’s not technical, but human beings need to experience an existing project as an upstream one. For example when you need to add something to Blender, it sounds obvious you have to submit them patches. This is not obvious per se, it’s a knowledge a project has to build up.

People have to hear about organizations and people’s intentions to review and merge their code. People have to hear their code is meant to be a part of a whole. It’s required to build-up visibility on those purposes and intentions.

If you believe it’s obvious that any patch you’ll make for the Linux kernel would better be upstreamed, it means someone succeeded to sow that feeling in your heart. The Android industry reveals this belief is not obvious: despite all efforts done on making the upstream clearly identifiable and code and tools designed for contributions, many companies act like if the Linux kernel was meant to be forked and they implement their features on what is already a Linux cadaver.

A very important thing is to lurk and live within communities, to listen to their needs, notice the growing aspirations, and accompany the births. It’s a bit like probing a market but it’s about being a friend.

We can’t underestimate the importance of being close to communities. I just discovered this month another somewhat alive XreaL-based project. A Kingpin remake over XreaL. I’m among the ones having the deepest knowledge about all those forks but I missed this one for a decade. Less than two weeks after having talked for the first time with one of the remaining hard-dying contributors he helped us to fix an issue for our upcoming release. It took me 10 years to meet this community, I hope we can do more things together.

Sometimes a design can be made better just by talking with others even if they would never use the software. This helps to step back and get a better picture on what could be done. Sometimes we discover the software can do more than our need just by doing the same thing differently. To open doors you have to look at people’s paths.

People underestimate the value of their contributions. Some forks are done by people tinkering for fun or students trying out something out of curiosity. They usually ignore what they do is precious. That may be due to some wrong assumptions coming from the scholar system making them think they’re not adults yet or not professional ones and anything they would do is not for real. But any patch that solves a problem is for real. Also while growing up we discover professionals may just be teenagers pretending to be serious… So, no one has to underestimate any contribution value.

Some people just fork while saying “I just want to add this”, assuming the original code is finished and there is nothing to get from it in the future. Closed source software may induce this mindset but for an open source project there is no such thing as a code being finished. Also, they ignore other people just ”want to add this” too.

As we see with the BSP and DPK formats, it’s sometimes possible to redesign things with very little differences in a way it makes it easier to extend the code without having to fork it. Sometimes one little change can reveal the idea to fork an entire code base to just tweak a picture size is a waste, while previously such a fork for such a minor change was made a requirement by design.

For Unvanquished we made it clear the engine is meant to be repurposable, and we got a lot of help from Xonotic people to split trees and make the Dæmon engine a project by itself. Because one of the first services that must be provided to a game developer is a reusable build, every issue that is still preventing this is now properly reported and is expected to be fixed one day.

Another important thing being the editor and tools, with the help of Xonotic again we did a lot of work to make the NetRadiant level editor more accessible to newcomers and improve the workflow in more natural ways.

However, so much effort did not prevent another NetRadiant fork to just appear and people have to choose between which game is supported and features, again. Because this fork is based on an old branch that predated a large revamp of the code, the effort to merge them is high and on a developer salary base, the cost of the amount of work to merge the two branches would be a number with four zeroes if valued in dollars. That would not be the cost of one feature, that would just be the cost of the waste. Making people work together is a really long and difficult road.

What is important to keep in mind is at some point the cost of forking is higher than the cost of contributing back. This can be hard to notice in communities of volunteer people when your own acts does not mean more work for you but more work for others.

We can’t do magic and have to rely on other ones’ will. This is the risk we get by letting a child grow: we can give the direction we think is the good one, but in the end everyone is free.

A service is an act of being of assistance to someone, and the first service everyone gets in life is a community.