As part of the effort to modernize our engine  we completely rewrote the the filesystem, the code responsible for finding and loading the game assets.

Perhaps the most visible change is that the home path, the directory where the user configuration and downloaded pk3s live, has changed. On Windows it is now My DocumentsMy GamesUnvanquished and ~/.unvanquished on Linux. On these platforms, that means your configuration will be reset (and we will take advantage of this to clean up the way configuration options are saved). Another effect of this change is that all the game assets and maps will have to be redownloaded (sorry to our bandwidth-limited friends).

Read on if you are interested in why needed to change the filesystem and what it changes for content-creators.

Background on the Quake3 filesystem and its drawbacks

In Quake3 the assets are packaged inside .pk3 files (which are just renamed .zip files) with usually a pk3 for the common data, a pk3 per map and a pk3 for the game code; they contain a hierarchy of files and folders that are merged together by the engine to produce a single virtual directory containing all the files of the pk3s. If foo.pk3 contains a file textures/a.png and bar.pk3 contains textures/b.png the game will be able to access both using only the names textures/a.png and textures/b.png without knowing they are in different pk3s. In addition if two pk3s contain the same file, the version of the pk3 with the name latest in the alphabetical order is loaded. This is great for modding because to change files you just drop a pk3 with a name near the end of the alphabet and it will automatically override parts of the other pk3s. This design that encouraged moddability has been so successfull that it was implemented in a standalone library. We still want mods to be as easy to distribute.

However when you start connecting to modded servers¹ pk3s pile up and eventually you end up having hundreds of pk3s (most of them starting with zzzzz to try to beat the others). At that point the Quake3 engine will start taking time to start because it will load all the pk3s at startup, and worse, you won’t be able to play a vanilla game locally because some modded pk3 will have changed the weapons, physics…

In addition map creators can’t know which pk3 a user has so they need to ship all their resources (esp. textures) along the map, which makes for big downloads and bloats hard drives.

The new filesystem handles pk3s more like packages

In order to fix these problems we added versions and dependencies to pk3s. Now all pk3s names must be of the form <name>_<version>.pk3 both being arbitrary strings (without underscores)² in addition, packages can contain a DEPS file at the root that lists dependencies. That way when the game starts only the latest version of the unvanquished pk3 is loaded as well as its dependencies and it ensures mods won’t degrade the user-experience of the main game or of other mods. If the user wants to load additional pk3s by default he can use the fs_extrapaks cvar.

The DEPS file contains lines of the form <name> to depend on the latest version of a package or <name> <version> to depend on a specific version; the latter form should be only used to depend on previous versions of the same package. For example map-meatgrinder_1.0.pk3 will have tex-ex to depend on Evillair’s eX texture set while a new release unvanquished_0.a197 will have unvanquished 0.a196. That way we can still do updates that ship only the modified files while content creators can use common pk3s to avoid unnecessary downloads. When loading a map, the game will search for the server’s version of the <mapname> package (latest when playing locally) and it will load the file maps/<mapname>.bsp: so now mappers shouldn’t include the version in the .bsp name but only in the package (you’ll also need to rename lightmaps, levelshots and minimaps).

Other changes include:

• The removal of profiles. There is now only a single set of settings, which is much easier to manage
• The removal of mod-specific folders. Everything is now centralized, which makes management easier.³
• Configurations files and scripts are now loaded from the config subdirectory of the home path.

To summarize, when ingame, the loaded pk3s are the game pk3 (unvanquished) and the map pk3, as well as their dependencies. This results in improved map loading time. You’ll also notice that we didn’t mention the VMs pk3, it’s because QVMs are not included along the assets in the game packages; if you are doing a modification you just have to make a mod-mymod_<version>.pk3 file and make it depend on unvanquished and set fs_basepak to mod-mymod when starting a server.

Making your development environment work with the new filesystem

Usually when developing the game you link the main directory of the git to a pk3dir⁴ to use the latest assets and sometime have additional pk3dir to test asset changes. Now the paks are in the pkg directory of the home path instead of main. The recommanded approach to setting up a dev environment is now to link main to unvanquished-git_1.0.pk3dir and then start the game with fs_extrapaks set to unvanquished-git. You can also add other .pk3dirs separated by a space so the startup line looks like ./daemon +set fs_extrapaks “unvanquished-git foo bar” to load the foo and bar pk3dirs in addition to unvanquished-git. To see which pk3s and pk3dirs have been loaded you can use the /listPaths command.

Closing Remarks

In addition to solving the problems of the Quake3 filesystem, this filesystem will ease the transition to PNaCl by avoiding roundtrips between the engine and the gamelogic for file operations. With PNaCl the engine can hand file descriptors to the sandboxes so that they can use it directly, the filesystem has been designed with that idea in mind and will have most of its code shared between the gamelogic and the engine to provide a unified interface. Speaking of PNaCl, we hope to expand on the proof of concept we did some time ago and actually release it soonish now. Stay tuned and don’t forget that Alpha 24 is coming this weekend.

Finaly I’d like to thank Amanieu for single-handedly rewriting the whole filesystem.

 

 

1. Quake3 has a way to handle mods cleanly but we are speaking here of lolmods such as “LSD texture pak” and other “bunny particles fly everywhere mod” that go next to the main game’s pk3s.
2. an exception is made for autodownloaded pk3s that have the form <name>_<version>_<checksum>.pk3 to prevent conflicts. You should never add a checksum manually.
3. they might get re-implemented at a later time.
4. a pk3dir is a folder that is treated as a pk3 except that it is uncompressed, mainly used for development to avoid cluttering the “main mod” folder with unrelated files like we used to do with Quake3