Test My Patience

Part nine in a weekly(🔨) devlog.

Standard development disclaimers apply. This is pre-pre-pre-Alpha content, everything is subject to change, features may not be present in the final version, there's a very strong chance none of this will ever be released, etc. etc.

There shouldn't be any plot spoilers in these posts, but there will be occasional discussions related to characters, locations, mechanics, and other aspects of my potential upcoming Shadowrun campaign (tentatively titled "CalFree in Chains"). You may wish to skip them if you'd like to be completely surprised.

I forget where I first heard this analogy, but working on a big project (like a video game) can seem like building a house. In particular, the view from the outside. If you live in a neighborhood where a new house is under construction, it will look like nothing's happening for months; then, apparently overnight, there's a big house sitting there; and then it seems like nothing else happens for many more months until it's finally sold.

House of Pies

The reality is that a consistent amount of work is being done on the house throughout the whole time, but most of it isn't visible to outside observers. You don't see the foundation being set, you don't see the plumbing going in, you don't see all the fixtures being installed. The framing is the part that looks like the house, but it's really just a small fraction of all the effort that goes into building the house.

In this analogy, I'm now starting work on the framing of my game: the walls are going up, and you can finally see something. Two weeks ago, there was no "game". Now, there is a "game": you can walk around and talk to people and fight monsters and solve puzzles and pick up items and all the stuff that feels like gameplay. Over the coming months, every minute I spend in front of the keyboard will be directly adding gameplay, and the project will seem like it's advancing relatively rapidly.

This is the most exhilarating phase of development, but it's also deceiving. The months and months of preparation that came before now (creating the plot, developing the characters, designing the missions, building maps, writing dialogue, and so on) didn't directly add a single minute of gameplay, but they're absolutely necessary for the stuff I'm doing now. Without that phase, development would be exponentially slower, and more importantly, the result would be far worse. And, on the back-end, the game might seem "done" once I've added all the characters and triggers to all of my scenes; but the game won't be fun (and might not even be beatable!) until I've tested and balanced the final result.

Each of these phases prepares the way for what follows. Pre-production improves the development phase. And, just as importantly, development sets the stage for testing. If I'm quick and sloppy while building, testing will be a nightmare and the final result will be buggy. But if I'm careful and diligent, it will go much more smoothly.

Why Test Now?

The time it takes to fix a bug is directly proportional to the time since you wrote the bug. If I test a trigger than I wrote five minutes ago, I'll instantly remember what I was trying to do, how I was trying to do it, what context the scene is unfolding in, what other possible interactions could occur, etc. I'll immediately notice the problem and be able to patch it quickly (and test the patch, of course!).

If I'm testing more than a day later, the process is slower. I'll probably notice the bug, but will need some more time to find the relevant bits of logic to tweak. And, if I've written a lot more triggers since then, there might be a cascading effect: the fix I put in now might break stuff that had previously been working.

If months have passed since I wrote the bug, then it may be very painful to resolve. I'll wonder "What was Chris thinking?!", spend time re-orienting myself to all the elements of a scene, and trying to think of all the potential side-effects of my change.

Bottom line: if I fix a bug immediately after writing it, it might be a 15-second fix. If I fix it a few days later, it could take five minutes. If I fix it after months, it might take an hour.

So, in the big scheme of things, it's much more efficient for me to test as I go along. This doesn't replace the testing at the end, of course - lots of issues are only apparent when going through the entire contiguous campaign. But the total time spent testing will be much less if I focus on it now.

Ugh, Do I Hafta?

Testing is boring and tedious and frustrating. Especially when you've hit your groove and are dashing off big blocks of the game, it's really tempting to just keep rolling. Experience has taught me that I'll be making future-me happier by testing today, so I try to make that as easy and painless as possible.

The naive way to test is to run through the scene: start at the beginning and go through to the end. When you encounter a bug, fix it. Keep doing this until there are no more bugs.

There are a bunch of issues with this:

• You end up testing the first part of a scene a lot, and the end very little. It's very tempting to stop after your first successful pass and say "Yep, it's working!", even though you've tested the early bits a hundred times and the final bits once or twice.
• It's very slow. Even if you run through at super-speed, you're retreating the same ground.
• It's hard to test alternate routes: different etiquettes, previous actions, etc.

A Journey to Better Tests

When I was making Antumbra Saga, I would get around these problems by adjusting my scene prior to testing. A really simple example: moving the player actor spawner from the start of the map to the end of the map. That way, I can test the bit at the end without running through everything before it. Likewise, if I don't need to re-test an "open the door" quest, I can temporarily move the door out of the way. Once I'm satisfied with how everything works, I move it back to where it was initially.

This did save a lot of time, but caused problems of its own.

• About 20% of the time, I'd forget to move stuff back to its proper place once I was done. This was especially true if I needed to hop over to another scene mid-testing, or was interrupted in the middle of a testing session.
• Sometimes I'd move things back, but not to the right places.
• By skipping over things, I wouldn't notice when things within those sections broke, since I was only jumping ahead to the end.

Starting in Antumbra 3: Corona, and continuing with The Caldecott Caper, I started working with debug triggers instead. Instead of me manually adjusting stuff in a scene just for testing, instead I would write a trigger to do the testing. This would look something like:

• When the map starts:
• Move PC0 to (region near the end of the map)
• Delete prop "Fallen Boulder".
• Add the item "Skeleton Key" to PC0.
• Set Etiquette: Security of PC0 to 1.
• Set the story variable "Ate a big sandwich" to "true".
• Kill all actors with tag "isRoamingPatrol".
• Set goal "Escape the scary corn maze" to "Started".

Now, by enabling this debug trigger, I can set my initial state and test that one section of the scene as much as I needed to.  If I want to change stuff around (like checking what happens without the etiquette), I can just modify the trigger. Once everything is done, I just uncheck the trigger, and everything's back to normal.

This approach worked great during development. In the two years since releasing Caldecott, though, I've come to realize that it could be better. In particular, since I just have a single debug trigger that I'm modifying, I still have the problem of needing to re-orient myself when I'm revisiting a scene much later. I might have done a lot of work initially to configure a particular scenario, only to lose it all when switching to test a simpler one.

Many Triggers

The solution is obvious in retrospect. Don't make a single "Debug" trigger; instead, make a bunch of different triggers for each major area of a scene. These can continue to exist separately, so as I iterate on a scene or jump back in to fix bugs, the previously-written tests will still be available.

The one part where I did this for Caldecott was the heist, and holy cow am I glad that I did. This was such a complex, intricate scene that I needed to test a ton and took a long time to run through, so I had a suite of tests: "Debug_MidCar", "Debug_MatrixPhase1", "Debug_Monsters", "Debug_Decouple", etc. Each trigger would update all of the in-scene variables, automatically fire the triggers that would have previously executed, and otherwise jump me ahead to the part I needed to focus on. I've had ample opportunities to pat past-Chris on the back for his foresight in keeping those triggers around.

Now, in CalFree in Chains, I'm doing all of this from the start. Each time I work on a major new section of a scene (a combat encounter, or a puzzle, or whatever), I create a unique debug trigger for it. I iterate on that trigger while testing, and once I'm done, I leave it in place and move on to the next trigger.

This is especially important because my scenes in CFiC are so much bigger than in Caldecott. I mentioned this briefly before, but I'm trying to emulate the Hong Kong design of having big single-scene missions rather than a series of smaller scenes that link together. There are game design advantages to this, but it does mean a lot more gameplay on a single map, so having the ability to jump ahead to later parts is crucial.

Just Show Me The Code

Here's a screenshot of one of the debug triggers for the opening scene.

Basically, this trigger is rewalking the path a player would have taken to reach this part of the scene. That involves adding some people to the party, physically moving them, running some other triggers,  updating scene variables, opening a door, and acquiring an item. That's a lot of stuff to remember, and if I need to test this section of the scene in the future (spoiler: I almost certainly will), then I'll be glad to have this all set up.

Better Building

One interesting thing I've noticed is that, for CFiC, I generally build scenes backwards: starting from the climactic boss battle at the end, making sure it works right, then setting up the puzzle that leads to the boss, then setting up the fight before the puzzle, and so on.

I'm not sure if that's actually any better than my traditional approach of starting at the beginning and writing to the end, but I think it does have some advantages. I'm sort of forced to think in terms of state rather than in terms of actions: carefully consider the composition of the game's variables and actors and goals at a moment in time and write from that baseline, rather than examining what the variables look like after a sequence has completed. It isn't unlike the shift from imperative to reactive programming that I've been making in my day job.

In other words: in the past, I would write a scene, then run through it progressively, checking in at spots along the way to see what things looked like. Now, I decide what things should look like, then build the scene to achieve that state.

The overall effect is that, behind the curtain, I'm able to think of the scene as a few big, concrete chunks of story, rather than as dozens of miniscule actions. Time will tell if that's actually any better, but my early sense is that it's helping to organize my triggers and my thoughts more logically, and may lead to a result that's more stable and more testable.