That’s so interesting that you mention that bug in world 5, because I noticed it happening a few times and I always felt like I wasn’t wrapping my mind around the concept of time properly.

I feel a little better now.

P.P.S. World 5 in Braid (the parallel universe one) works by a relatively straightforward version of the input recording idea. I hated that and in fact the xbox360 version shipped buggy, in the sense that sometimes on world 5 when you rewind, the thing that was supposed to happen doesn’t happen the same way again. I looked for the cause of the bug and couldn’t figure it out and was dead tired of working on the game and the bug wasn’t that bad, so I shipped it.

Later, for the PC version, I finally figured it out the problem. It was something a little unexpected but also that was obvious in hindsight.

To fit fully in RAM on the Xbox 360 I restricted rewind recordings to about 40MB, which indeed gives 30-60 minutes depending on what level you are playing (more objects == more memory).

However, as I say in the talk that Sean linked, the rewind system for Braid isn’t rocket science, it is just a baseline competent implementation with some useful choices made about how to reduce size, but with no real micro-optimization kinds of work done beyond those basic choices. If reducing size were really a priority, I probably could have gotten the rewind data 2x-6x smaller. I just had too many other things to think about.

We have created a commercial product for playtesting browser games based on input tracking. While having ability to replay sessions is cool, it’s also quite ineffective in the long run – how many 40 minutes gameplays could you watch? It quickly turned out that our users find much more value in having a complete log of user inputs they can search in (by version, game level, specific behavior) and aggregate (i.e. to create heat maps or trends).

Replays are only used for quick discovery process and the typical scenario for analysis looks like this:
1. you aggregate recordings and find out that people are dropping out at level 2
2. you filter out those recordings
3. your *replay* a couple of them
4. you find a problem (i.e. people are confused by UI)
5. you read a log of user behaviors or analyze its visualization to find a hook (i.e. ‘myButton.click()’)
6. then you compare how many players among those who dropped out and those who didn’t behaved that way (clicked the button) to verify your assumption.

I believe, playtesting becomes really powerful if you can easily combine qualitative and quantitative analysis into a single process. It’s impossible if you are recording just output (or is it?).

As for the problem with game versions and ability to replay the session – it turned out not to be that crucial. Of course staging client helps (no need for staging backend), but usually, you either analyze sessions pretty quickly (before a new version goes live) or you forget about old ones because new sessions got recorded (we record tens of thousands of sessions).

The bigger problem we have with replays based on input is accurate synchronization of user input, especially in 3D environment. But it can be done quite well to with some extra effort.

And there’s one more case for input based recordings – we use them to detect cheaters in competitions. If a cheater attacks code or memory the score achieved during replay different from a submitted one :)

(sorry for a looong comment!)

The abstract for Jon’s talk below says “30 to 60 minutes”, but I don’t know how accurate that is.

https://store.cmpgame.com/product/5900/The-Implementation-of-Rewind-in-Braid

Yes, smoothness is a good thing, often.

But there is such a thing as taking it too far (which is what pretty much every AAA game company does in focus testing).

Once new people are playtesting the game I probably will make a lot of changes for smoothness purposes. But what I am saying here is just about overall philosophy of playtesting. It is not the case that everything rough is bad. It is okay for people to get stuck, and it is okay for people to misunderstand things… sometimes. Sometimes smoothing those things out makes the game better, but sometimes it makes the game worse.

Can’t smoothness be a good thing?

Mathematicians do their best to break new concepts down into a manageable progression. And when they want to give examples of a phenomenon, they choose things that mostly simple and ordinary, so that they can draw your attention to a particular thing.

Life is short; I feel games should try to press the interesting stuff they have to show you into as small an amount of time as they can.

We all like “Portal” here. When teaching you the “fling”, the makers broke down every aspect of it, giving you a lot of training wheels before they expected you to understand and experiment with it on your own.

Jon, remember the puzzle in my game you got stuck on? With the triangle you kept firing into, to no avail? Well, after we talked I turned that triangle upside down, so you can’t really fire into it. You weren’t the only one who got fixated on the idea that that triangle shot might help them, so I’ve disposed of it. I also added another puzzle that nudges you in the right direction.

I think you already support me on some of this? You’ve praised the cutting out of red herrings, and the simplification of puzzles. In these games, where we are intending to express stuff to players, is “getting stuck” a good thing to have happen?

Agreed Phil, both input and output recording are useful – and they are useful for quite different purposes.

I used to work on the user interface for an air traffic control system. Frequently we would be asked to investigate situations that occurred in the operational environment. Unfortunately our recordings were mostly input based, which made it very hard to confirm rare bugs. Even with good input recordings, hardware & network conditions could make a replay not produce a bug.

I think output recording is great for:
– Confirming the presence and presentation of bugs which may be hard for an end user to describe (“Yes, you’re not crazy… that really did happen!”)
– Looking at how the user interacts with the system in general

And input recordings are very useful for:
– Tracking a confirmed bug back through the system & code to it’s root cause
– Assisting in confirming that you have resolved the bug against the original cause
– Regression testing (if you have a library of input recordings to test with). Although this approach suffers from the versioning.

– Mark

I wonder though, how you will tackle the problem of affording players too many choices. I often feel in non-linear games as though there are too many things demanding my attention at one time, and I don’t know what is the best to pursue first. What do I want to do today, as opposed to tomorrow?

Choice fatigue, lol.

I think it may be fine as long as it is clear that there is no best path. That all choices are equal, and you will never miss an opportunity by choosing the wrong set of puzzles first.

If by fortitude, you find a spot for The Witness in the next console generation: I would be happy to buy a new console to play it. It is unlikely that I will be thus compelled by a shinier Uncharted.

The problem with input recording is that it’s difficult to make it robust between versions. If you have a game that is played and updated a lot your replays would quickly not work. You can jump through hoops to make this work but I think recording the output makes more sense.

But what sort of information do you seek in recordings like these, once all that low-level data is lost?

Input-recording can be useful to track down hard to find bugs related to how you got *into* a particular state in the first place. I’m also using it now to help validate that I don’t introduce any gameplay bugs with last minute changes – if a variety of recorded replays still conclude with the same results, then I can be reasonably sure I haven’t broken anything. Obviously this is really only useful when the game is nearly in a final shipping state (but it is *really* useful). It’s very brittle, but that’s kind of the point.

For the purpose of “recording” playtests, what you’re doing makes a lot more sense.

The water is looking great in those screenshots.

By the way, I realise the save game screen is very incomplete, but I noticed that the save on the bottom said “1 panels solved”. Are you planning to bother coding it to say “panel” for when just a single panel is solved or not? I’ve seen a plural used for a value of 1 in lots of websites and games and it’s just a bit unsettling, it should take minimal code to correct it. I know it’s a bit picky of me and not the worst thing ever, but I just don’t see why not.

(BTW: I always wanted to know how many hours of replay data Braid was able to store and how it was compressed. I could only find this so far: http://gamedev.stackexchange.com/questions/6080/how-to-design-a-replay-system )

I am not necessarily super-interested in the specifics of how much people get confused about specific puzzles. It is too easy to way overthink that stuff. That is exactly the kind of information that prods many game designers into filing down their game until it is smooth and generic.

It is okay to get stuck on things if, when you finally understand them, they were worth being stuck on.

If something is a huge blocker, I will see that when people spend a long time there. But, right now I am not even saving puzzle attempts.

Some puzzles in The Witness are brute-forceable but most are not (within practical time and effort limits). If you try brute forcing puzzles, it feels bad; it is obvious you are doing it wrong. I have no doubt that some people will try and play the game that way anyway… but then they are just sort of losing the game.

Sometimes there is a mechanism where panels will shut off and require you to redo the previous panel. This is meant to discourage brute-forcing and also to clearly communicate that you don’t want to play the game that way. I don’t use it super-often though.

Very few people fully understand all the puzzles in Braid. That is part of what makes them interesting puzzles: they provide capacity for varying levels of understanding, depending on how much effort and interest the player brings to them.

How do you, as a game designer, react to the possibility of brute-forcing solutions to puzzles? Did you know of any Braid players solving its puzzles without fully understanding them? (Not sure why, but Fickle Companion and Jumpman levels come to mind, mostly because I’ve read complains about them.)

That is, it’s a really good idea to have a functions that dumps the state of a system. When it comes to debugging you can then sprinkle calls to the dump function around the place and work out exactly where you went wrong.

Once you have a dumper a loader isn’t far behind.

Here is a video of us doing something like that with our random terrain generator:
http://www.youtube.com/watch?v=3PRsHyyFoaM

Each frame of the video is generated from a call to a function that dumps the current state of the map.

We have used this functionality to debug generator bugs, work out differences between windows and linux builds and train artists about how the generator works.

I don’t have too many answers in terms of how this will work out specifically; I am going to just start using it and see what happens.

Second thing I noticed is that the water and bushes and maybe even grass moves, and their shadows with them. I don’t see any shadows of clouds which I thought would be cool. can you could put shadows of clouds on the ground that travel/move over time to indicate some sort of progression of time running by… and somehow affecting the puzzles that have to do with shadows not being on the touch screen panels

P.S. Captcha took 4 tries

It looks like a cool system. I imagine that the harder part of the overall playtest process for you will be deciding what player interactions are actually worth observing. Do you have a particular method to cull it down a bit?

I guess you don’t only want to respond to tester comments, since you’ll need to know how they are interacting with the game outside of just perceived problems on their part.

If it was me, I could imagine getting lost watching how people play to the detriment of my development work.

– Mark