Puzzles and Analytics: Data Insights from ERChamp Stress Test

There’s an oft-cited phrase “Data is the new oil.” The phrase, first attributed to British mathematician Clive Humby in 2006, refers to the tremendous value of data and how data needs to go through a layer of refinement before it can bring any kind of value. This principle applies very much to ERChamp and how we make sure the experience goes as smoothly as possible for participants.

Prior to the Eliminations round of the 2023 ERChamp tournament, we completed a Stress Test. The test serves two purposes- it gives players a sneak peek of this year’s theme and some fresh, engaging puzzles, and it allows the ERChamp team to determine if the infrastructure for the year is running smoothly. When the Eliminations round starts, we need to be confident that the only blocker facing teams will be the challenge of the puzzles.

As a fun side effect of collecting all the logs to ensure the system is running smoothly, we also have the ability to take a peek at how the hundreds of teams participating in ERChamp interact with the game. Sometimes this helps us to adjust difficulty or flow or to ensure an equitable experience across teams with different sizes and different languages. Other times it’s just fun to get a little window into how people think and what unintended paths players explore. This year we’re offering a little sneak peek based on the data collected thus far.

A little bit of housekeeping, and spoilers if you haven’t played the Stress Test yet (and if you haven’t, you can find it here). These are the milestones I refer to in my analysis.

BatteryTaken / WrenchTaken – Objects found early in the game, visible from the very beginning.
BatteryPut – Placed the battery in the correct location to be able to use it to access other locations.
FirstAidOpen / PinPadPower – Examples of using the battery puzzle to make access other puzzle elements.
PinPadSolved – Solve a standalone pin pad puzzle to access the space suit.
SolvedSignal – Solved a puzzle to redirect satellites with a series of button clicks. I also refer to this as the “satellite” puzzle.
PressureSolved – Solved a puzzle to correctly order pressure canisters. Any time I refer to canisters or pressure, I’m referring to this puzzle.
FixedSpacesuit – Used in-game items to repair the spacesuit. The Pin Pad puzzle must be solved to access the spacesuit.
CompletedGame – Solved all puzzles and reached the final screen of the game.

To start things off, let’s take a look at the overall flow of the first chapter. Ninety teams finished the game within the first day of launch. This chart shows the average minutes to complete major milestones throughout the game. The average completion time among these teams was just shy of 30 minutes. The graph tells a really fun story.

Teams blast off at the start. The battery, one of the first elements available in the game, takes just about a minute to find, often less.

The first 5-10 minutes are largely about discovery- exploring the space and interacting with objects. Between 5-10 minutes, teams start to understand the mechanism of the wires and activating different features like the first aid compartment and the space suit pin pad.

After the initial discovery period, there’s a steep incline in the graph, representing the three different paths of puzzles players can explore. Players can solve the signal, pin pad, and pressure canisters in any order, but the average solve time is lowest on the pin pad, then the satellite, and finally the pressure canisters, which averages almost four minutes longer than the other two puzzles.

From a design perspective, this can indicate a few things. It could mean the pressure canister puzzle is more difficult than the other two. It could also indicate that players weren’t sure if they had all elements required for the canister puzzle and waited until resolving everything else to come back around to it.

The difficulty of the Pressure Canister puzzle is echoed by observing the number of teams reaching each milestone. Of the first group of Stress Test solvers, about twenty less solved the pressure canister puzzle than either the satellites or the pin pad.

The biggest dropoff observed was between the game’s “discovery” phase and the solving of any of the three puzzles. Almost 40 teams realized how the battery worked but didn’t make it further than that. But of the teams that made it past that point, more got stuck on the pressure canisters than either of the others.

Of course, some teams don’t get stuck at all. Looking at the shortest time for each milestone, some teams get through the game at blazing speed.

ERChamp is a competition, after all, so it’s not too surprising that some players explore the first two screens in under a minute and take only a few minutes to conquer the rest. Even here the trend holds with the challenge of the pressure canisters, although a six minute mark is quite impressive.

Our ERChamp data can also tell us other fun stories with regards to specific puzzles. Often times the data points out commonly guessed incorrect answers or provides other hints to the difficulty of a puzzle, or how different of an experience players have with it. One example of this is the satellite puzzle.

A histogram shows the number of players in each bucket based on number of clicks - for example, the far left column indicates just over seventy players rotated the satellites between one and forty one times, while the far right column indicates that just shy of ten players made over two hundred moves.

The minimum number of satellite moves to complete the puzzle is less than ten, and over seventy teams got to the solution somewhat quickly. Some took quite a few more, with the highest number being 399 moves by one team. In general, this trend lines up with what we would expect to see for a puzzle like this, but it’s always important to think through the experience for players on the left side and the right side of this histogram. This chart is not filtered to only customers who solved the puzzle, so some of these players may still be glued to the satellite panel, pushing the buttons in hope of the right combination!

These visuals are just a sample of what the ERChamp data can tell us about how to make an ideal experience for players. The game needs to be rewarding for the most competitive teams while still remaining accessible to casual players. The game needs to be fun whether it takes you one hour to complete or ten. There are endless ways to spin and pivot our data, and every one of them helps tell a bit more of the story!