Log in

How the Grinch Almost Stole Mystery Hunt, but I Stopped It, with a Little Help from My Friends - devjoe [entries|archive|friends|userinfo]

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

How the Grinch Almost Stole Mystery Hunt, but I Stopped It, with a Little Help from My Friends [Jan. 18th, 2016|10:57 pm]
[Tags|, , , , ]

My long-time readers are used to these posts, multi-part wrapups of Mystery Hunt over the week or so following. In this week's posts, the links will go to our muttsteryhunt.com site which currently has the puzzles and most of the solutions. In this part, I will describe some of the activities I was involved with immediately before and during the first day of Mystery Hunt. Since some of this involves test-solving puzzles, there will be spoilers (but in this part, not much for the kind of puzzles people are likely to want to do after Hunt).

Like apparently happens almost every Mystery Hunt, there were puzzles not test-solved or not working right up to Hunt. I took Thursday off work and arrived at MIT late that morning to start testing mostly on-site runaround-type puzzles. Once we got reasonably set up in our headquarters at 10-105, i started doing runarounds.

Bowen and I had solved Follow the Pipes up to the part that became a runaround in MIT basements the night before, so that was first on my plate. Some of the locations were right below our headquarters so I went there and, due to the name of the puzzle, looked for some clue on the pipes which are visible above some MIT hallways but especially in basements. I found nothing at three sites. A bit later, I went out with the author briefly and we visited this site and one other and I found what I was looking for at the second site, but not at the first one, 10-063, which the author pointed out to me had three doors, even after checking all three. I postponed further work on this to attend a test-run of the Escape from Mars event, but afterward I came back to work on this puzzle.

The Escape from Mars testing went pretty well, but the event ran a little long. They decided to give teams a little more time, to simplify the coloring puzzle so its number could be identified more quickly, and to make a couple other changes to fix things that were unclear or just wrong in other puzzles.

I went back out, with Shrenik this time, looking for a thing I had now seen one of, a drawing of a sword on paper with a multi-colored blade. I let him find the one I had already seen without any help, and he did so. After finding a couple more, and not finding them at other sites, we decided that we only needed to look at the locations that had the red circles numbered 1-10 on the map. Shrenik correctly guessed that the colors corresponded to the lines on the map. By following a path on the map through these colors, a letter would be formed. (It is apparently also possible to follow the real pipes in this way, but that is the time-consuming way to do it!)

We found 7 of the 10 swords and guessed the answer from O??I?EPORN, keeping in mind that it was a puzzle about the Internet. The ones we did not find were at 10-063 (which turned out to be because it was six feet beyond one of the doors; we made them move it to directly in front of one of the doors), 1-029 (because we had mis-decoded the number and gotten a room that did not exist; it was easily found at 4-029), and 57-001 (because we did not know where to look. We could not enter 57, the Alumni Pool, without an MIT ID, and we looked in the basement of Stata which got us close, but an emergency exit map led us astray; if we had gone in a different direction from this map, we would have found it).

After this, I tested the Dog Show Runaround, the puzzle you got after completing the Dog Show meta. Completing this unlocked the next round and revealed to you that you were in a dream - a series of dreams, even - a la Inception. It was a standard set of directions for navigating campus, but "dog" had been substituted for many words, as "smurf" was used by those little blue guys. This was too long, and solvers got a shorter version with clearer text at the part where you re-enter Lobby 7 and cross it to take the stairs down on the other side.

I also spent some time looking for a thing that we couldn't find. dalryaug wanted to use a set of things that should exist at MIT for a runaround, but had never located one of them. My group this evening failed in this endeavor and a backup puzzle was subtituted.

I had earlier asked Megan, one of the MIT students on our team who was not spoiled on Field Scribe to look at my notes and add her contributions, and she did so. When I came back to HQ after the above activities, I found another two teammates (one a student) working on the puzzle. Both Megan and the other group had independently determined that the photos were all in the Infinite Corridor, and expected to draw something by connecting the dots in this vertical plane. But the other group did not have my and Megan's notes, so when we collaborated, we had almost all of them, and could limit the remaining ones to certain parts of the corridor (though on any floor). I went out and located the remaining ones, and we completed our picture. Adding this data to the drawing, after a few guesses I was able to figure it out (Alvah), but we thought the v was much too closed, and one photo was retaken, moving it from just past 10-250 to the women's restroom in building 4. (And yes, I know that the solution for this puzzle is missing everything but the final answer. I have added it to our to-do list.)

Also this evening, I designed the point totals for puzzles to open at. We tried to balance the rate at which fast teams could open up the whole hunt with the danger of stuck teams not being able to advance past a certain point.

Overnight, I solved a bunch more puzzles. I forget all of what these were. It was nearly dawn when I trudged to the hotel room and collapsed into sleep.

I only slept for 3 hours, hearing some bustle in the room. Overnight, our web team (who had put lots of hours into making sure the front end was solid) discovered bugs in the back end, a program called spoilr which has run the last few Mystery Hunts. (Note that most of the details about spoilr are second hand, so some details might be slightly wrong, but the essence of this is correct.) Because of the way their hunt worked, where all puzzles appeared on one big map, and metas appeared along with the regular puzzles, 2015's team had basically removed rounds from spoilr as a concept (rather than just making everything one big round, as they should have done), and removed the ability to make solving one puzzle in a round open up more puzzles in that round. The ONLY way to open puzzles was to accumulate score points. Our web team, led by Mike Booth, had attempted to restore this, working from an older copy of the code. They had done some simple test runs, but a large scale run simulating an entire hunt the night before had revealed previously undiscovered bugs. I don't know the details of the bugs, but they could have broken the hunt badly for users, and it wasn't possible to fix them before Hunt opened. Rather than try to continue hacking on the code while the site was live, possibly bringing down the entire hunt if they screwed something up, they went to Plan B. A plan they were hastily making up.

By the time I showered, dressed, and got back to HQ around 10 AM Friday, the web team had decided that Plan B for the front end was going to be multiple static sites run on subdomains of muttsteryhunt.com, a domain name we had originally intended to use only for the dog show round. Each domain name used a set of random characters after the name of the round, which (in combination with the DNS server hiding the list of all registered subdomains) prevented teams from guessing the sites for future rounds. Each site also contained all earlier rounds, so once a team switched to a newer site they could use just that site - or at least, this was what they eventually realized they needed to do. The scoring Implementing this was actually the work that kept Hunt from starting on time, though at least we were only one hour late.

This meant each round would be released all at once, rather than the intended plan of releasing 3 or 4 puzzles at first and then two more puzzles with every 1-2 solves until the whole round was open, with a higher point total as a backup for teams that get stuck. The point thresholds for opening each round were retained, with minor tweaks.

This also meant there was no logging in of teams, and no answer tracking system. Other parts of our team were trying to come up with a solution. Around 10:30, I suggested "What if we use Google Docs forms for answer submission?" I explained further that the answers all go into a Google spreadsheet which can form a callback queue and guess log. A formula column can automatically check answers from an answer list on another sheet, and we can use a manually entered column for tracking the callback status. "The only thing," I continued, "is that I have never actually created one of these forms, only used the spreadsheets that resulted from, for example, the registration form for one of our BAPHLs."

My teammates had favorable reactions, some along the lines of "That just might work," and considered this a better idea than anything they had come up with in a short time of knowing they needed a solution. But now they looked at me as if they expected me to create it, so I got down to learning how to make this form. Fortunately, it was really easy, and in an hour, with a little help from Jasters, I had a prototype up with two test teams on a dropdown list of teams, and callback status field in the spreadsheet, as well as a URL format delivered to the front-end team, into which they could insert the name of the puzzle to pre-populate that field, and also a contact HQ form. The system seemed to work, and we agreed to use it. I added the full list of teams to both forms (painfully - it seemed there was no way put pasting each name in individually) and got the real answer database instead of my test one, while Jasters added answer checking. We also put in a formula to pull the team's phone number from the team list. This was ready to go, in a rudimentary form without score tracking, at the time Hunt should have started.

That score tracking was necessary for determining when to release rounds. Due to the way we were doing it, in the spreadsheet, it was only ever going to be seen internally. Sorry, teams. It was more important to be able to get you puzzles to solve, and this was a tradeoff we had to make for that. Derek Kisman worked on scoring in the early afternoon, even after the hunt actually opened, knowing that the Dog Show round was at least two hours long (Palindrome was the first to solve it at 4:09, so this was a near-perfect estimate), Rip van Winkle only opened on completion of the dog show runaround, and teams probably needed at least an hour to unlock the following round, so we had plenty of time to make this right. The last part of this was dealing with the fact that with sometimes as many as 10 of us doing callbacks and thus having the spreadsheet open, and a lot of activity in the spreadsheet, and lots of formula activity, the sheet bogged down. This was dealt with by moving scoring tracking into another Google doc which most people did not have to have open. Yet a third Google doc just had the scoring summary - score, status on each meta, and at the end, status on each Limbo puzzle - and this was what regular team members just wanting to see the current scores would look at. The forms were almost all me; the spreadsheet hacking, after my initial setup, was Jasters and Derek.

Once this was settled, it was possible for me to look at more puzzles. One of the ones I looked at was To Serve and Protect. This was originally called something different and appeared in the Endymion round. A puzzle in King Arthur that required constructing some physical contraption with wood and mirrors that had not gotten done needed to be replaced. To Serve and Protect was moved in to replace it, to give up more time to make the replacement puzzle. But it needed a new appropriately Arthurian quest, and so it needed the 11 letters that spelled out the answer to be replaced. I looked over this with Gabby and confirmed that these 11 letters could be changed to anything without affecting the rest of the puzzle. So now we had to think of what they should be. A couple of my ideas were shot down because we already had similar things in the round. We thought about SLAY A DRAGON but we didn't have a good way to do it that didn't also require constructing something, though I suggested jokingly that they could play TROGDOR! until they managed to get him killed. :-)

A little while later, though, I came up with LOCATE GRAIL which everybody loved. We were not sure, however, how we were going to have them do it. canadianpuzzler suggested playing a version of the 3 cup shuffle game which is often used as a street hustle. This could perhaps three identical opaque plastic cups, one of which is marked inside as the grail. They'd get three tries, and if they failed, we'd send them away to make another appointment later. In any case, I redid the image for this puzzle due to the author being too busy to do so, and we inserted the new image into the site and got the puzzle renamed and moved to its new location in the hunt. When teams opened King Arthur (soon!), they would not have a missing puzzle. That is where I left it when I went to the hotel around midnight, but when teams first reached this stage (Left Out appears to have been first, around 2:30 AM), neither of us was around and the details were lost to the people still awake, so they instead had people construct a grail themselves, based on the guidelines that it must have a stem and hold liquid. This was probably a lot better idea and certainly more fun. It resulted in a display of some of the grails at wrapup.

[User Picture]From: nonelvis
2016-01-19 02:07 pm (UTC)
OH MY GOD. I was wondering what specifically had broken to require the unique URL scheme you'd pulled together, and now that I know, my god, what a mess. This also explains why if you loaded a round from an older URL, you couldn't get to any of the newer content.

Under the circumstances, you guys did an amazing job with your backup plan, and at least now SETEC has plenty of time to fix spoilr for next year!
(Reply) (Thread)
[User Picture]From: devjoe
2016-01-20 12:07 am (UTC)
Thanks. I've edited the description above to indicate Derek Kisman's role in spreadsheet hacking, which I didn't realize. This is what happens when you have a TEAM: You get people helping who you don't even know are helping.

And yes, I am sure Mike will be talking with tech people on Setec to explain what was broken, so they can fix it early if they want to use spoilr.
(Reply) (Parent) (Thread)
[User Picture]From: aarondf
2016-01-19 08:56 pm (UTC)
Very interesting. Thanks for the write-up. I was disappointed with how vague Mike was at wrap-up although I can understand why he was.

And I should add that it was a fine solution, amazingly good given the limited time to act!! I and many others I spoke to were actually very happy with the whole rounds being released at once. You take a short break when that happens to look over the whole round and see if any puzzles are right in your wheelhouse and then either move on to the new one or go back to the old. I guess this is the same thing I do normally but I am having to do an interruption to check the new puzzles much more often; I prefer this (along with pi).

Edited at 2016-01-19 09:00 pm (UTC)
(Reply) (Thread)
[User Picture]From: vvvexation
2016-01-20 02:34 am (UTC)
Yeah, I think I prefer it too.
(Reply) (Parent) (Thread)
From: mechfish
2016-01-20 02:59 am (UTC)

Late software project is late!

Hi, this is Mike Booth! I was, and perhaps still am, planning to write a more thought-out post-Hunt tech retrospective once I catch up on my sleep, which might be as soon as a week from now. Though I am more tired than I've ever been in my life, so maybe it'll take a month? Let's all keep our fingers crossed.

But I want to put in a good word for Random's 2015 codebase, which was well designed and which we put to good use -- it rendered all our puzzles, and it supervised our dynamic puzzles.

It is true that the 2015 codebase had a lot of 2015-specific customizations, some related to its funky structure. And I considered rolling the code all the way back to 2014, but that code had 2014-specific elements of its own, and it was missing a bunch of 2015-era improvements that I was loath to lose.

And in the end, customizing the code for a particular Hunt is just one of those chores that needs to be done. 2016 had a custom unlock mechanic of its own, after all. (Dreamtime, which amused us enough that we kept it, even though we had to implement it by hand with a stopwatch.)

Everyone asks what the technical problem was with the Hunt, but I have no answer; that wasn't the nature of the problem. Our problem was one of capacity, and the difficulties we were having at the end weren't necessarily with the parts that were most "broken", or with the parts that cost us the most time; they're just the things that we happened to be working on at the moment the clock ran out.

The Hunt was a big project with a software component, an utterly immobile deadline (plus or minus one hour), a limited budget, a limited number of volunteer programmers, and a scope that is set by tradition. Our writing team was not large, and rather than deliver a correspondingly smaller Hunt we operated with a reduced factor of safety. We took on a lot of project risk, and we're fortunate the cost was not higher -- there were portions of this year's Hunt where the critical path ran through one person, and it's made me a nervous wreck. My fundamental message to future writing teams is: Be larger. Have more redundancy.

The other advice is to be ruthless about scope. We seem to have had good luck this year with a round-by-round release system. If we had made that choice a few days earlier, I think we might have freed up enough time to save the answer-submission system. But you know what they say about hindsight!
(Reply) (Thread)
[User Picture]From: rivenwanderer
2016-01-20 04:05 am (UTC)

Re: Late software project is late!

As one of the people involved in the 2015 tech, thanks for clarifying this :) All told, I was relieved to see your Plan B working so well. Congrats on making it work!
(Reply) (Parent) (Thread)
[User Picture]From: teferi
2016-01-20 04:40 am (UTC)

Re: Late software project is late!

What Riv said, and we were really glad to see you were able to pull it off!

(A few minor points: we actually did have the concept of rounds, and the points system was entirely separate from the core puzzle rendering + answer submission logic. In fact, we had two entirely separate unlock systems; one for the workshop round (puzzles unlocked in a graph) and one for the main ocean (puzzles unlocked by points requirements). Solving a puzzle fired an event that we registered handlers for to cause unlocks to happen as necessary.)
(Reply) (Parent) (Thread)
[User Picture]From: cananian
2016-01-20 06:44 pm (UTC)

Re: Late software project is late!

We also deferred writing the hunt-running software in the 2012 hunt, although not quite so long. I described our last minute rush in http://motris.livejournal.com/181790.html?thread=1518366#t1518366 -- but I'll repeat the advice I gave there: do your test-solving with the hunt-running software! That will ensure (a) it's done on time, and (b) there aren't red herrings related to the final appearance of the puzzles.

I really enjoyed this year's hunt, but *man* were there ever red herrings in the post production. Just to name two examples, I can't begin to describe how much time we wasted identifying dogs in the dog show meta, or on the "sheep clock" in the Endymion meta. (I suppose its possible these were used in some part of the Limbo metameta, but they were very misleading regardless.)

It appears that Luck didn't have a separate "fact check" stage before the test solve. We insisted that puzzle authors provide complete solutions, and we had a separate team check the puzzles using the solution *before* letting loose test-solvers on it. This is a much cheaper way to ferret out broken clues -- and it (again) has the side-benefit of ensuring that correct and complete solutions are in place *before* the hunt begins.

I don't envy Luck the task of trying to get their team to finish up the solutions now after the hunt.

Edited at 2016-01-20 06:55 pm (UTC)
(Reply) (Parent) (Thread)
[User Picture]From: canadianpuzzler
2016-01-21 01:15 pm (UTC)

Re: Late software project is late!

I need to get caught up on sleep before I write copious amounts about Mystery Hunt (which I do intend to do eventually), so I won't reply to this in very much detail now.

I do wish to point out, however, that even when other considerations are not taken into account, the expense of throwing people into a role is twice as much for a team of about fifty people as it is for a team of about one hundred people.
(Reply) (Parent) (Thread)
[User Picture]From: doctorskuld
2016-01-20 08:58 pm (UTC)
Props to pulling this off given the circumstances and jury-rigging a Google Docs system in the very best of MIT styles. I found most of the puzzles well-written and very elegant, and errata were pretty minimal.

Thanks for a great Hunt!

Edited at 2016-01-20 09:00 pm (UTC)
(Reply) (Thread)
[User Picture]From: beinsane
2016-01-21 01:40 am (UTC)
I was stuck on Follow the Pipes for most of hunt. After reading the posted "solution" to it, I understand less about it than I did before. Any chance of a more thorough solution getting posted? Or, heck, just an explanation of how to get anything out of n fhojnl znc bs QAF ebbg ybpngvbaf would be fine.
(Reply) (Thread)
[User Picture]From: zandperl
2016-01-21 02:14 am (UTC)
Fascinating! We (Grand Unified Theory of Love) had basically guessed much of this scenario. Our guess was that Luck had chosen not to use the archived backend code and had instead had the graphics team codge together some new code, for reasons unfathomable to us. We guessed that went down from what was essentially the DOS attack of kick-off. We then figured that in the two hours between kick-off and official launch, y'all had cobbled together the multi-URL and Google Forms thing. (We did a warmup mini-hunt that used Google Forms for answer submission.)

We further guessed that unlocking of rounds was done based on number of puzzles solved for the faster teams, and just wholesale for mid-pack and later teams (such as GUToL), though it sounds like you actually did have a formula.

The one thing I'm wondering about still though is the purpose of the events, such as "Adventures in Dreamland." These do not seem to be in the posted puzzles/solutions at all, sadly, so I'm guessing they were for a hint system that did not get implemented due to the breakdown and workaround. The first Luck visitor to our room though (Saturday evening I believe) did say that we could ask for hints if we did so judiciously.

Thanks for posting! :)
(Reply) (Thread)
[User Picture]From: Dan Katz
2016-01-21 08:42 pm (UTC)
I would guess that events gave teams additional "points," and rounds were unlocked when teams achieved certain score benchmarks. So an event movd every team forward from where they currently were, though it might not immediately unlock the next round.
(Reply) (Parent) (Thread)
[User Picture]From: devjoe
2016-01-21 09:41 pm (UTC)
That is correct, Dan. We actually gave a substantial bonus in unlocking to events, so completing an event was worth 5 regular (not dog show or dreamtime) puzzles.
(Reply) (Parent) (Thread)
[User Picture]From: devjoe
2016-01-21 09:46 pm (UTC)
We have a to-do item to post whatever we can about the events. We can probably post the entire set of materials for The Matrix and their solutions, for instance. For the trivia event, we would probably just post the rules for the event, as we drew randomly from cards which passed testing for most of the areas there, from actual 1980s Trivial Pursuit cards.
(Reply) (Parent) (Thread)
[User Picture]From: livejournal
2016-01-23 07:59 am (UTC)

I dream of puzzles

User dr_whom referenced to your post from I dream of puzzles saying: [...] d. The Hunt server problems were a pain, but I salute team Luck for hacking together a solution [...]
(Reply) (Thread)