Our Quiz Philosophy

I wrote another article on my personal website about my process for creating quizzes, plus tips and tricks in case you want to design quiz experiences too.

This article, instead, is about a more high-level choice that we made for all the quizzes on this specific online store. An important decision, and perhaps a bit of an invisible one. Which is why we wanted to explain it!

The Issue With Quizzes

At time of writing, we’ve worked nearly 30 full-size quizzes for this store. Many are not 100% finished yet, in case you’re counting and wonder why you’re coming up short ;) Some only have a large list of questions. Some are waiting for some final thumbnail or drawing we need for a question. Some only have a few questions as we’re figuring out the most fun way to approach the topic.

There is one problem, however, that we encountered with every quiz. No matter the topic, or the length, or how we approached it.

In a nutshell … we had way too many questions.

This basically happens with every creative project, so it wasn’t unexpected. Once you start brainstorming, once the flood gates open so to speak, you will pump out idea after idea until you have too many. That’s good! Because now you have a long list of ideas, and you can simply remove the weakest among them, until it’s the right length. Until only the best ideas are left.

Once you start researching a topic, it’s far too easy to get too many questions. Too many interesting facts. Too many fun little details to ask about. Too many associations you can make—and you expect many of the quiztakers to figure out too.

What is “too many”? Well, if you look at other (pub) quiz providers, you’ll find they all agree on some general statistics.

• If the pub quiz is very involved (many difficult questions, many different types of questions, lots of people playing), then 40 questions is a nice number.
• If the quiz is a bit easier and snappier, you can go to 60–70 questions.
• Even so, it’s recommended to cut it into rounds and never have more than 10 questions per round.

As expected, we agree with this! Whenever we tested a quiz of ours, anything that had fewer than ~30 questions felt too short and not worth the energy/price, while anything with more than ~80 questions always felt too long and started to drag by the end. That said, we are fans of simplicity and minimalism, so our questions and expected answers are usually very short. The rare times we crafted more “involved” quizzes, yeah, 80 questions was waaay too much.

With this knowledge, let me now tell you that we usually have over 150 question ideas.

Obviously, that’s way too much.

So we try to shorten the list. We try to prune it, only keeping the best ideas. Except there are two issues.

• “Pruning” doesn’t really work when you need to remove over 60% of the list. Removing a few weak questions is only possible, of course, if you only have a handful of questions too many.
• Different questions have different reasons to be included. Certain questions will speak to one target audience, other questions will speak to another. Some questions might feel too easy … but you need some easy questions to balance out the hard ones and get some nice variety. And so on, and so on …

In practice, there are usually only 10–20 questions that are clearly not that interesting or not “good enough” for whatever reason. After removing those, we’re still left with too many questions. And none of them feel like they can be removed, because all have good reasons to be in a quiz like this.

Solution 1: Multiple Quizzes

This is why you’ll notice that we’ve split most topics into multiple quizzes. After a while, we settled on …

• “Kids”: the easiest, shortest, most textless questions we had.
• “Family”: most questions are of average difficulty and interest to people, so they fit here.
• “Pro”: all the hardest questions, more experimental/creative ideas for rounds, longest quiz

This is a fine structure. It works in practice. Splitting 150 questions into three quizzes like this means every quiz can have a nice length. Usually, we can easily tell if a question should be in “Family” or “Pro”. In the Family/Kids quizzes, we add more hints and more subtle things to help guide you towards the right answer. In the Pro quizzes, it’s more about deep thinking or expert knowledge on some topic.

If needed, we can also simply rewrite a question to make it fit somewhere else. Most “bits of information” have an easy way to ask about them … and a hard way.

But this has a downside. As the online store grew, this “littered” the store by creating 2 or 3 quizzes for a single topic … almost every time.

Solution 2: Easier To Cut Than To Add

If you’ve used our quiz system, you’ll know that it’s very flexible. There are many easy ways to entirely skip rounds, for example.

Now you know the reason!

It is much better for us to add too many questions (and allow you to skip/cut it short), than the other way around: leaving you with too few questions and too short of an experience.

It’s pretty impractical to ask you, the buyer of a quiz, to somehow make it longer. Or to buy two quizzes because you wanted a 3-hour experience instead of 1-hour.

It’s much easier if we simply add more questions and clearly explain you don’t have to do them all. (At least, not at once.) In this way, for a low price, you basically buy two or three quiz experiences for the future.

Once we realized this, and our quiz system was solid, we shifted towards this approach. At least, a little bit. It became more common for our quizzes to have 100 questions included, but they’re cut into 10 clear rounds … and you simply don’t play them all.

Unless you really want to, of course. That’s the power. The choice is yours! It’s flexible and easy, while allowing us to put everything into one product.

That’s why, if you look at our quizzes over time, you’ll see less Kids/Pro versions than before. They still appear when they make sense, but not always.

Because, after making a few more quizzes, we had a third realization …

Solution 3: Hyperfocus On Target Audience

Remember how I talked about every question having a “reason” to be there?

A question that asks about a number will interest people who like math/numbers. A question that asks for a word, or to play with language, will interest people who like that. A question with clear educational value will appeal to teachers and classroom experiences. A question with a sound bite/short video will appeal to music nerds. And so forth, and so forth.

The meaning of the “Family” quiz, in our eyes, is to appeal to a wide variety of target audiences. That’s the idea! A family (or group of friends/classroom) will contain a wide variety of personalities, so the questions should be diverse too.

But we actually don’t want this for the other quizzes. It makes it hard to market the quiz in the first place, because the person buying it doesn’t know for sure if they’re the right target audience. But it also simply ruins the experience a bit. When you, say, bought a quiz about video games expecting many technical/numeric/computer questions … but it also asks loads of questions about visual stuff, or the money/business side, or whatever else.

We want to be able to say “This quiz is for you if …” In such cases, want to design quizzes that mostly stick to a specific type of question or “problem solving”. The quiz should be focused in that way, with only a “sprinkling” of other question types. Because that allows us to clearly communicate if a quiz is for you/your playing group or not.

In general, this means …

• “Kids”: questions are as textless as possible, with as much media as possible (image/sound/video). It’s about simple recognition and knowledge on topics that interest (young) children. The quiz should be so short and snappy that you can use it educationally, going through it several times and discussing answers.
• “Family”: as stated, this has the widest variety of question types, with a few speaking to every possible player.
• “Pro”: questions require longer thought or actual problem solving—they’re more puzzles than just flat-out have you memorized this specific thing or not?. Rounds are more creative and unique. The questions are only for people who are very experienced with or very interested in the topic at hand.

If a topic warrants it, we might have a more specific categorization than this. For example, a topic such as “Music” is so broad that you can design twenty different quizzes about it, each targeting a completely different audience. (One for musicians, one for people learning an instrument, one for general music enthusiasts, one for people who go to festivals/concerts, etcetera)

But in 99% of the cases, this structure covers most of it. It’s simply up to us to communicate this clearly and stick to that focus.

All Together

Bringing it all together, the general process for designing quizzes evolved into the following over time.

• We pick a topic and research absolutely everything about it.
• We come up with some creative round types or fresh ways to ask these questions.
• We prune a handful of questions that are just too boring/weak/similar to others.
• The absolute easiest, textless questions can go to a Kids version. This one gets only at most 30 questions; really, most time is spent on the media needed for it.
• All the creative and experimental rounds get the deepest questions, and move to a Pro version. This one can run up to 100 questions; most of all, we really focus on the creative approaches and hard questions. This often means removing questions that are fun or interesting … but just don’t fit the focus.
• What remains is collected into a balanced and varied Family version. (Which is, for most intents and purposes, the “main” version that’s bought and played the most.)

That’s why you’ll find a file called discarded with most quizzes (next to the raw questions file), which often contains questions we removed, research notes, and more that didn’t make it.

Quite often, however, this work is not wasted. To ensure quizzes don’t get too long/stale, we are most likely to cut an entire round. (Instead of being very cautious and only cutting one or two questions here and there.) Because the round already has a self-contained theme, we can usually place that exact round into some other quiz we might make in the future.

A final question might remain: Why do you brainstorm until you have too many questions!? Why do you not just gather 40 questions and stop there!? There’s no actual problem here!

If you’re not a creative/artist yourself, then I fully understand why this seems silly to you. A lot can be written/said about the creative process, so I’ll just try to summarize it here.

Reason 1: You often don’t know if an idea is going to work out or not. You don’t know which of the questions are fun or not, or which round will actually get 10 questions and which will simply not be interesting enough to get more than 2.

You have to see things through to the end … and then look back to pick out the strongest parts. To make it a cohesive whole.

If you just take the first 40 questions that come to you … you’re likely to have many boring ones in there. You’ll struggle to cut them into rounds, because it’s a random assortment of questions and you don’t have any others to work with! You’ll only get the low-hanging fruit instead of reaching for more creative questions or higher potential. Most of all … you don’t know if any of this is the case.

That’s why all creative processes start with “brainstorming”—or diverging. Going from a seed of an idea to absolutely anything you can think of. Once that’s done, you switch to converging: bringing that massive list of ideas back to one final project. And, well, with a bit of experience and effort you can easily diverge into 150 question ideas.

Reason 2: It’s hard, in general, for people to throw away work that they’ve done. I’ve been teaching myself for 15 years that I’m allowed to “kill my darlings” and “throw out the garbage” … and I still struggle. I can only allow myself to ruthlessly “cut the fat” with tricks like moving it to another future project, or rewriting 10 questions to cleverly fit inside only 5 ;)

In many situations, things that are “cut” from a project are not “bad”. They simply didn’t fit inside that project (anymore).

As such, spending hours crafting more questions and rounds than needed isn’t “wasted”. It’s something for which we can find the energy because we know we can likely use that work in something else later. But after putting so much effort into it … it’s simply hard to throw it away. And so we tend to maintain a lot of that work, and there’s no real harm in putting it inside a separate quiz experience.

Conclusion

These are the lessons we learned and our thought process behind how we design, label and categorize our quizzes. As with most things, it evolved naturally as we got more experience and realized what worked and what didn’t.

If you buy an older quiz from the store, you might not recognize anything we talked about here. That one was probably made before this became our standard. With more recent quizzes, you’ll hopefully see what we mean. And, of course, we hope you enjoy playing them.

We make an effort to explain this clearly too. Most people just don’t realize that you don’t have to play the entire quiz. It’s similar to how some people don’t realize that you can simply … stop playing a game if all players agree it’s boring or taking too many time. I’ve played board games with many different people, and it’s really not that uncommon to say: “Are you having fun? No. Me neither. This game turned out way different and way harder than we thought. Let’s just pick something else.” As such, we try to make it easy and obvious that rounds are standalone and you can skip/modify to your liking.

Because if one thing has become clear, hopefully, it’s that no single quiz experience will suit everyone. We all have different questions we like or don’t like. We all have a different amount of time or attention we can spare on such a quiz night. We all have a different general interest in the topic being quizzed, in the first place.

Despite all our efforts, the best way to improve your quiz experience will always be to modify/tweak it yourself if needed. You know yourself and your group best. We are absolutely fine with you cutting out half of the rounds that we worked so hard on, if you just don’t have the time ;)