When I started QuiJutsu I assumed the hard part would be the game: matchmaking, timers, scoring, making sure two players in different countries see the same question at the same moment. That turned out to be the easy part. The hard part was the questions. A trivia game is only as good as its worst question, and a bad question in a ranked match does not just annoy someone. It costs them rating points they earned.
The bank today holds more than 4,700 approved text questions and close to 1,000 visual ones, across a few dozen anime. This post is about how they were made: the rules every question has to pass, how the answer options are built, how difficulty is graded, and the two systems I use to produce them. I am also going to be honest about what went wrong, because most of these rules exist because something broke first.
What a good question is
After a lot of trial and error, I boiled it down to four properties. Every question in QuiJutsu has to have all of them.
- One correct answer, with one wording. If two people who both know the answer could write it down differently, the question is not ready.
- Answerable only by someone who watched. You should not be able to guess it from the shape of the sentence or the look of the options.
- Tied to a specific work. Every question opens with "In [title]", so you always know which show, season or film it is about.
- Not a repeat. The same fact asked twice in different words is still the same question.
Those sound obvious. They are surprisingly easy to break. Early on I had a Chainsaw Man question asking how one of Makima's victims died, where the stored answer was "Crushed". Someone who knew the scene perfectly well could have said "squashed", "pressed" or described it some other way. The fact was fine; the answer had no single wording. That question was rewritten, and the rule about canonical answers came out of cases like it.
The question rules
Beyond those four properties, there is a list of rules I check every question against.
Anime only. If something only happens in the manga or the light novel, it does not go in, even if the anime later adapts it. Players should never lose a point because they watched the show instead of reading ahead. Filler is labelled as filler, and manga-only material is kept out of matches entirely.
It has to be verifiable on screen. I need to be able to point at the episode where the answer is shown or said. Character heights are a good example of what this rules out. Profile sites list them, but nobody learns that Loid Forger is 187 cm tall by watching Spy x Family, so a question about it tests whether you read a wiki, not whether you watched the show. Every single height question I tried was rejected for that reason.
The answer must not be in the question. This one sounds silly until it happens. In one Hunter x Hunter question the stem described a Nen ability based on darts, and the answer was "Darts De Darts". Anyone could match the word without having seen the episode.
Counts are allowed only when the set is closed and shown. "How many Sailor Starlights are there?" is fine, because the show presents the group and the answer is three. "How many times does Goku use the Kaio-ken?" is not, because nobody counts that and the number depends on what you count.
No episode numbers as answers, and no trick questions. Knowing that something happened in episode 14 is not anime knowledge. And a question that is technically true but designed to mislead is not fair in a ten second round.
Short enough to read in five to seven seconds. You get ten seconds per question. If reading takes eight of them, the question is testing reading speed.
Judge the answer, never the shape of the question. This is the rule I learned the hard way. For a while I had an automatic filter that threw out any question containing the word "why", because "why" questions tend to have long, arguable answers. Then I noticed it was rejecting Dr. Stone questions about the Why-Man, a mysterious figure whose name happens to contain the word. A filter on the word "goal" threw out a perfectly good question about the Kingdom of Science's central goal. The lesson was that the wording of a question tells you almost nothing about its quality. What matters is whether the answer has one canonical form.
Why the wrong answers matter more than the right one
Every text question has four options: the correct answer and three distractors. I spend more time on the distractors than on anything else, because that is where the difficulty actually lives.
Here is the problem. Fans do not only answer by remembering. They answer by elimination and recognition. If the correct answer is a main character and the three wrong options are background characters nobody remembers, a player can pick the one name they recognise and be right without knowing the fact at all. So the rule is that distractors have to match the answer's prominence. A main character's name sits next to other main characters' names. A signature technique sits next to other signature techniques.
Then there are the structural rules, which I check mechanically on every question:
- Three words or fewer per option. Long options are slow to read and tend to give themselves away.
- No option starts with "a", "an", "the" or "to". The answer should name something, not describe it.
- No aliases. A distractor cannot be another name for the right answer. In Sailor Moon, "Silver Crystal" and "Ginzuishou" are the same object in two languages, so they can never appear together.
- No second correct answer. This is subtler. "School festival" and "culture fair" can both be true descriptions of the same event. "Racing" and "motorcycling" can both describe the same character's hobby. If a distractor is also true, the question is broken.
- No format tells. If three options are mountains and the fourth is a city, or three are shrines and one is not, you can spot the odd one out without knowing anything. All four options have to be the same kind of thing and look alike.
A good test for a finished question is to hand the four options to someone who has never seen the show. If they can do better than a coin flip, something is leaking.
How difficulty is graded
Every question gets a level from 1 to 10, and the level puts it in a band: easy (1 to 4), medium (5 to 7) or hard (8 to 10). The scale has anchors so that a 3 means the same thing for every show:
- 1, instantly recognisable: the main character, the core premise, the most famous transformation.
- 3, surface recall: famous attacks, well known rivalries, lines people quote.
- 5, engaged viewer: turning points in an arc, why a character made a decision.
- 7, dedicated fan: how a sub-plot resolved, tactics in a specific fight, the limits of an ability.
- 9, precision recall: exact dialogue, small visual details, minor lore.
- 10, extreme detail: things even someone who watched last week might have missed.
The test I use for the easy band is simple: would someone who watched this once, a year ago, still know it? If yes, it is easy. Medium is for someone who watched attentively and finished the series. Hard is for the moment you would only know if it stuck with you.
The most important thing I learned about difficulty is that it comes from the topic, not from the wording. You cannot make an easy fact hard by phrasing it awkwardly, and you cannot make an obscure fact easy by phrasing it clearly. So I now decide how well known a topic is before any question about it gets written. On one show, before I did this, only 13 of 81 questions meant to be easy actually came out easy. On Haikyu, after the change, it was 31 of 48.
I also grade on how a real viewer remembers things, not on how obscure a fact feels on paper. That matters more than it sounds. When I let an AI model grade its own hard questions, it rated them 3 to 5. People rated the same questions 9. Facts always feel easier to whoever just looked them up.
System 2: planning before writing
QuiJutsu's questions come from two systems. The one that produced most of the bank I call System 2, and it uses AI models to draft questions, with a lot of guard rails around them.
The first version did the obvious thing: ask a model for 160 questions about a show. It does not work. You get the same 30 famous facts, reworded over and over. So System 2 starts with a blueprint instead. A blueprint is a list of topics, which I call slots, worked out before any question exists: characters, relationships, arcs, abilities, locations, and specific events like "the object Ishigami gives Iino". Each slot is tagged with how well known it is, which is what sets the difficulty.
The blueprint turned out to be the ceiling on everything. A show with 122 good slots gives you about 122 good questions, and no amount of extra generation gets you past it. I found that out on Chainsaw Man, which took 211 rounds to reach 475 questions because the later rounds were scraping the bottom, with only 4% of drafts being accepted. That run is why the system now has stop limits. Without event slots, Kaguya-sama's blueprint had 97 of its 150 topics squeezed into the first season, because season one is what everyone talks about.
After the blueprint, each question goes through a set of separate steps:
- Compose. A model writes a question for a slot, with every existing question for the show in front of it so it does not repeat one.
- Evaluate. A separate step grades it. A writer grading its own work rates what it meant to write, not what it actually wrote.
- Options. The distractors are built and checked against the rules above.
- Source check. Is this in the aired anime, or only in the manga, or filler?
- Duplicate check. First by meaning, then by asking directly whether two questions test the same fact.
The duplicate check deserves a note. "Who is the Flame Alchemist?" (Roy Mustang) and "What kind of alchemy is Roy Mustang known for?" (flame) look like the same question. They are not. One asks you to go from the title to the person, the other from the person to the ability. I keep both, and the match builder makes sure they never land in the same match. Early on, a purely similarity-based check threw out 32 good Haikyu topics this way.
I also only generate for seasons I have agreed on first. A show's full run is not always its best run. Sword Art Online is limited to its first season, and Tokyo Ghoul to its first, because those are the seasons people actually watched and liked. And I do not generate for anything that aired after the models' training data, because when I tried, a request for one show's second season came back with questions about its first.
System 1: reading the episodes
System 2 has a weakness I could not engineer away, and I will get to it in a moment. System 1 is my answer to it. It does not ask a model what it remembers about a show. It starts from the episode itself.
For each episode I work from the cleaned subtitle file, read every line, and write questions only about things the episode actually shows or says. Each question is saved with the lines that support it, so anyone reviewing it later can see exactly where the answer comes from. Anything the subtitles cannot settle on their own, like who said a line (subtitles do not name speakers) or what an object looked like, goes into a "confirm from watching" list instead of being guessed. Every new question is also checked against the questions already approved for that show.
It is slow, and that is the point. The first show done this way was Mushoku Tensei, all 62 episodes across three seasons, which gave around 800 approved questions with evidence attached to every one.
The part I did not expect: human review is not optional
Here is the weakness. Across every version of System 2, roughly 17 to 21% of drafted questions stated something that was not true. Not badly worded. Wrong. The number barely moved no matter how the prompts or the checks changed, and the automatic fact-checking step could not reliably catch them, because it was relying on the same kind of memory that made the mistake.
So nothing goes live on a model's word. Questions are reviewed before they are approved, and the reviewer works from the episodes, not from memory. One rule I hold reviewers to is that silence in a transcript is not proof that something is wrong. If a question asserts something the subtitles do not mention, that might mean it happened on screen without dialogue. You check, rather than reject.
Translating the bank into other languages also turned out to be a second review. When a question has to be rewritten in Japanese or French, an English question with two defensible answers suddenly becomes obvious, and a number of those were fixed that way.
The last line of defence: you
Even with all of this, some mistakes get through. That is why every question can be reported from inside a match, and reports go straight to a queue I read. Community members can also submit their own questions, and those go through the same rules and the same review before they are used.
If you play a round on QuiJutsu and a question feels off, report it. Every report I have acted on has made the bank a little better, and most of the rules in this post started as exactly that: a question that looked fine until someone who really knew the show pointed out it was not.