User Guide

TrashCat: A Data Analyst Guide

Academic outcomes (per-game skill funnels): For field meanings, traps, and worked examples over GET /outcomes/v1/games/{gameId}, use the DSS skill at https://api.trashcat.learnwith.ai/skill (integration: https://api-integration.trashcat.rp.devfactory.com/skill). This guide covers Caliper and Mixpanel event streams only.

This guide helps data analysts understand TrashCat's analytics data published via Caliper (for educational analytics) and MixPanel (for product analytics).

Analytics Systems Overview

TrashCat publishes data through two distinct analytics pipelines:

Caliper (via TimeBack)

Purpose: Standards-based learning analytics for educational analytics. Integration: Events are sent to the TimeBack Platform, which handles Caliper 1.2 event formatting and QuickSight data set generation.

Key Event Types:

  • AssessmentItemEvent - Question display and completion
  • GradeEvent - Question scoring and XP awards
  • AssessmentEvent - Competition lifecycle (started, submitted)
  • AssignableEvent - Skill and fact set lifecycle (started, completed)

Common Use Cases:

  • Standards-based learning record tracking
  • Skill and fact set progress tracking
  • Competition session tracking

MixPanel

Purpose: Product analytics and user behavior tracking

Integration: Events are sent directly from both the backend (learning events) and frontend (UI/loading events) to MixPanel.

Key Event Types:

  • Learning events (question answered, fact progression, rewards)
  • Session events (competition completion, practice milestones)
  • UI events (page loads, game loading, authentication)

Common Use Cases:

  • Student engagement analysis
  • Session duration and completion tracking
  • Performance trend analysis
  • Feature usage monitoring

Understanding XP (Experience Points)

XP awards time-on-task, with architectural controls to ensure quality practice.

XP is calculated as 1 XP per 60 seconds of active running time. "Active running time" means the cat is moving and the student is answering questions. Time spent in interventions (error corrections) does NOT count toward XP.

Implementation details:

  • XP_PER_TIME_UNIT = 1
  • TIME_UNIT_SECONDS = 60
  • Formula: totalXp = Math.floor(activeDurationSec / 60) * 1

Mode-Specific Caps:

Practice Mode:

  • XP is awarded continuously during practice sessions at 1 XP per 60s of active running time (each minute must clear ≥80% accuracy)
  • Once a skill reaches 100% completion (all facts mastered), full-rate active-learning XP stops, but post-completion ("maintenance") XP accrues separately: 1 XP per 120s of active review time (≥80% accuracy per window), clamped to a daily cap of 10 and a weekly cap of 30, summed globally across all skills
  • Post-completion XP is tracked separately as postCompletionXpAwarded and flagged on reward events as is_post_completion_xp; active-learning xpAwarded for a completed skill stays at 0 while postCompletionXpAwarded accrues
  • This rewards continued review without allowing unlimited XP farming on mastered content (constants in backend/app/learning/constants/reward.constants.ts)

Speedrun (Competition) Mode:

  • XP is awarded only once per day per skill
  • XP amount equals completion time in minutes (e.g., 7-minute run = 7 XP)
  • Only completed runs award XP (aborted runs award 0 XP)
  • Subsequent completions on the same day award 0 XP

Why students might not get XP despite long practice:

  • On a completed skill, post-completion XP can still be 0 if the daily (10) or weekly (30) cap is already reached, accuracy is below 80% within the 120s windows, or all mastered facts are still on cooldown (Practice mode)
  • Already completed a Speedrun today for this skill (Competition mode)
  • Session was aborted before completion (Competition mode only)
  • Student is spending time in interventions (doesn't count as active running time)

Caliper Events Published

Event Schema Overview

All Caliper events follow the IMS Global Caliper Analytics 1.2 specification and include these base fields:

  • @context - Always http://purl.imsglobal.org/ctx/caliper/v1p2
  • id - Unique event identifier (URN UUID format)
  • type - Event type (AssessmentItemEvent, GradeEvent, AssessmentEvent, or AssignableEvent)
  • actor - Student user ID as URN UUID string (e.g., urn:uuid:student-platform-id)
  • action - Action performed (Started, Completed, Skipped, Graded, Submitted)
  • edApp - Application identifier string (e.g., https://trashcat.learnwith.ai)
  • session - Session identifier (currently urn:tag:auto-attach)
  • eventTime - ISO 8601 timestamp when event occurred

Recent Schema Changes (November 2024):

  1. Simplified actor and edApp fields - Changed from nested objects to simple string identifiers. This reduces redundancy since these identifiers are constant per user/application and can be resolved by the consumer.

    • Old: actor: { id: "urn:uuid:123", type: "Person" }
    • New: actor: "urn:uuid:123"

  2. Removed profile field - The profile field was redundant with the type field and has been removed from all event schemas.

  3. Added lifecycle tracking - New events track the complete lifecycle of skills, fact sets, and competitions (start/complete events) for comprehensive learning progress tracking.

  4. Question isPartOf now distinguishes Practice vs Speedrun - Question events (AssessmentItemEvent) now use different isPartOf parent types based on the learning mode:

    • Practice questions: isPartOf links to the Fact Set as AssignableDigitalResource with mediaType: curriculum/section
    • Speedrun questions: isPartOf links to the Speedrun Assessment as Assessment type
    • This allows TimeBack to filter/group questions by parent type to analyze Practice engagement vs Speedrun performance separately.

Available Actions:

  • Started - Resource/question was started or viewed
  • Completed - Question was answered or resource was completed
  • Skipped - Question was skipped or timed out
  • Graded - Question or XP was scored
  • Submitted - Competition was submitted

Score Types:

  • QUESTION_RESULT - Individual question score (0 or 1)
  • XP - Experience points awarded
  • MASTERY - Fact set mastery achievement (100/100)

Event Types

1. Question Viewed (AssessmentItemEvent)

Action: Started

When Published: When a question is displayed to the student

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Question ID (URN UUID)
  • object.type - AssessmentItem
  • object.name - Question text (e.g., "7 × 8")
  • object.isPartOf - Parent resource (varies by mode, see below)
  • object.dateToStartOn - Question display timestamp
  • object.dateToSubmit - Expected answer deadline (if timed)
  • object.maxAttempts - Always 1
  • object.maxSubmits - Always 1
  • object.maxScore - Always 1
  • object.isTimeDependent - Whether question is timed
  • object.version - Always "1.0"
  • eventTime - Event timestamp

isPartOf by Mode:

  • Practice Mode:
    • object.isPartOf.id - Fact Set URL (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})
    • object.isPartOf.type - AssignableDigitalResource
    • object.isPartOf.mediaType - curriculum/section
    • object.isPartOf.name - Fact Set ID
  • Speedrun Mode:
    • object.isPartOf.id - Skill URL (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})
    • object.isPartOf.type - Assessment
    • object.isPartOf.name - Skill name

2. Question Answered (AssessmentItemEvent)

Action: Completed (for answered) or Skipped (for timed out/skipped)

When Published: When student submits an answer or times out

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object - Same assessment item as viewed event (including mode-specific isPartOf, see Question Viewed above)
  • generated.id - Response ID (URN UUID)
  • generated.type - Response
  • generated.attempt.id - Attempt ID (URN UUID)
  • generated.attempt.type - Attempt
  • generated.attempt.assignee - Student user ID (URN UUID)
  • generated.attempt.assignable - Assessment item ID (URN UUID)
  • generated.attempt.count - Number of attempts (includes interventions)
  • generated.attempt.dateCreated - Response creation timestamp
  • generated.attempt.startedAtTime - When question was displayed
  • generated.attempt.endedAtTime - When answer was submitted
  • generated.dateCreated - Response creation timestamp
  • generated.startedAtTime - When response started
  • generated.endedAtTime - When response ended
  • generated.extensions.answerType - CORRECT, INCORRECT, TIMED_OUT, or SKIPPED
  • generated.extensions.choiceId - Selected answer choice ID
  • generated.extensions.timeTookToAnswerMs - Response time in milliseconds
  • eventTime - Event timestamp

Note: Skipped/timed-out questions do not include generated fields.

3. Question Graded (GradeEvent)

Action: Graded

When Published: Immediately after a question is answered (not for skipped questions)

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object - Attempt ID from the answer event (URN UUID string)
  • generated.id - Score ID (URN UUID)
  • generated.type - Score
  • generated.scoreType - QUESTION_RESULT
  • generated.attempt - Attempt ID reference (URN UUID)
  • generated.maxScore - Always 1
  • generated.scoreGiven - 1 for correct, 0 for incorrect
  • generated.comment - Human-readable result (e.g., "Correct=56, User Answer=56, Result=CORRECT")
  • generated.dateCreated - Grading timestamp
  • eventTime - Event timestamp

4. XP Awarded (GradeEvent)

Action: Graded

When Published: When XP is earned during practice or competition

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Attempt ID (URN UUID)
  • object.type - Attempt
  • object.assignee - Student user ID (URN UUID)
  • object.assignable.id - Course ID (URN UUID)
  • object.assignable.type - AssignableDigitalResource
  • object.assignable.mediaType - curriculum/course
  • object.assignable.name - Course name
  • object.assignable.isPartOf.id - Subject ID (URN UUID)
  • object.assignable.isPartOf.type - AssignableDigitalResource
  • object.assignable.isPartOf.mediaType - curriculum/subject
  • object.assignable.isPartOf.name - Subject name (e.g., "Math")
  • generated.id - Score ID (URN UUID)
  • generated.type - Score
  • generated.scoreType - XP
  • generated.attempt - Attempt ID reference (URN UUID)
  • generated.scoreGiven - Amount of XP awarded
  • generated.dateCreated - Award timestamp
  • eventTime - Event timestamp

5. Competition Started (AssessmentEvent)

Action: Started

When Published: When a student begins a competition (Speedrun) session

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Assessment ID (skill API URL, e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})
  • object.type - Assessment
  • object.name - Skill name
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Assessment ID (skill API URL)
  • generated.startedAtTime - Competition start timestamp (ISO 8601)
  • eventTime - Event timestamp

6. Competition Submitted (AssessmentEvent)

Action: Submitted

When Published: When a student completes a competition (Speedrun) session

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object - Assessment ID as string (skill API URL, e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Assessment ID (skill API URL)
  • generated.startedAtTime - Competition start timestamp (ISO 8601)
  • generated.endedAtTime - Competition end timestamp (ISO 8601)
  • eventTime - Event timestamp (same as endedAtTime)

Note: Competition aborted events do not generate Caliper events.

7. Skill Started (AssignableEvent)

Action: Started

When Published: When a student answers their first question in a skill (tracked across both practice and competition modes)

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Skill resource URL (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})
  • object.type - AssignableDigitalResource
  • object.mediaType - curriculum/course
  • object.name - Skill name
  • object.isPartOf.id - Parent subject ID (URN UUID)
  • object.isPartOf.type - AssignableDigitalResource
  • object.isPartOf.mediaType - curriculum/subject
  • object.isPartOf.name - Subject name (e.g., "Math")
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Skill resource URL
  • generated.startedAtTime - First answer timestamp (ISO 8601)
  • eventTime - Event timestamp

Note: This event is triggered once per student per skill, regardless of mode (practice or competition).

8. Skill Completed (AssignableEvent)

Action: Completed

When Published: When a student masters all facts in a skill (reaches 100% completion)

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object - Skill resource URL as string (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Skill resource URL
  • generated.startedAtTime - First answer timestamp (ISO 8601)
  • generated.endedAtTime - Completion timestamp (ISO 8601)
  • eventTime - Event timestamp (same as endedAtTime)

Note: Skill completion represents mastery of all fact sets within the skill.

9. Fact Set Started (AssignableEvent)

Action: Started

When Published: When a student answers their first question in a fact set

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Fact set resource URL (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})
  • object.type - AssignableDigitalResource
  • object.mediaType - curriculum/section
  • object.name - Fact set ID
  • object.isPartOf.id - Parent skill URL
  • object.isPartOf.type - AssignableDigitalResource
  • object.isPartOf.mediaType - curriculum/course
  • object.isPartOf.name - Skill name
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Fact set resource URL
  • generated.startedAtTime - First answer timestamp (ISO 8601)
  • eventTime - Event timestamp

10. Fact Set Completed (AssignableEvent)

Action: Completed

When Published: When a student masters all facts in a fact set (all facts reach final mastery stage)

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object - Fact set resource URL as string (e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})
  • generated.id - Attempt ID (URN UUID)
  • generated.type - Attempt
  • generated.assignee - Student user ID (URN UUID)
  • generated.assignable - Fact set resource URL
  • generated.startedAtTime - First answer timestamp (ISO 8601)
  • generated.endedAtTime - Completion timestamp (ISO 8601)
  • eventTime - Event timestamp (same as endedAtTime)

Note: Fact set completion represents mastery of all individual facts within that set.

11. Fact Set Mastery Graded (GradeEvent)

Action: Graded

When Published: Immediately after a fact set is completed (mastered), alongside the Fact Set Completed event

Key Fields:

  • actor - Student user ID (URN UUID string)
  • object.id - Attempt ID (URN UUID)
  • object.type - Attempt
  • object.assignee - Student user ID (URN UUID)
  • object.assignable - Fact set resource URL (section, e.g., https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})
  • object.startedAtTime - First answer timestamp (ISO 8601)
  • object.endedAtTime - Completion timestamp (ISO 8601)
  • generated.id - Score ID (URN UUID)
  • generated.type - Score
  • generated.scoreType - MASTERY
  • generated.attempt - Attempt ID reference (URN UUID)
  • generated.maxScore - Always 100
  • generated.scoreGiven - Always 100 (indicates full mastery)
  • generated.dateCreated - Grading timestamp
  • eventTime - Event timestamp (same as endedAtTime)

Note: This event is always fired alongside the Fact Set Completed event. The object.assignable references the section (fact set) to indicate what was mastered.

Lifecycle Tracking Implementation

TrashCat tracks learning activity lifecycles across both practice and competition modes to provide comprehensive analytics:

Skill Lifecycle:

  • Started: Triggered on the first question answered in a skill, whether in practice or competition mode. The system checks both practice and competition states to determine the earliest start time.
  • Completed: Triggered when all fact sets within a skill reach mastery (100% completion). This represents full skill mastery.

Fact Set Lifecycle:

  • Started: Triggered on the first question answered in a specific fact set. The start time is tracked per fact set.
  • Completed: Triggered when all facts in a fact set reach the final mastery stage. This indicates complete mastery of that fact set.

Competition Lifecycle:

  • Started: Triggered when a student begins a Speedrun (competition) session.
  • Submitted: Triggered when a student successfully completes a Speedrun session. Aborted competitions do not generate a submitted event.

Key Implementation Details:

  • Start times are calculated by looking at the earliest answer timestamp across both practice and competition states
  • All lifecycle events include precise ISO 8601 timestamps for duration analysis
  • The system ensures each "started" event is only sent once per student per entity (skill/factset/competition)
  • Completion events include both start and end timestamps for accurate duration tracking

MixPanel Events Published

Backend Events

question_displayed

When Published: When a question is generated and shown

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • question_id, question_text, fact_id, fact_set_id
  • stage_type / learning_mode - e.g., "introduction", "practice", "review"
  • time_started, time_to_answer
  • choices, choice_count
  • timestamp

question_answered

When Published: When a student answers or times out on a question

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • question_id, question_text, fact_id, fact_set_id
  • answer_type - "correct", "incorrect", "timed_out", "skipped"
  • answer_value, correct_answer_value
  • time_started, time_ended, actual_time_to_answer_seconds
  • intervention_type - e.g., "build_fact", "show_answer" (if applicable)
  • stage_type / learning_mode
  • timestamp

individual_fact_progression

When Published: When a single fact advances or regresses between stages

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • fact_id, fact_set_id
  • from_stage, to_stage - Stage IDs (e.g., "intro_stage_1" → "practice_stage_1")
  • answer_type - Trigger for progression
  • consecutive_count - Number of consecutive correct answers
  • timestamp

bulk_promotion

When Published: When multiple facts in a fact set are promoted due to demonstrated mastery (e.g., from Speedrun placement)

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • fact_set_id
  • promoted_facts_count - Number of facts promoted
  • consecutive_correct_count - Trigger threshold
  • coverage_percentage - Percentage of fact set answered correctly
  • timestamp

fact_set_review_ready

When Published: When all facts in a fact set complete their introduction stage and are ready for review

Key Properties:

  • user_id, skill_id, session_id
  • fact_set_id, next_fact_set_id
  • total_facts_count, total_answer_count
  • timestamp

fact_set_completion

When Published: When a fact set reaches full mastery (all facts in final stage)

Key Properties:

  • user_id, skill_id, session_id
  • completed_fact_set_id, next_fact_set_id
  • total_facts_count, total_answer_count
  • started_at_time - When first question in fact set was answered
  • ended_at_time - When fact set was completed
  • timestamp

fact_set_started

When Published: When a student answers their first question in a fact set

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • fact_set_id
  • started_at_time - Timestamp of first answer
  • timestamp

review_started

When Published: When a fact set review session begins

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • fact_set_id
  • started_at_time
  • timestamp

review_completed

When Published: When a fact set review session completes

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • fact_set_id, passed
  • total_facts, correct_fast, correct_slow, incorrect, skipped
  • active_duration_sec, started_at_time, ended_at_time
  • timestamp

reward_event

When Published: When XP or cosmetic rewards are earned

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • reward_xp_amount - XP awarded (may be 0 if only cosmetic)
  • reward_cosmetic - Cosmetic item ID or 0
  • timestamp

competition_started

When Published: When a Speedrun (competition) session begins

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • started_at_time - Competition start timestamp
  • timestamp

competition_completion

When Published: When a Speedrun is successfully completed

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • successful - Always true for completed events
  • time_spent_sec - Total elapsed time including penalties
  • active_duration_sec - Active running time (basis for XP)
  • questions_answered, correct_count, incorrect_count
  • xp_earned - XP awarded (0 if not first completion today)
  • items_earned - Cosmetic items unlocked
  • total_facts, total_mastered_facts - Skill-level stats
  • started_at_time - Competition start timestamp
  • ended_at_time - Competition end timestamp
  • timestamp

competition_aborted

When Published: When a Speedrun is automatically stopped due to too many errors

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • reason - Abort reason (e.g., "too_many_errors")
  • questions_answered - Questions attempted before abort
  • incorrect_in_window - Number of errors in recent window
  • window_size - Size of error window checked
  • started_at_time - Competition start timestamp
  • timestamp

skill_started

When Published: When a student answers their first question in a skill (across practice and competition modes)

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • started_at_time - Timestamp of first answer
  • timestamp

skill_completed

When Published: When a student masters all facts in a skill (reaches 100% completion)

Key Properties:

  • user_id, skill_id, session_id, algorithm_id
  • started_at_time - When first question was answered
  • ended_at_time - When skill was completed
  • timestamp

Frontend Events

page_load

When Published: When the application loads

Key Properties:

  • user_id, organization_id, organization_name
  • timestamp

sign_up / sign_in / sign_out

When Published: During authentication flows

Key Properties:

  • user_id, organization_id, organization_name
  • timestamp

The server sends sign_up (mp_lib = node), once per new account, in the request that makes the account (One Tap and Google's button) or in the page's first request after it holds tokens (Google redirect and email). The page never sends it. method is google, google_one_tap, google_button, google_automatic or email. The event carries user_id, $user_id, organization_id, organization_name, environment, is_guest: false, the page's $device_id (so it joins the visitor's earlier landing, ad click and device) and the page's own context keys (app_version, landing_audience, landing_hook and so on). is_automated_traffic: true is present only for our automated checks. A guest who converts keeps the guest's $user_id. Properties the browser SDK computes per event ($browser, $os, $city) are not on sign_up; read them from the same person's page events.

An email sign-up's sign_up (method: email) also carries surface and email_domain; see Email sign-up steps below.

sign_in_failed

When Published: An in-page Google sign-in (One Tap or Google's button) was refused and the player saw the "did not finish" message.

Key Properties: method (google_one_tap, google_button, google_automatic), surface, reason (token_rejected, sign_in_disabled, account_link_unavailable, request_failed, session_refused) and error_name. A refused Google redirect is sign_in_redirect_failed.

Email sign-up steps

When Published: At every step of an email sign-up, from every place the email sign-up form or its code step appears: the sign-up page, the sign-in panel's sign-up form, the guest sign-up asks (the sign-up wall, the results screen, the "Save progress" window, Trash Cat's Team Frenzy card), and the sign-in form's code step, where an account that stopped at the code step finishes on a later visit.

Event Published when
sign_up_form_shown The email sign-up form appears, once per showing. "Use a different email" returns to the form without a new sign_up_form_shown.
sign_up_submitted The sign-up page or the sign-in panel's form passed its own checks and sends the sign-up.
guest_conversion_started The same moment on a guest ask (method: email_password).
sign_up_code_sent A code is on its way. trigger: sign_up (the sign-up asked for a code), resend ("Send a new code"), sign_in (an unconfirmed account signed in and a code was sent).
sign_up_code_submitted A code was entered.
sign_up_email_changed "Use a different email" left the code step; email_domain is the replaced address's.
sign_up_confirmed Cognito accepted the code.
sign_up_failed A refusal or error at any step; step and reason say which.
sign_in The sign-up page and the sign-in panel sign the new account in (unchanged).
guest_conversion_completed A guest ask signed the new account in and kept the guest's progress.
sign_up The account is made (method: email), once per account.

Key Properties:

  • surface - Where it happened: sign_up_page, sign_in_dialog, sign_in_page, or a guest ask's source (sign_up_gate, results_screen, conversion_dialog, or a game's own value such as team_frenzy_card). On every new event and on sign_up. guest_conversion_started and guest_conversion_completed carry the same value as source. sign_in carries neither.
  • email_domain - The address's domain, lowercased, never the address. "null" when the typed address has no @. On every event above except sign_up_form_shown and sign_in.
  • step and reason (sign_up_failed):
    • submit: invalid_name, invalid_email, invalid_password (the form's first refused field), account_exists, sign_up_disabled (the operator switch is off), rate_limited (Cognito's daily email cap), request_failed
    • confirm: code_mismatch, code_expired, rate_limited, request_failed
    • resend: rate_limited, request_failed
    • finish: sign_in_failed (the sign-in after confirmation was refused), request_failed (a later request failed, such as a guest's conversion)
  • error_name (sign_up_failed) - The thrown error's name, such as ResponseError or CodeMismatchException. Absent for a refusal the form makes before sending anything. A refused sign-in after confirmation on the sign-up page, the sign-in panel and the sign-in form reports auth:sign-in, the name the sign-in step gives its error report.

Funnel: in the production project, filter environment = production and is_automated_traffic not set, with a one-day conversion window:

  1. sign_up_form_shown
  2. guest_conversion_started with method = email_password, or sign_up_submitted
  3. sign_up_code_sent
  4. sign_up_code_submitted
  5. sign_up_confirmed
  6. sign_up with method = email

Break down by surface at step 1, by email_domain, and by device read from $mp_raw_user_agent ($browser names an iPhone app's built-in browser "Mobile Safari"). sign_up_failed by step and reason says why people stopped. Steps repeat: a wrong code, a resend or a VERIFY pressed again after a failed finish sends sign_up_code_submitted or sign_up_code_sent again, so count unique people per step.

History: until #1951, the page sent sign_up at the boot after sign-in, through the batched queue, so a player who closed the page first was never recorded: from 21 to 25 October, 8 of 32 One Tap and Google-button accounts have no sign_up. Count those accounts from the Cognito pool. Before #1951 an account confirmed later through the sign-in form's code step reported method: google; now it reports email.

History: before the release of #1924, email sign-up published only guest_conversion_started, guest_conversion_completed and sign_up, and none of them carried email_domain. An account confirmed later through the sign-in form's code step published sign_up at its next page load with method: google.

game_picker_viewed

When Published: When the game picker is shown, in frontends that offer more than one game. Deep links (?gameId=) and single-game frontends never show the picker, so they never publish it. Returning to the picker from a game publishes it again.

Key Properties:

  • game_ids - The enabled game ids in picker order
  • last_played_game_id - The game highlighted as last played, or "null" for a first visit
  • game_id, game_type - "null" on this event, since no game is active while the picker shows
  • timestamp

game_loading_started

When Published: When a game becomes the active one, once per activation, whichever way it was chosen. Filter on activation_source = game_picker for "which game did the player click". The funnel game_loading_started → game_loading_success measures boot completion per game.

Key Properties:

  • game_id - The activated game
  • activation_source - How the game was chosen: game_picker (a tile click), single_game_auto (a single-game frontend activating its only game), or deep_link (a ?gameId= URL, activated once the signed-in home screen opens, so a visitor who first has to sign in is counted after signing in)
  • timestamp

game_loading_success / game_loading_failure

When Published: When the web-game host reports that the active game finished booting, or that its boot failed.

Key Properties:

  • game_id, game_type - The booted game (context on every event once a game is active)
  • loading_time_ms - Milliseconds from activation to the report
  • error_message - The failure message (game_loading_failure only)
  • timestamp

History: the Unity host published these three events until 2026-09-01. No boot event of any name exists for 2026-09-02 to 2026-09-09. From 2026-09-10 until the release of #1537 the web host published the same moments as game_activated, game_boot_succeeded, and game_boot_failed; a Lexicon merge folds those rows into the series above. Rows before 2026-09-01 carry the boot duration as loading_time_ms, no activation_source, and Trashcat only (Unity booted a single game); rows from 2026-09-10 to the release carry activation_source on the activation event and the duration as boot_duration_ms, and later rows as loading_time_ms again.

game_loading_left

When Published: When a guest hides the page (switches tab or app) or closes it after their guest account exists (guest_session_started) and before the game reports game_loading_success or game_loading_failure. Each hide publishes one event, a hide followed by a close publishes one, and a return to the page lets the next hide publish again. A guest who switched away while their guest account was being made publishes one as the account is made, and a close before they return adds nothing. Signed-in players' game loads never publish it. It is sent by beacon, so it survives the page closing.

Key Properties:

  • time_since_play_ms - Milliseconds from the guest's PLAY press to the hide or close. A tap made before the landing finished loading is timed from the tap itself. For a guest already away when their account was made, it counts to that moment, so they left no later than it; on an audience landing their landing_left (pressed: true) carries the moment they left.
  • loading_progress_percent - The whole percent the loading screen showed, 0 to 100. "null" when the guest left on the "LOADING..." screen that shows before the game's own loading screen; game_id is also "null" then.
  • game_id, game_type - The loading game once it is active (context on every event)
  • timestamp

Reading it: a guest whose game never loaded is a leaver when their last game_loading_left has no later game_loading_success: "left after time_since_play_ms / 1000 seconds at loading_progress_percent%". A guest with no game_loading_left and no load is a hang, a crash or a lost beacon. A game_loading_left followed by game_loading_success is a guest who came back. The event carries no network type: take it from the guest's landing_rendered.connection_type.

Custom Game Events

Games can send MixPanel events via runtime.analytics. Event names and properties are defined in the game/frontend codebase. These appear in MixPanel with:

  • Standard user context (user_id, organization_id, etc.)
  • Custom properties from the game (converted from strings to appropriate types)

Session Data Structure

Session data is stored in DynamoDB and queryable via the backend API.

Key Fields:

  • userId, skillId, date (YYYY-MM-DD)
  • sessionId - Unique session identifier
  • algorithmId - "practice", "competition", or "review"
  • createdAt - Session creation timestamp (ISO format)
  • lastActiveDurationSec - Most recent active duration reported
  • lastXpAwardedDurationSec - Duration at which XP was last calculated
  • xpAwarded - Total XP earned in this session
  • isCompleted - (Competition only) Whether run finished
  • completedAt - (Competition only) Completion timestamp
  • hasRecommendedSessionCompletion - Whether system recommended rest

Common Analysis Patterns

Calculating Accuracy

From MixPanel:

Accuracy = (COUNT WHERE answer_type = "correct") / (COUNT WHERE answer_type IN ["correct", "incorrect"])

Note: Exclude timed_out and skipped from denominator, or include them as errors depending on your analysis goal.

Verifying XP vs Time Spent

Expected XP:

// Active skill (not yet 100% complete):
expectedXp = Math.floor(lastActiveDurationSec / 60)

// Completed skill (post-completion review): half rate, accuracy-gated,
// clamped to the remaining daily (10) and weekly (30) budget
expectedPostCompletionXp = min(
  Math.floor(activeReviewDurationSec / 120),
  remainingDailyBudget,
  remainingWeeklyBudget,
)

Actual XP:

actualXp = xpAwarded (from session data or reward_event)

If they don't match:

  • Skill may be 100% complete: active-learning XP stops, but post-completion review XP (tracked as postCompletionXpAwarded) uses the 120s rate and the daily/weekly caps (Practice mode)
  • Daily Speedrun already completed today (Competition mode)
  • Session was aborted (Competition mode)

Identifying Learning Patterns

Questions to ask:

  • How many interventions per student per session?
  • Average time to answer by stage type?
  • Fact progression velocity (intro → practice → review → mastery)?
  • Drop-off points (which fact sets cause students to quit)?

Key Events:

  • question_answered - Filter by intervention_type IS NOT NULL for intervention frequency
  • individual_fact_progression - Track stage transitions
  • fact_set_completion - Measure milestone achievement rate

Monitoring Engagement

Session Duration:

sessionDuration = lastActiveDurationSec (from session data)

Session Completion Rate:

completionRate = (COUNT WHERE isCompleted = true) / (COUNT sessions) (Competition only)

Daily Active Users:

  • Count distinct user_id per day with any session activity

Retention:

  • Track users with sessions on consecutive days

Troubleshooting Common Data Questions

"Student practiced for 30 minutes but got 0 XP"

Check:

  1. Is skill at 100% completion? Active-learning XP stops; check postCompletionXpAwarded for review XP, which uses the 120s rate and is capped at 10/day and 30/week (Practice mode)
  2. Did they already complete a Speedrun today? (Competition mode daily cap)
  3. Is lastActiveDurationSec actually progressing? (System may have paused for cooldown)

"XP doesn't match session duration"

Expected: XP is based on active running time, not total elapsed time or wall clock time. Interventions don't count. Also check for completion caps.

"Accuracy seems low but student is progressing"

Expected: The system adapts by showing easier facts or moving backward through stages. Low accuracy may indicate the algorithm is actively working to find the right difficulty level, which is normal during early practice.

"Student has many sessions on the same date"

Expected: Students can have multiple sessions per skill per day:

  • Multiple practice sessions (ending due to lives, cooldown, or exit)
  • One scored Speedrun + unlimited unscored attempts
  • Sessions across different skills

Each session has a unique sessionId. Aggregate by date for daily activity summaries.

"Caliper events missing for some questions"

Check:

  1. Does user have a platformId? (Required for Caliper, not for MixPanel)
  2. Was the answer skipped/timed-out? (No graded event is sent for non-attempts)
  3. Check error logs for TimeBack API failures

Data Access

MixPanel Dashboard:

  • Access product analytics and user behavior data
  • Log in with your organization credentials
  • Contact trashcat@trilogy.com if you need access

Caliper Events (via TimeBack):

  • Events are forwarded through the TimeBack learning records platform to QuickSight
  • Access data sets via AWS QuickSight
  • Contact timeback@trilogy.com for QuickSight access or TimeBack platform credentials
Previous
Teacher/Parent Guide