TrashCat: A Data Analyst Guide
This guide helps data analysts understand TrashCat's analytics data published via Caliper (for educational analytics) and MixPanel (for product analytics).
Analytics Systems Overview
TrashCat publishes data through two distinct analytics pipelines:
Caliper (via TimeBack)
Purpose: Standards-based learning analytics for educational analytics. Integration: Events are sent to the TimeBack Platform, which handles Caliper 1.2 event formatting and QuickSight data set generation.
Key Event Types:
- AssessmentItemEvent - Question display and completion
- GradeEvent - Question scoring and XP awards
- AssessmentEvent - Competition lifecycle (started, submitted)
- AssignableEvent - Skill and fact set lifecycle (started, completed)
Common Use Cases:
- Standards-based learning record tracking
- Skill and fact set progress tracking
- Competition session tracking
MixPanel
Purpose: Product analytics and user behavior tracking
Integration: Events are sent directly from both the backend (learning events) and frontend (UI/loading events) to MixPanel.
Key Event Types:
- Learning events (question answered, fact progression, rewards)
- Session events (competition completion, practice milestones)
- UI events (page loads, game loading, authentication)
Common Use Cases:
- Student engagement analysis
- Session duration and completion tracking
- Performance trend analysis
- Feature usage monitoring
Understanding XP (Experience Points)
XP awards time-on-task, with architectural controls to ensure quality practice.
XP is calculated as 1 XP per 60 seconds of active running time. "Active running time" means the cat is moving and the student is answering questions. Time spent in interventions (error corrections) does NOT count toward XP.
Implementation details:
XP_PER_TIME_UNIT = 1TIME_UNIT_SECONDS = 60- Formula:
totalXp = Math.floor(activeDurationSec / 60) * 1
Mode-Specific Caps:
Practice Mode:
- XP is awarded continuously during practice sessions
- Once a skill reaches 100% completion (all facts mastered), no more XP is awarded for that skill
- Students can still practice for maintenance, but
xpAwardedwill remain at 0 - This prevents XP farming on mastered content
Speedrun (Competition) Mode:
- XP is awarded only once per day per skill
- XP amount equals completion time in minutes (e.g., 7-minute run = 7 XP)
- Only completed runs award XP (aborted runs award 0 XP)
- Subsequent completions on the same day award 0 XP
Why students might not get XP despite long practice:
- Skill is already 100% complete (Practice mode)
- Already completed a Speedrun today for this skill (Competition mode)
- Session was aborted before completion (Competition mode only)
- Student is spending time in interventions (doesn't count as active running time)
Caliper Events Published
Event Schema Overview
All Caliper events follow the IMS Global Caliper Analytics 1.2 specification and include these base fields:
@context- Alwayshttp://purl.imsglobal.org/ctx/caliper/v1p2id- Unique event identifier (URN UUID format)type- Event type (AssessmentItemEvent, GradeEvent, AssessmentEvent, or AssignableEvent)actor- Student user ID as URN UUID string (e.g.,urn:uuid:student-platform-id)action- Action performed (Started, Completed, Skipped, Graded, Submitted)edApp- Application identifier string (e.g.,https://trashcat.learnwith.ai)session- Session identifier (currentlyurn:tag:auto-attach)eventTime- ISO 8601 timestamp when event occurred
Recent Schema Changes (November 2024):
-
Simplified actor and edApp fields - Changed from nested objects to simple string identifiers. This reduces redundancy since these identifiers are constant per user/application and can be resolved by the consumer.
- Old:
actor: { id: "urn:uuid:123", type: "Person" } - New:
actor: "urn:uuid:123"
- Old:
-
Removed profile field - The profile field was redundant with the type field and has been removed from all event schemas.
-
Added lifecycle tracking - New events track the complete lifecycle of skills, fact sets, and competitions (start/complete events) for comprehensive learning progress tracking.
-
Question isPartOf now distinguishes Practice vs Speedrun - Question events (AssessmentItemEvent) now use different
isPartOfparent types based on the learning mode:- Practice questions:
isPartOflinks to the Fact Set asAssignableDigitalResourcewithmediaType: curriculum/section - Speedrun questions:
isPartOflinks to the Speedrun Assessment asAssessmenttype - This allows TimeBack to filter/group questions by parent type to analyze Practice engagement vs Speedrun performance separately.
- Practice questions:
Available Actions:
Started- Resource/question was started or viewedCompleted- Question was answered or resource was completedSkipped- Question was skipped or timed outGraded- Question or XP was scoredSubmitted- Competition was submitted
Score Types:
QUESTION_RESULT- Individual question score (0 or 1)XP- Experience points awardedMASTERY- Fact set mastery achievement (100/100)
Event Types
1. Question Viewed (AssessmentItemEvent)
Action: Started
When Published: When a question is displayed to the student
Key Fields:
actor- Student user ID (URN UUID string)object.id- Question ID (URN UUID)object.type-AssessmentItemobject.name- Question text (e.g., "7 × 8")object.isPartOf- Parent resource (varies by mode, see below)object.dateToStartOn- Question display timestampobject.dateToSubmit- Expected answer deadline (if timed)object.maxAttempts- Always 1object.maxSubmits- Always 1object.maxScore- Always 1object.isTimeDependent- Whether question is timedobject.version- Always "1.0"eventTime- Event timestamp
isPartOf by Mode:
- Practice Mode:
object.isPartOf.id- Fact Set URL (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})object.isPartOf.type-AssignableDigitalResourceobject.isPartOf.mediaType-curriculum/sectionobject.isPartOf.name- Fact Set ID
- Speedrun Mode:
object.isPartOf.id- Skill URL (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})object.isPartOf.type-Assessmentobject.isPartOf.name- Skill name
2. Question Answered (AssessmentItemEvent)
Action: Completed (for answered) or Skipped (for timed out/skipped)
When Published: When student submits an answer or times out
Key Fields:
actor- Student user ID (URN UUID string)object- Same assessment item as viewed event (including mode-specificisPartOf, see Question Viewed above)generated.id- Response ID (URN UUID)generated.type-Responsegenerated.attempt.id- Attempt ID (URN UUID)generated.attempt.type-Attemptgenerated.attempt.assignee- Student user ID (URN UUID)generated.attempt.assignable- Assessment item ID (URN UUID)generated.attempt.count- Number of attempts (includes interventions)generated.attempt.dateCreated- Response creation timestampgenerated.attempt.startedAtTime- When question was displayedgenerated.attempt.endedAtTime- When answer was submittedgenerated.dateCreated- Response creation timestampgenerated.startedAtTime- When response startedgenerated.endedAtTime- When response endedgenerated.extensions.answerType-CORRECT,INCORRECT,TIMED_OUT, orSKIPPEDgenerated.extensions.choiceId- Selected answer choice IDgenerated.extensions.timeTookToAnswerMs- Response time in millisecondseventTime- Event timestamp
Note: Skipped/timed-out questions do not include generated fields.
3. Question Graded (GradeEvent)
Action: Graded
When Published: Immediately after a question is answered (not for skipped questions)
Key Fields:
actor- Student user ID (URN UUID string)object- Attempt ID from the answer event (URN UUID string)generated.id- Score ID (URN UUID)generated.type-Scoregenerated.scoreType-QUESTION_RESULTgenerated.attempt- Attempt ID reference (URN UUID)generated.maxScore- Always 1generated.scoreGiven- 1 for correct, 0 for incorrectgenerated.comment- Human-readable result (e.g., "Correct=56, User Answer=56, Result=CORRECT")generated.dateCreated- Grading timestampeventTime- Event timestamp
4. XP Awarded (GradeEvent)
Action: Graded
When Published: When XP is earned during practice or competition
Key Fields:
actor- Student user ID (URN UUID string)object.id- Attempt ID (URN UUID)object.type-Attemptobject.assignee- Student user ID (URN UUID)object.assignable.id- Course ID (URN UUID)object.assignable.type-AssignableDigitalResourceobject.assignable.mediaType-curriculum/courseobject.assignable.name- Course nameobject.assignable.isPartOf.id- Subject ID (URN UUID)object.assignable.isPartOf.type-AssignableDigitalResourceobject.assignable.isPartOf.mediaType-curriculum/subjectobject.assignable.isPartOf.name- Subject name (e.g., "Math")generated.id- Score ID (URN UUID)generated.type-Scoregenerated.scoreType-XPgenerated.attempt- Attempt ID reference (URN UUID)generated.scoreGiven- Amount of XP awardedgenerated.dateCreated- Award timestampeventTime- Event timestamp
5. Competition Started (AssessmentEvent)
Action: Started
When Published: When a student begins a competition (Speedrun) session
Key Fields:
actor- Student user ID (URN UUID string)object.id- Assessment ID (skill API URL, e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})object.type-Assessmentobject.name- Skill namegenerated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Assessment ID (skill API URL)generated.startedAtTime- Competition start timestamp (ISO 8601)eventTime- Event timestamp
6. Competition Submitted (AssessmentEvent)
Action: Submitted
When Published: When a student completes a competition (Speedrun) session
Key Fields:
actor- Student user ID (URN UUID string)object- Assessment ID as string (skill API URL, e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})generated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Assessment ID (skill API URL)generated.startedAtTime- Competition start timestamp (ISO 8601)generated.endedAtTime- Competition end timestamp (ISO 8601)eventTime- Event timestamp (same as endedAtTime)
Note: Competition aborted events do not generate Caliper events.
7. Skill Started (AssignableEvent)
Action: Started
When Published: When a student answers their first question in a skill (tracked across both practice and competition modes)
Key Fields:
actor- Student user ID (URN UUID string)object.id- Skill resource URL (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})object.type-AssignableDigitalResourceobject.mediaType-curriculum/courseobject.name- Skill nameobject.isPartOf.id- Parent subject ID (URN UUID)object.isPartOf.type-AssignableDigitalResourceobject.isPartOf.mediaType-curriculum/subjectobject.isPartOf.name- Subject name (e.g., "Math")generated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Skill resource URLgenerated.startedAtTime- First answer timestamp (ISO 8601)eventTime- Event timestamp
Note: This event is triggered once per student per skill, regardless of mode (practice or competition).
8. Skill Completed (AssignableEvent)
Action: Completed
When Published: When a student masters all facts in a skill (reaches 100% completion)
Key Fields:
actor- Student user ID (URN UUID string)object- Skill resource URL as string (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId})generated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Skill resource URLgenerated.startedAtTime- First answer timestamp (ISO 8601)generated.endedAtTime- Completion timestamp (ISO 8601)eventTime- Event timestamp (same as endedAtTime)
Note: Skill completion represents mastery of all fact sets within the skill.
9. Fact Set Started (AssignableEvent)
Action: Started
When Published: When a student answers their first question in a fact set
Key Fields:
actor- Student user ID (URN UUID string)object.id- Fact set resource URL (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})object.type-AssignableDigitalResourceobject.mediaType-curriculum/sectionobject.name- Fact set IDobject.isPartOf.id- Parent skill URLobject.isPartOf.type-AssignableDigitalResourceobject.isPartOf.mediaType-curriculum/courseobject.isPartOf.name- Skill namegenerated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Fact set resource URLgenerated.startedAtTime- First answer timestamp (ISO 8601)eventTime- Event timestamp
10. Fact Set Completed (AssignableEvent)
Action: Completed
When Published: When a student masters all facts in a fact set (all facts reach final mastery stage)
Key Fields:
actor- Student user ID (URN UUID string)object- Fact set resource URL as string (e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})generated.id- Attempt ID (URN UUID)generated.type-Attemptgenerated.assignee- Student user ID (URN UUID)generated.assignable- Fact set resource URLgenerated.startedAtTime- First answer timestamp (ISO 8601)generated.endedAtTime- Completion timestamp (ISO 8601)eventTime- Event timestamp (same as endedAtTime)
Note: Fact set completion represents mastery of all individual facts within that set.
11. Fact Set Mastery Graded (GradeEvent)
Action: Graded
When Published: Immediately after a fact set is completed (mastered), alongside the Fact Set Completed event
Key Fields:
actor- Student user ID (URN UUID string)object.id- Attempt ID (URN UUID)object.type-Attemptobject.assignee- Student user ID (URN UUID)object.assignable- Fact set resource URL (section, e.g.,https://api.trashcat.learnwith.ai/learning/v1/skills/{skillId}/factsets/{factSetId})object.startedAtTime- First answer timestamp (ISO 8601)object.endedAtTime- Completion timestamp (ISO 8601)generated.id- Score ID (URN UUID)generated.type-Scoregenerated.scoreType-MASTERYgenerated.attempt- Attempt ID reference (URN UUID)generated.maxScore- Always 100generated.scoreGiven- Always 100 (indicates full mastery)generated.dateCreated- Grading timestampeventTime- Event timestamp (same as endedAtTime)
Note: This event is always fired alongside the Fact Set Completed event. The object.assignable references the section (fact set) to indicate what was mastered.
Lifecycle Tracking Implementation
TrashCat tracks learning activity lifecycles across both practice and competition modes to provide comprehensive analytics:
Skill Lifecycle:
- Started: Triggered on the first question answered in a skill, whether in practice or competition mode. The system checks both practice and competition states to determine the earliest start time.
- Completed: Triggered when all fact sets within a skill reach mastery (100% completion). This represents full skill mastery.
Fact Set Lifecycle:
- Started: Triggered on the first question answered in a specific fact set. The start time is tracked per fact set.
- Completed: Triggered when all facts in a fact set reach the final mastery stage. This indicates complete mastery of that fact set.
Competition Lifecycle:
- Started: Triggered when a student begins a Speedrun (competition) session.
- Submitted: Triggered when a student successfully completes a Speedrun session. Aborted competitions do not generate a submitted event.
Key Implementation Details:
- Start times are calculated by looking at the earliest answer timestamp across both practice and competition states
- All lifecycle events include precise ISO 8601 timestamps for duration analysis
- The system ensures each "started" event is only sent once per student per entity (skill/factset/competition)
- Completion events include both start and end timestamps for accurate duration tracking
MixPanel Events Published
Backend Events
question_displayed
When Published: When a question is generated and shown
Key Properties:
user_id,skill_id,session_id,algorithm_idquestion_id,question_text,fact_id,fact_set_idstage_type/learning_mode- e.g., "introduction", "practice", "review"time_started,time_to_answerchoices,choice_counttimestamp
question_answered
When Published: When a student answers or times out on a question
Key Properties:
user_id,skill_id,session_id,algorithm_idquestion_id,question_text,fact_id,fact_set_idanswer_type- "correct", "incorrect", "timed_out", "skipped"answer_value,correct_answer_valuetime_started,time_ended,actual_time_to_answer_secondsintervention_type- e.g., "build_fact", "show_answer" (if applicable)stage_type/learning_modetimestamp
individual_fact_progression
When Published: When a single fact advances or regresses between stages
Key Properties:
user_id,skill_id,session_id,algorithm_idfact_id,fact_set_idfrom_stage,to_stage- Stage IDs (e.g., "intro_stage_1" → "practice_stage_1")answer_type- Trigger for progressionconsecutive_count- Number of consecutive correct answerstimestamp
bulk_promotion
When Published: When multiple facts in a fact set are promoted due to demonstrated mastery (e.g., from Speedrun placement)
Key Properties:
user_id,skill_id,session_id,algorithm_idfact_set_idpromoted_facts_count- Number of facts promotedconsecutive_correct_count- Trigger thresholdcoverage_percentage- Percentage of fact set answered correctlytimestamp
fact_set_review_ready
When Published: When all facts in a fact set complete their introduction stage and are ready for review
Key Properties:
user_id,skill_id,session_idfact_set_id,next_fact_set_idtotal_facts_count,total_answer_counttimestamp
fact_set_completion
When Published: When a fact set reaches full mastery (all facts in final stage)
Key Properties:
user_id,skill_id,session_idcompleted_fact_set_id,next_fact_set_idtotal_facts_count,total_answer_countstarted_at_time- When first question in fact set was answeredended_at_time- When fact set was completedtimestamp
fact_set_started
When Published: When a student answers their first question in a fact set
Key Properties:
user_id,skill_id,session_id,algorithm_idfact_set_idstarted_at_time- Timestamp of first answertimestamp
reward_event
When Published: When XP or cosmetic rewards are earned
Key Properties:
user_id,skill_id,session_id,algorithm_idreward_xp_amount- XP awarded (may be 0 if only cosmetic)reward_cosmetic- Cosmetic item ID or 0timestamp
competition_started
When Published: When a Speedrun (competition) session begins
Key Properties:
user_id,skill_id,session_id,algorithm_idstarted_at_time- Competition start timestamptimestamp
competition_completion
When Published: When a Speedrun is successfully completed
Key Properties:
user_id,skill_id,session_id,algorithm_idsuccessful- Always true for completed eventstime_spent_sec- Total elapsed time including penaltiesactive_duration_sec- Active running time (basis for XP)questions_answered,correct_count,incorrect_countxp_earned- XP awarded (0 if not first completion today)items_earned- Cosmetic items unlockedtotal_facts,total_mastered_facts- Skill-level statsstarted_at_time- Competition start timestampended_at_time- Competition end timestamptimestamp
competition_aborted
When Published: When a Speedrun is automatically stopped due to too many errors
Key Properties:
user_id,skill_id,session_id,algorithm_idreason- Abort reason (e.g., "too_many_errors")questions_answered- Questions attempted before abortincorrect_in_window- Number of errors in recent windowwindow_size- Size of error window checkedstarted_at_time- Competition start timestamptimestamp
skill_started
When Published: When a student answers their first question in a skill (across practice and competition modes)
Key Properties:
user_id,skill_id,session_id,algorithm_idstarted_at_time- Timestamp of first answertimestamp
skill_completed
When Published: When a student masters all facts in a skill (reaches 100% completion)
Key Properties:
user_id,skill_id,session_id,algorithm_idstarted_at_time- When first question was answeredended_at_time- When skill was completedtimestamp
Frontend Events
page_load
When Published: When the application loads
Key Properties:
user_id,organization_id,organization_nametimestamp
sign_up / sign_in / sign_out
When Published: During authentication flows
Key Properties:
user_id,organization_id,organization_nametimestamp
game_loading_started / game_loading_success / game_loading_failure
When Published: During Unity game initialization
Key Properties:
gameType- "unity"gamePath- Unity build pathloadingTimeMs- Time to load (success only)timestamp
unity_error
When Published: When Unity encounters a runtime error
Key Properties:
error- Error messagetimestamp
Custom Unity Events
Unity can send arbitrary MixPanel events via the bridge. Event names and properties are defined in the Unity codebase. These appear in MixPanel with:
- Standard user context (
user_id,organization_id, etc.) - Custom properties from Unity (converted from strings to appropriate types)
Session Data Structure
Session data is stored in DynamoDB and queryable via the backend API.
Key Fields:
userId,skillId,date(YYYY-MM-DD)sessionId- Unique session identifieralgorithmId- "practice" or "competition"createdAt- Session creation timestamp (ISO format)lastActiveDurationSec- Most recent active duration reportedlastXpAwardedDurationSec- Duration at which XP was last calculatedxpAwarded- Total XP earned in this sessionisCompleted- (Competition only) Whether run finishedcompletedAt- (Competition only) Completion timestamphasRecommendedSessionCompletion- Whether system recommended rest
Common Analysis Patterns
Calculating Accuracy
From MixPanel:
Accuracy = (COUNT WHERE answer_type = "correct") / (COUNT WHERE answer_type IN ["correct", "incorrect"])
Note: Exclude timed_out and skipped from denominator, or include them as errors depending on your analysis goal.
Verifying XP vs Time Spent
Expected XP:
expectedXp = Math.floor(lastActiveDurationSec / 60)
Actual XP:
actualXp = xpAwarded (from session data or reward_event)
If they don't match:
- Skill may be 100% complete (Practice mode)
- Daily Speedrun already completed today (Competition mode)
- Session was aborted (Competition mode)
Identifying Learning Patterns
Questions to ask:
- How many interventions per student per session?
- Average time to answer by stage type?
- Fact progression velocity (intro → practice → review → mastery)?
- Drop-off points (which fact sets cause students to quit)?
Key Events:
question_answered- Filter byintervention_type IS NOT NULLfor intervention frequencyindividual_fact_progression- Track stage transitionsfact_set_completion- Measure milestone achievement rate
Monitoring Engagement
Session Duration:
sessionDuration = lastActiveDurationSec (from session data)
Session Completion Rate:
completionRate = (COUNT WHERE isCompleted = true) / (COUNT sessions) (Competition only)
Daily Active Users:
- Count distinct
user_idper day with any session activity
Retention:
- Track users with sessions on consecutive days
Troubleshooting Common Data Questions
"Student practiced for 30 minutes but got 0 XP"
Check:
- Is skill at 100% completion? (Practice mode caps XP)
- Did they already complete a Speedrun today? (Competition mode daily cap)
- Is
lastActiveDurationSecactually progressing? (System may have paused for cooldown)
"XP doesn't match session duration"
Expected: XP is based on active running time, not total elapsed time or wall clock time. Interventions don't count. Also check for completion caps.
"Accuracy seems low but student is progressing"
Expected: The system adapts by showing easier facts or moving backward through stages. Low accuracy may indicate the algorithm is actively working to find the right difficulty level, which is normal during early practice.
"Student has many sessions on the same date"
Expected: Students can have multiple sessions per skill per day:
- Multiple practice sessions (ending due to lives, cooldown, or exit)
- One scored Speedrun + unlimited unscored attempts
- Sessions across different skills
Each session has a unique sessionId. Aggregate by date for daily activity summaries.
"Caliper events missing for some questions"
Check:
- Does user have a
platformId? (Required for Caliper, not for MixPanel) - Was the answer skipped/timed-out? (No graded event is sent for non-attempts)
- Check error logs for TimeBack API failures
Data Access
MixPanel Dashboard:
- Access product analytics and user behavior data
- Log in with your organization credentials
- Contact trashcat@trilogy.com if you need access
Caliper Events (via TimeBack):
- Events are forwarded through the TimeBack learning records platform to QuickSight
- Access data sets via AWS QuickSight
- Contact timeback@trilogy.com for QuickSight access or TimeBack platform credentials