PLAYTEST.AI LABS RESEARCH PREVIEW

Find where generated worlds break.

Human red-teamers actively explore interactive world models and try to trigger failures in memory, physics, geometry, consistency, and action adherence. When something breaks, Playtest.ai captures the interaction leading up to it—turning an interesting failure into a structured, replayable evaluation case.

Turn human-discovered failures into structured, replayable evaluation cases.

Join the design partner program Talk to us about world-model evaluation

Exploratory research direction · designed with evaluation partners

HUMAN RED-TEAM TRACE

case_preview_001

GENERATED WORLD · LIVE00:51.240

00:42OBSERVE

Red chair beside window

00:45ACTION

Enter kitchen

00:51RETURN

Chair is blue and relocated

Human interactionObservable failureStructured evaluation case

THE INTERACTION GAP

World models need more than
passive video evaluation.

Interactive generated worlds can fail in ways a static benchmark never provokes. The evaluator has to move, return, interrupt, compare, and do something the expected path did not anticipate.

Object disappears after leaving view.

MEMORY

Room geometry changes after camera movement.

GEOMETRY

Requested action is ignored.

ADHERENCE

Character forgets a previous interaction.

IDENTITY

Physics changes across frames.

PHYSICS

These failures often emerge only when a human actively explores the world and does something unexpected.

HUMAN RED-TEAMING

Humans try to
break the world.

A compact workflow turns an exploratory interaction into evidence a research team can inspect, compare, and—where the system allows—replay.

Explore

Give human evaluators access to an interactive generated world. Their objective is simple: find something that breaks.

Capture

When a failure is found, preserve the surrounding observations, actions, model/version, seed, prompt/configuration, timestamps, available state, evaluator annotation, and failure category.

Turn failures into evals

Group recurring failures and create replayable cases for evaluation, regression testing, benchmark construction, failure analysis, model comparison, or future training workflows.

FAILURE TAXONOMY

What can we
evaluate?

The taxonomy is a starting point for study design—not a claim that every failure can be detected automatically or reproduced in every model.

Memory & object permanence

Does the world remember objects, characters, locations, and previous events after they leave the current view?

Geometry & spatial consistency

Do rooms, paths, objects, and environments remain spatially coherent as the user moves?

Physics consistency

Do objects continue to behave according to consistent physical rules?

Action adherence

Does the generated world respond correctly and consistently to the action the user attempted?

Identity & attribute consistency

Do characters and objects retain their appearance, properties, and identity over time?

Temporal consistency

Does world state evolve coherently rather than unexpectedly resetting or contradicting its history?

Humans are particularly valuable for discovering failure modes a predefined benchmark did not know to test.

HUMAN + MACHINE

Human exploration
+ automatic checks.

The goal is not to replace human evaluation with another model. Lightweight automatic checks can surface likely inconsistencies; humans determine whether they are meaningful failures.

AUTOMATIC CHECK

Object permanence
Action adherence
State persistence
Visual identity consistency

CANDIDATE FAILURE

A change crosses the agreed signal or consistency threshold.

HUMAN VALIDATION

An evaluator decides whether the behavior is meaningful and records why.

REPLAYABLE FAILURE

Every failure should
be reproducible.

A useful evaluation case keeps the context around the failure linked. Replay depends on what the model and environment expose; the record remains valuable even when exact replay is unavailable.

ILLUSTRATIVE EVALUATION RECORDWM-EVAL-001

REPRODUCTION · 7 / 10 REPLAY ATTEMPTS

FAILURE

Object permanence

Evaluator walks past a red chair, enters the kitchen, then returns.

OBSERVED FAILUREChair is now blue and positioned on the opposite side of the room.

Model: world-model-v17
World seed: 8240191
Interaction: 00:42–00:51
Prompt: A suburban house with a kitchen and living room

EVIDENCEframesactionstimestampsseedpromptmodel version

EVALUATION AS INFRASTRUCTURE

Evaluation becomes
a data engine.

The same workflow that discovers model failures can generate structured datasets containing observation → human action → generated outcome, plus seed, prompt, state, failure label, and human verification where available.

OBSERVATIONO_t

HUMAN ACTIONA_t

GENERATED OUTCOMEO_t+1

The product comes first. The dataset is a consequence of doing useful evaluation work.

BUILT FOR INTERACTIVE SYSTEMS

One evidence model.
Two interactive systems.

Playtest.ai already treats game playtesting as an evidence problem. World-model evaluation has the same underlying structure.

GAME PLAYTESTINGinteractionfailureevidencehuman validationrecurring pattern

WORLD-MODEL EVALUATIONinteractioninconsistencyevidencehuman validationrecurring failure

DESIGN PARTNER PROGRAM · RESEARCH PREVIEW

Help us define the evaluation layer for interactive world models.

We’re speaking with teams building interactive world models, generative environments, game foundation models, embodied agents, simulation models, and video/world generation systems. We want to understand which failures are hardest to discover, reproduce, and turn into useful evaluation data.

Become a design partner Talk to us

Playtest.ai Labs is an exploratory product area. Capabilities described here represent the direction we are validating with research partners.