#Prove a skill works using evaluation

#Prerequisites

• Authentication, either a logged-in session, confirmed with tessl whoami and started with tessl login, or a TESSL_TOKEN API key in the environment.
• Workspace access at Publisher level or above.
• A Tessl project, so Tessl knows which repository an evaluation belongs to and your results stay tied to your codebase over time. Create one with tessl project create <name> --workspace engteam, link an existing one with tessl project link, and repair a broken link with tessl project repair. If you belong to more than one Organization, add --org <name> or set TESSL_ORG; a non-interactive shell cannot prompt you to pick one. See Projects overview.

By default, an evaluation runs an agent on real tasks twice, once without the skill and once with it, then scores the difference. You can also declare your own variants to compare several configurations on the same tasks. That difference is the skill's value: proof it changes what the agent does, not just that it reads well. Where a review checks the skill itself, an evaluation measures its effect.

Evaluation is more rigorous than review and takes longer to set up and run. Reach for it when you need confidence a skill works, for example before you publish it or share it into a repository where you will not see how others' agents use it.

You can evaluate a single skill against generated scenarios, or evaluate context files against real tasks taken from your codebase's commit history. Both run through the same tessl eval commands.

#Skills are evaluated inside Tessl plugins

A Tessl plugin is a structure that bundles the skills, rules, and documentation an agent needs for an area of context, packaged so it can be versioned and shared. An evaluation generates scenarios and produces results, and a plugin is the natural place to keep them with the skill they belong to.

For this reason tessl eval run expects the generated scenarios to live inside the plugin, and the shorthand tessl eval run ./my-plugin expects the plugin root: the directory that contains .tessl-plugin/plugin.json. The examples on this page use ./my-plugin for that directory.

A plugin has one evals/ directory, at the plugin root. Do not put an evals/ directory under each skill, even when the plugin has several skills. See Plugin layout for a tree.

If you have a single standalone skill, import it into a plugin first:

tessl skill import ./my-skill

skill import makes the skill folder itself the plugin root. If your repository has several skills, do not run skill import for each one. Create one plugin at the repository root instead. See Import an existing skill.

#Generate scenarios

An eval needs scenarios. Generate them first.

Scenarios are the tasks an agent is assessed against. Generate them from the plugin:

tessl scenario generate ./my-plugin

Each generated scenario is put through a feasibility check on the server. Only scenarios that pass are saved, so the generator gives you tests that actually exercise the skill rather than filler.

--count sets how many scenarios to target, and defaults to 3. Pass a higher number to cover more of what the skill claims to do. For a narrowly scoped skill the generator may return fewer than requested, because it will not save scenarios that fail the feasibility check.

tessl scenario generate ./my-plugin --count 5

Generation runs server-side. Check progress with tessl scenario list --mine, then download the scenarios once it completes:

cd my-plugin
tessl scenario download --last

Run the download from inside the plugin directory (or any subdirectory of it). tessl scenario download then writes the scenarios into that plugin's evals/ directory, so tessl eval run picks them up. The download does not know which plugin the scenarios were generated from: it decides by your current directory alone. Outside a plugin, for example from the repository root, it writes to ./evals, and tessl eval run ./my-plugin then fails with "No valid scenarios found". To download from somewhere else, pass --output ./my-plugin/evals.

--last downloads your most recent generation, from any plugin or directory. If you started another generation since, pass the generation ID that tessl scenario generate printed instead: tessl scenario download <generation-id>. Override the location with --output, and choose whether new scenarios merge with or replace existing ones with --strategy merge (the default) or --strategy replace. The command always prints where the files landed.

The scenarios are saved with the skill, so anyone who changes it later can re-run the same scenarios and confirm it still behaves as expected.

You do not have to generate scenarios. You or your agent can write them by hand instead, as long as they sit in the plugin's evals/ directory, one scenario per subdirectory. See File formats for the structure.

#Generate scenarios from your codebase

Instead of generating scenarios from a skill, you can build them from real changes in your repository, so the tasks reflect work your team actually does. Point tessl scenario generate at a repository with commit hashes or PR numbers:

# From commits
tessl scenario generate org/repo --commits abc123,def456 --workspace engteam

# From pull requests
tessl scenario generate org/repo --prs 42,107 --workspace engteam

Tessl analyses each diff and reconstructs the intent of the change as a task brief and a scoring rubric. The starting state has your context files stripped out, so the agent solves the task without them.

--context tells Tessl which files are context files. The patterns are saved in each scenario as the codebase fixture's exclude list. When omitted, Tessl excludes *.mdc, *.md, tile.json, .tessl-plugin/plugin.json, tessl.json, and .tessl/ by default.

To browse recent commits and choose which to turn into scenarios, use tessl repo select-commits org/repo. It accepts --keyword, --author, --since / --until, and --count, and prints a table of commit hashes to pass to generate.

Because these scenarios start from a state with your context files stripped out, pass --context again at run time with a glob matching those files. This adds a with-context variant on top of the baseline so you can measure the delta:

tessl eval run ./evals/ --context 'src/**/*.knowledge.md'

Match the --context glob to the files each scenario excludes. With no --context, only the baseline runs.

#Write commit-fixture scenarios by hand

When you write a scenario with a commit fixture yourself, keep three things in mind:

• Set include globs that cover only the paths the task needs, especially in large repositories and monorepos. Without include, each commit is packed as the whole repository tree, which can make archives of hundreds of megabytes. exclude does not shrink the upload.
• Use as few distinct commits as you can, and write each one as the same ref string everywhere. Each distinct ref value is a separate archive, so a full SHA and an abbreviated SHA of the same commit upload twice. One fixture without include, or with an empty one, makes that ref's archive the whole tree.
• Do not depend on git history. The fixture is installed as a fresh repository with one commit, and no branches or tags from the source repository. Do not write tasks or rubric checks that rely on git log, git blame, or earlier commits. If a scenario needs history, build it in a setup script, or put the relevant facts in task.md.

See the configuration reference for the fixture fields.

#Run an evaluation

Point tessl eval run at the plugin root:

tessl eval run ./my-plugin

This is shorthand for tessl eval run ./my-plugin/evals --context ./my-plugin: the plugin's evals/ are the scenarios, and the plugin is the context.

By default, the agent solves each scenario twice, without the skill and with it, and a judge scores both against a per-scenario rubric. A run with explicit variants solves the same scenarios for each configuration instead. The run reports each score and any applicable difference. Runs can take a while; pressing Ctrl+C detaches and the run continues server-side.

Label a run to find it again later:

tessl eval run ./my-plugin --label "baseline"

# List your recent runs
tessl eval list --mine

# View a run, including the per-scenario breakdown
tessl eval view --last
tessl eval view abc123

# Retry a failed run
tessl eval retry abc123

#Read the result

A run produces output like this:

Eval Results: baseline

Agent:     claude:deepseek-v4.1-flash
Variants:  Without this plugin, With this plugin
Status:    Completed

Scenario: Commit message formatting for complex refactor

  Without this plugin
  Check                         Score
  ------------------------------------------
  Conventional Commits format   30/30  (100%)
  Valid type                    0/20   (0%)
  Subject line <= 72 chars      0/30   (0%)
  Blank line separator          20/20  (100%)
  Total                         50/100  (50%)

  With this plugin
  Check                         Score
  ------------------------------------------
  Conventional Commits format   30/30  (100%)
  Valid type                    20/20  (100%)
  Subject line <= 72 chars      30/30  (100%)
  Blank line separator          20/20  (100%)
  Total                         100/100  (100%)

Scenario: Type selection for a small fix

  Without this plugin
  Total                         90/100  (90%)

  With this plugin
  Total                         95/100  (95%)

Summary
  Scenarios: 3 completed
  Without this plugin avg:  71%
  With this plugin avg:     92%

The score difference tells you what the skill is worth:

• A large positive difference means the skill reliably moves the agent toward the right behaviour.
• Little or no difference means the agent already did the task well without the skill, or the skill is not landing.
• A small negative difference is usually normal model variance, not evidence the skill is harmful.
• A large or consistent negative difference is a real sign the skill is confusing the agent or pushing it toward unhelpful behaviour, and the content needs work.

Read the per-scenario breakdown rather than just the headline average, since a strong result on one scenario can hide a weak one on another.

#Outcome and activation

A standard evaluation measures outcome: with the context forced on, does the agent produce better work against the rubric. It does not test whether the agent would have loaded the skill on its own.

To measure activation, whether the agent picks up the skill itself without being told to, add --skip-forced-context-activation. A skill can score well on outcome but fail to activate, which means it would not help in practice because the agent never reaches for it.

# Measure activation only: don't force the context, and skip outcome scoring
tessl eval run ./my-plugin --skip-forced-context-activation --skip-scoring

The two flags are independent: --skip-forced-context-activation stops forcing the context so activation can be observed, and --skip-scoring skips rubric scoring. Used together they give a pure activation check.

Example activation output:

Activation Results

Agent:     claude:deepseek-v4.1-flash
Variants:  baseline, with context
Status:    Completed

Injected context
Plugin: format-commit-messages@0.1.0

Scenario                                            Activated skills
--------------------------------------------------------------------
1. Conventional Commits format and type selection   format-commit-messages
2. Commit message formatting for complex refactor   format-commit-messages
3. Conventional Commits with multi-line message     —

— in the Activated skills column means the skill was not selected by the agent on its own. A skill that does not activate is a description problem, best fixed with a review. A skill that activates but does not improve the outcome is a content problem to fix in the skill body.

#Multi-skill plugins

A plugin can hold several skills. The commands are the same as for a single skill, and you always run them against the plugin root:

• tessl scenario generate ./my-plugin generates scenarios for the whole plugin. Download them into the one evals/ directory at the plugin root: run tessl scenario download --last from inside my-plugin/, or pass --output ./my-plugin/evals from elsewhere. If you started another generation since, pass the generation ID that scenario generate printed instead of --last. See Generate scenarios.
• tessl eval run ./my-plugin uses the whole plugin, with all its skills, as the context.
• --skill <name> narrows the context to the named skills. Repeat the flag to keep more than one. The other skills are left out of the with-context variant. Use the skill's name, as declared in its SKILL.md.

# Evaluate with every skill in the plugin as context
tessl eval run ./my-plugin

# Evaluate with only two of the skills as context
tessl eval run ./my-plugin --skill write-endpoint --skill review-endpoint

--skill works only with a local plugin context. It is not valid with a published --context workspace/plugin@version.

#Comparing test variants

A test variant is one configuration in an evaluation. Every variant solves the same scenarios and is scored against the same rubric, so you can compare models, agents, fixture combinations, or revisions of a plugin.

Declare variants with --variant-json. Give the flag a JSON array or the path to a JSON file:

tessl eval run ./my-plugin --variant-json ./variants.json

The run solves only the variants you list. It does not add a baseline for you, so include one in the list when you want to compare against it (see Compare a plugin with no plugin). --variant-json cannot be combined with --skip-baseline or --arms-json. A JSON file can be up to 1 MiB.

--arms-json is deprecated but still works. Use --variant-json for new runs.

Each variant needs a unique label. A label uses lowercase letters, digits, hyphens, and underscores, starts with a letter or digit, and is at most 64 characters, so a model name such as kimi-k2.6 needs a label without the dot. Fields you omit use the run-level agent, model, and fixture setup. The bare string "baseline" is the exception: it removes the run-level subject fixture. An object variant, including { "label": "baseline" }, uses the ordinary object rules and includes that fixture.

[
  { "label": "opus", "model": "claude-opus-5" },
  { "label": "sonnet", "model": "claude-sonnet-5" },
  { "label": "haiku", "model": "claude-haiku-4-5" }
]

#Variant spec reference

Common errors:

• Duplicate labels return Variant labels must be unique within a run.
• A label with a dot returns an error that includes label must match pattern "^[a-z0-9][a-z0-9_-]{0,63}$".
• Combining --skip-baseline with --variant-json returns --variant-json and --skip-baseline are mutually exclusive. Use --variant-json for explicit variants, or --skip-baseline for a classic eval run.

agentMustRead contains fixture names, not paths or file content. For a commit or directory fixture, the agent is told that fixture's installPath. For a plugin fixture, the agent is told the locations of the installed skills. A name must identify a fixture available to that variant after its overlay has been applied.

For example, this scenario supplies a commit fixture at the repository root and excludes draft instructions:

{
  "fixtures": {
    "repository": {
      "type": "commit",
      "repoUrl": "https://github.com/org/repo.git",
      "ref": "abc1234",
      "installPath": ".",
      "exclude": ["docs/drafts/**"]
    }
  }
}

The variant adds a directory fixture named guidance and names both fixtures in agentMustRead. The agent is told where the repository and guidance fixtures were installed.

The CLI packages the contents of a localPath fixture and uploads them for the evaluation. The evaluation agent can access that uploaded content, so point localPath only at files you intend to share.

[
  {
    "label": "with-guidance",
    "fixtures": {
      "guidance": { "localPath": "./variants/guidance" }
    },
    "agentMustRead": ["repository", "guidance"]
  }
]

#Fixture values in a variant

A fixture value in a variant takes one of two forms:

• { "localPath": "./path" } for content on your disk. The path is relative to the directory you run the command from. A plugin directory is installed as a plugin. Any other directory is copied into the solution at its path relative to your repository root. localPath must be the only key in the object. The CLI uploads each distinct path once.
• A published plugin, declared by name and version. The run resolves it from the registry, so nothing is uploaded:

{
  "label": "published",
  "fixtures": {
    "context": {
      "definition": { "type": "plugin", "name": "engteam/my-plugin", "version": "1.2.0" }
    }
  }
}

Other fixture types need uploaded content, so declare them with localPath and let the CLI upload them. A localPath that resolves outside your Git repository (or outside the source directory, when it is not in a repository) is refused unless you pass --allow-unsafe-fixture-paths.

#Compare a plugin with no plugin

Passing ./my-plugin already supplies the run-level subject fixture. The bare string "baseline" is a special case that removes that fixture. Every object variant, including { "label": "baseline" }, inherits it, so use an object with no fixtures to run the plugin once.

[
  "baseline",
  {
    "label": "with-plugin"
  }
]

#Repeats and limits

Repeat each variant with --runs (or -n) when a decision depends on a small score difference. One attempt per variant cannot show how much agent behavior varies between runs. If two variants' score spreads overlap heavily, the run does not show a real difference between them. It does not show that they are equivalent either. Add repeats before you decide.

• --runs accepts 1 to 10.
• A run can plan at most 2,000 solves. The count is scenarios × variants × runs. Reduce one of them if the product is higher.
• A run can contain at most 100 fixture overlay entries across all object variants.
• The variant list can contain at most 50 localPath references, which resolve to at most 20 different local directories.

#Choose how rigorous to be

Evaluation is a toolkit, not a single fixed test. Scale the rigour to how much confidence you need.

• More scenarios. Generate a higher --count to cover more of what the skill claims to do.
• Repeated runs. An evaluation runs a real agent, and agents are non-deterministic, so a single run can mislead. Pass --runs (or -n) to run each scenario several times and average out the variance. Worth doing before you publish.
• Different agents. Run the evaluation against more than one agent with --agent to check the skill holds up across the models your team uses.
• Quality filter. Pass --quality-check to exclude scenarios the platform judges low-feasibility, rubric-leaking, or low-signal at run time, a stricter filter than the one applied at generation.
• Skip the baseline. Pass --skip-baseline to run only the with-context variant when you do not need the comparison.
• Override the scorer. Pass --scorer-agent to score with a different agent from the one solving the tasks.

#Passing secrets to an evaluation

Some scenarios need external credentials, for example an API key or a database URL. The --env-file flag encrypts a local .env file client-side and injects the variables into the eval sandbox, where setup scripts and the agent inherit them:

tessl eval run ./my-plugin --env-file ./secrets.env

The file uses standard KEY=VALUE lines; blank lines and # comments are ignored. The CLI rejects reserved keys at submit time, including LITELLM_ADMIN_KEY, LITELLM_MASTER_KEY, UPLOAD_WORKSPACE, DOWNLOAD_WORKSPACE, and any key prefixed AWS_, S3_, DAYTONA_, or RECIPE_RUN_. Rename conflicting keys before running.

#Choosing an agent

By default an evaluation runs against agent claude, and in the US region model deepseek-v4.1-flash. Choose a different agent and model with --agent and --model. To compare models or agents in one run, set model or agent on explicit test variants.

The default model and the available models both depend on the region you are connected to: the open-weight models below (DeepSeek, GLM, Kimi, Qwen, MiniMax, GPT OSS, and Nemotron) are not offered in the EU region, which defaults to a Claude model instead. Run tessl eval run --list-agents to print the models and the default for your region.

tessl eval run ./my-plugin --agent claude --model claude-opus-4-8

--agent values: claude, codex, tessl. Pair each with a --model value from its tab below.

#Evaluations when you publish

Publishing a plugin to the registry runs an evaluation as part of publishing, and the results appear on the registry. The evaluation is optional: pass --skip-evals to publish without it, or --with-scenario-quality-check to apply the quality filter to the publish-time run. See Distributing via registry.

#Plugin layout

A plugin that holds evaluations looks like this on disk. evals/ sits at the plugin root, next to skills/, and holds the scenarios for every skill in the plugin:

my-plugin/
├── .tessl-plugin/
│   └── plugin.json          workspace, name, version
├── skills/
│   ├── write-endpoint/
│   │   └── SKILL.md
│   └── review-endpoint/
│       └── SKILL.md
├── evals/
│   ├── scenario-0/
│   │   ├── task.md          the task brief the agent sees
│   │   ├── criteria.json    the scoring rubric
│   │   └── scenario.json    fixture and include declarations
│   └── scenario-1/
└── tessl.json               links the directory to a Tessl project

#File formats

#task.md

Free-form markdown, and the only file the agent sees. Typically structured with problem, expected behaviour, and acceptance criteria sections. Edit it freely before running.

#criteria.json

Defines how a solution is scored, as a weighted checklist:

{
  "context": "Brief description of what is being evaluated.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "fixes_syntax_error",
      "description": "The solution adds the missing closing parenthesis.",
      "max_score": 1,
      "category": "INTENT"
    }
  ]
}

Each checklist item carries a category: INTENT, DESIGN, MUST_NOT, MINIMALITY, REUSE, INTEGRATION, or EDGE_CASE. The score is the sum of awarded scores over the sum of maximum scores, as a percentage, and the grader can award partial credit.

#scenario.json

Optional. Declares fixtures, included files, and setup scripts. A skill scenario can simply list include paths to copy into the working directory. A codebase scenario carries a commit fixture with the repository URL, the starting commit, and the context exclude patterns. See the configuration reference for the full schema.

#Next

• Codifying and enforcing your skill standards - set an organisation-wide quality bar and enforce it.
• Troubleshooting reviews and evaluations - what to do when a run fails or scores look wrong.