Evaluating Policies in Visual Agents – Camio

Evaluations let you measure how well a Visual Agents policy performs before you put it live. First you build a dataset of past results that you've labeled with the correct answer. Then you run one or more policy versions against it and compare their scores side by side.

Before you start: Original media

A dataset is built from the original images or video behind each result, so a result can only be added if Camio kept that original media. Original media isn't kept when a policy analyzes only part of the footage. For example:

Include Original Media is turned off in your policy settings and the policy evaluates only zone or tracing crops.
Video is converted to extracted frames and the original video isn't uploaded.

When a result can't be added, the Label & Add to Dataset window shows a message saying so. If you'd like to build datasets from results like these, email support@camio.com and we can keep original media for your account.

Step 1: Create a dataset

Go to the Datasets page, then click + Create Dataset.
Enter a Name and, optionally, a Description, then click Create.

Add labeled results

Add results to the dataset from the Results Search page:

Open a result and click Label & Add to Dataset (or press L).
Tick the dataset. Choose Compliant or Noncompliant for what actually happened, and add an optional note explaining why.
Changes save automatically. Click Close when you're done.

Tip: label quickly with Labeling Mode. On the Results Search page, click Labeling Mode (or press T) and select your dataset(s). Use ↑/↓ to move between results and ←/→ to step through each result's images or video, then:

C labels the result Compliant.
N labels it Noncompliant.
O adds a note.
D removes it from the dataset.

Open a dataset from the Datasets page to review its items, edit labels, notes and camera names (one at a time or in bulk), and see how your original results scored against your labels.

Tip: Build your dataset from a representative sample instead of hand-picking examples. For example, label every result from a block of time (such as one hour), or a random selection of results. Hand-picking examples, or balancing Compliant and Noncompliant ones, gives scores that don't reflect how the policy performs day to day. Larger datasets give more reliable scores, and evaluations of fewer than 20 items are marked as a small sample.

Recommended: Use policy version history

We recommend evaluating saved policy versions rather than the live policy. If the live policy is edited while an evaluation is running, items tested after the edit are scored against different content, and the run can't be reproduced later. A saved version stays the same, so you can compare versions fairly and rerun them at any time.

The quickest way to start: if you choose Live policy in the Run evaluation window for a policy without version history, click Enable version history there. This saves the policy's current content as version 1, and that version is evaluated instead.

To set up version history and try changes from the policy itself:

Open your policy on the Policy page. If version history isn't on yet, click Enable Version History in the more policy actions menu.
To try a change without affecting the live policy, edit the policy and click Save as Version. The change is saved as a new version in Version History, and the live policy stays as it is.
Evaluate the new version against the live policy's version. When it scores better, make it live

Step 2: Run an evaluation

Go to the Evaluations page and click Run evaluation, or open the Run evaluation window directly. You can also click Run evaluation on a dataset's page, or click the play icon next to a version in a policy's version history.
Choose a Dataset.
Under Policies to evaluate, choose a policy and a version. To compare, click + Add a policy or version to compare. You can add up to 5 in total.
Optionally, raise the × count to run the same version several times and see how much its results vary.
Check the summary of how many tests will run and roughly how long they'll take, then click Start evaluation.

Tip: Evaluate a saved version rather than Live policy (see Recommended: use policy version history).

Limits: each evaluation can include up to 5,000 tests (items × versions), and your account can have up to 2 evaluations running at once. You can leave the page while an evaluation runs. To stop one early, click Cancel run.

Step 3: Review the results

The Evaluations page lists every run with its progress and scores. Click on a row to see the details.

Each version is scored against your labels. Noncompliant is treated as the "positive" outcome:

Accuracy: the share of all items the policy got right.
Precision: of the items the policy flagged Noncompliant, the share that really were. Low precision means false alarms.
Recall: of the items that really were Noncompliant, the share the policy caught. Low recall means missed violations.
Specificity: of the items that really were Compliant, the share the policy correctly passed.
Error %: the share of tests that failed to run. These are left out of the other scores.

Inside a run:

The Leaderboard ranks versions by the score you pick. Use Details to break the results down by confusion matrix, iteration or camera.
Turn on Compare against a baseline to measure against your original results or another version, including one from an earlier run.
The Items panel shows the Incorrect items by default. Click any mark to view the media and each version's verdict. Use the filters to see wins, losses, disagreements and more.

Note: The individual results from an evaluation, including their media, are kept for only a few days, so review them soon after the run finishes. The run's scores stay on the Evaluations page.

Improving your policy

To use Evaluations to improve a policy:

Review the incorrect items in a run.
Fix any items that were labeled incorrectly in the dataset, or edit your policy and save it as a new version.
Run a new evaluation comparing to the previous run as a baseline.
Check the wins and losses to confirm the change helped before you make it live.