Measuring useful work

Distinguish provisioned capacity from productive work

A reservation describes access under a set of conditions. A measured run describes useful work done under its own conditions. Account for queueing, checkpointing, restart and failed work in the elapsed time. Do not multiply nominal devices by peak throughput and treat that as delivered capacity.

Use MLPerf within its boundaries

MLCommons publishes benchmark workloads with target-quality and reporting rules. Compare like benchmark, version, division and system configuration, then test your own model and data. Submitted outcomes do not establish that a supplier will meet your workload target.

Make acceptance repeatable

• Record system shape, framework and version, data recipe and quality threshold.
• Measure both useful throughput and elapsed time, and document recovery after a failure.
• Preserve the results, date, logs and responsibility for a repeat test.

Convert evidence into a requirement

Build an evidence brief and mark unconfirmed findings as open questions.