AI Agents by the Numbers: Step Success Rates, Retries, Tool-Call Costs, Human Hand-Offs & Cost per Completed Task

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Multiply six step success rates into the end-to-end success of a fictional invoice agent, find its weakest step, and work out how many equally good steps a chain can have before it misses a target.
• Calculate how retries lift each step's success, how many attempts a step needs on average and what that costs, and you compare the end-to-end success with and without retries.
• Price each step of one agent run from its input tokens, output tokens and tool calls, add them up to the cost of one attempt and see how much of the token bill the output tokens cause.
• Turn the cost per attempt of three agent designs into the cost per completed task, once when failed runs are re-run and once when a person fixes them, and let formulas pick the cheapest design.
• Route 20 logged runs to the agent or a person with a confidence threshold, measure the hand-off share and the wrong answers that slip through, and price the reviews for a month.
• Add the retry delays to 20 logged run times, measure the share of runs that miss a time limit and compare the 90th percentile with the average.
• Turn an evaluation log of 30 test invoices into reached and passed counts per step, honest step success rates and the end-to-end completion rate, and find where a check step would pay off.
• Compare the monthly cost of handling invoices by people with the agent's cost including reviews, fees and maintenance, work out the payback of the build cost and chart the two costs.