runtimequestion 367 Profiling & Optimization · medium
Flag benchmark regressions in CI
A CI job runs every benchmark several times on the main branch and on a pull request. Write regressions(baseline, current, tolerance) that lists the benchmarks whose best time got worse by more than tolerance (0.1 means 10%). Benchmarks that are new in current have nothing to compare with and are skipped.
Function
regressions(baseline, current, tolerance)baseline, currentdictionaries from benchmark name to a list of round times
Returnsthe sorted names where
min(current) > min(baseline) * (1 + tolerance)Examples
callregressions({'parse_orders': [0.4, 0.41], 'export_csv': [1.2, 1.3]}, {'parse_orders': [0.52, 0.5], 'export_csv': [1.25, 1.21], 'new_report': [0.9]}, 0.1)returns['parse_orders']
callregressions({'a': [1]}, {'a': [1]}, 0.05)returns[]
callregressions({'b': [2], 'a': [1]}, {'b': [3], 'a': [2]}, 0.2)returns['a', 'b']