This page is primarily intended for developers of Mercurial.

Performance tracking infrastructure

Status: In production

Main proponents: Pierre-YvesDavid PhilippePepiot RaphaëlGomes

Provide a continuous integration infrastructure to measuring and preventing performance regressions on mercurial

1. Goal

Mercurial code change fast and we must detect and prevent performances  regressions as soon as possible.

2. Metrics

We already have code that produce performance metrics:

Another idea is to produce metrics from annotated portions of unit test execution time.

These metrics will be used (after some refactoring for some of the tools that  produce them) as performance metrics, but we may need some more specifically written for the purpose of performance regression detection

3. Tool selection

After evaluating several tools we choose to use Airspeed velocity that already handle most of our needs.

This tool aims at benchmarking Python packages over their lifetime.  It is mainly a command line tool, asv, that run a series of benchmarks (described  in JSON configuration file), and produces a static HTML/JS report.

When running a benchmark suite, ASV take care of clone/pulling the source repository  in a virtual env and running the configured tasks in this virtual env.

Results of each benchmark execution are stored in a "database" (consisting in JSON files). This database is used to produce evolution plots of the time required to run a test (or any metrics; out of the box, asv has support for 4 types of benchmark:  timing, memory, peak memory and tracking), and to run the regression detection algorithms.

One key feature of this tool is that it's very easy for every developer to use it on  its own development environment. For example, it provides an asv compare command allowing to compare  the results of any 2 revisions.

4. Q & A

5. Plan

6. Code

The current work in progress code can be cloned with hg clone -b default

CategoryDeveloper CategoryNewFeatures

PerformanceTrackingSuitePlan (last edited 2020-02-13 22:37:48 by Pierre-YvesDavid)