FuzzBench: edges report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_edges 88.09
afl 74.36
By avg. rank
average rank
fuzzer
afl 1.48
afl_edges 1.52
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl afl_edges
    FuzzerMedian 97.25 97.70
    FuzzerMean 95.29 92.76
    arrow_parquet-arrow-fuzz 97.15 97.77
    aspell_aspell_fuzzer 99.80 99.89
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan
    file_magic_fuzzer 98.33 97.63
    grok_grk_decompress_fuzzer 91.84 99.56
    libarchive_libarchive_fuzzer 86.47 93.73
    libgit2_objects_fuzzer 99.75 99.80
    libhevc_hevc_dec_fuzzer 78.73 7.15
    libhtp_fuzz_htp 99.95 99.94
    libxml2_libxml2_xml_reader_for_file_fuzzer 91.60 91.94
    matio_matio_fuzzer 97.00 96.47
    muparser_set_eval_fuzzer 98.01 98.01
    ndpi_fuzz_ndpi_reader 100.00 100.00
    njs_njs_process_script_fuzzer 95.05 96.92
    openh264_decoder_fuzzer 99.44 99.74
    php_php-fuzz-execute 94.00 98.35
    php_php-fuzz-parser-2020-07-25 99.24 99.80
    poppler_pdf_fuzzer 99.71 98.47
    proj4_standard_fuzzer 100.00 99.47
    stb_stbi_read_fuzzer 86.90 91.79
    systemd_fuzz-varlink 88.48 88.48
    tpm2_tpm2_execute_command_fuzzer 95.60 89.28
    usrsctp_fuzzer_connect 94.56 93.79
    wireshark_fuzzshark_ip 97.34 95.15
    zstd_stream_decompress 97.97 93.17
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_edges afl
    FuzzerMedian 50.00 33.33
    FuzzerMean 42.84 37.02
    arrow_parquet-arrow-fuzz 88.10 90.48
    aspell_aspell_fuzzer nan nan
    ffmpeg_ffmpeg_demuxer_fuzzer 25.00 50.00
    file_magic_fuzzer nan nan
    grok_grk_decompress_fuzzer 80.00 20.00
    libarchive_libarchive_fuzzer nan nan
    libgit2_objects_fuzzer 66.67 33.33
    libhevc_hevc_dec_fuzzer 0.00 0.00
    libhtp_fuzz_htp 60.00 80.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 16.67 16.67
    matio_matio_fuzzer 73.08 76.92
    muparser_set_eval_fuzzer nan nan
    ndpi_fuzz_ndpi_reader nan nan
    njs_njs_process_script_fuzzer 0.00 0.00
    openh264_decoder_fuzzer 57.14 42.86
    php_php-fuzz-execute 33.33 0.00
    php_php-fuzz-parser-2020-07-25 75.00 12.50
    poppler_pdf_fuzzer 33.33 66.67
    proj4_standard_fuzzer nan nan
    stb_stbi_read_fuzzer 70.00 90.00
    systemd_fuzz-varlink nan nan
    tpm2_tpm2_execute_command_fuzzer nan nan
    usrsctp_fuzzer_connect 0.00 0.00
    wireshark_fuzzshark_ip 50.00 50.00
    zstd_stream_decompress 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl_edges afl
    FuzzerSum 228 181 179
    arrow_parquet-arrow-fuzz 99 81 84
    aspell_aspell_fuzzer 0 0 0
    ffmpeg_ffmpeg_demuxer_fuzzer 8 6 7
    file_magic_fuzzer 0 0 0
    grok_grk_decompress_fuzzer 5 5 2
    libarchive_libarchive_fuzzer 0 0 0
    libgit2_objects_fuzzer 3 3 3
    libhevc_hevc_dec_fuzzer 11 1 11
    libhtp_fuzz_htp 5 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 14 10 12
    matio_matio_fuzzer 21 19 17
    muparser_set_eval_fuzzer 0 0 0
    ndpi_fuzz_ndpi_reader 0 0 0
    njs_njs_process_script_fuzzer 1 0 1
    openh264_decoder_fuzzer 9 9 5
    php_php-fuzz-execute 14 11 4
    php_php-fuzz-parser-2020-07-25 12 12 6
    poppler_pdf_fuzzer 7 4 5
    proj4_standard_fuzzer 0 0 0
    stb_stbi_read_fuzzer 12 11 12
    systemd_fuzz-varlink 0 0 0
    tpm2_tpm2_execute_command_fuzzer 0 0 0
    usrsctp_fuzzer_connect 1 1 0
    wireshark_fuzzshark_ip 4 2 4
    zstd_stream_decompress 2 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 37.647059 2.956797 29.0 37.00 38.0 39.0 42.0
    afl_edges 82800 20.0 36.700000 2.754900 32.0 35.75 37.0 37.5 42.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 5178.550000 75.025241 5017.0 5134.75 5182.0 5247.5 5295.0
    afl 82800 17.0 5167.882353 79.497549 5059.0 5101.00 5149.0 5226.0 5300.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 5551.9 6.273587 5538.0 5548.75 5554.0 5556.0 5559.0
    afl 82800 20.0 5539.8 20.793724 5508.0 5511.75 5549.0 5555.5 5560.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.25 1.208522 0.0 1.75 2.0 3.0 4.0
    afl_edges 82800 20.0 1.35 1.182103 0.0 0.00 1.0 2.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3737.65 184.373270 2997.0 3711.0 3794.5 3823.25 3859.0
    afl_edges 82800 20.0 3740.55 62.769818 3639.0 3677.0 3767.5 3791.25 3823.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 4.00 0.794719 2.0 4.0 4.0 4.25 5.0
    afl 82800 20.0 1.15 0.587143 0.0 1.0 1.0 1.25 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 9523.75 134.497456 8959.0 9539.75 9548.0 9564.75 9590.0
    afl 82800 20.0 8807.90 28.228205 8754.0 8791.75 8807.0 8828.00 8863.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 6356.45 263.854919 5926.0 6119.00 6409.0 6569.25 6838.0
    afl 82800 20.0 5906.85 409.748482 5227.0 5585.25 5912.5 6031.50 6608.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 2.10 0.552506 1.0 2.0 2.0 2.0 3.0
    afl 82800 20.0 1.25 0.550120 1.0 1.0 1.0 1.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 2439.3 1.301821 2437.0 2439.0 2439.0 2439.25 2444.0
    afl 82800 20.0 2438.3 0.656947 2437.0 2438.0 2438.0 2439.00 2440.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.80 2.440880 0.0 0.0 0.0 1.0 11.0
    afl_edges 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 8843.00 4211.903808 878.0 8776.5 9674.5 12206.5 12288.0
    afl_edges 82800 20.0 5416.05 5187.084816 876.0 878.0 878.0 11097.5 12186.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3.5 1.357242 0.0 3.00 4.0 4.25 5.0
    afl_edges 82800 20.0 2.9 1.020836 1.0 2.75 3.0 3.00 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 6579.85 1.694418 6575.0 6579.0 6580.0 6581.0 6583.0
    afl_edges 82800 20.0 6578.85 1.424411 6576.0 6578.0 6579.0 6580.0 6581.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.5 2.236068 2.0 2.0 2.0 2.0 12.0
    afl_edges 82800 20.0 2.4 1.788854 2.0 2.0 2.0 2.0 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 20719.35 536.576602 20146.0 20393.5 20531.5 20769.0 22085.0
    afl 82800 20.0 20618.55 679.225060 19622.0 20220.0 20455.0 20829.0 22331.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 9.60 1.231174 7.0 8.75 10.0 10.25 11.0
    afl_edges 82800 20.0 9.65 1.348488 8.0 9.00 9.5 10.00 13.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2759.4 23.270492 2707.0 2747.00 2762.5 2774.50 2800.0
    afl_edges 82800 20.0 2759.1 34.396909 2722.0 2736.75 2747.5 2770.75 2848.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 935.45 5.020746 925.0 935.00 935.0 936.0 954.0
    afl_edges 82800 20.0 934.00 8.503869 906.0 932.75 935.0 936.0 950.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ndpi_fuzz_ndpi_reader summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 90.6 7.829768 77.0 91.0 95.0 95.0 95.0
    afl_edges 82800 20.0 94.1 4.024922 77.0 95.0 95.0 95.0 95.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.3 0.470162 0.0 0.0 0.0 1.0 1.0
    afl_edges 82800 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 7340.00 131.842410 7121.0 7226.0 7299.0 7477.25 7531.0
    afl 82800 20.0 7185.45 132.880427 6940.0 7115.0 7158.5 7239.50 7469.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 4.35 0.812728 4.0 4.00 4.0 4.00 7.0
    afl 82800 20.0 2.95 0.998683 1.0 2.75 3.0 3.25 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 14308.55 53.627835 14151.0 14316.0 14327.0 14341.0 14348.0
    afl 82800 20.0 14265.80 83.651281 14109.0 14210.0 14284.0 14339.5 14365.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 1.20 0.951453 0.0 0.75 1.0 2.00 3.0
    afl 82800 20.0 0.25 0.444262 0.0 0.00 0.0 0.25 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 181694.2 1338.835614 179513.0 180776.00 181820.5 182363.25 184879.0
    afl 82800 20.0 173110.5 3311.151755 162715.0 171942.25 173795.5 175480.75 177260.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 6.0 0.973329 4.0 5.0 6.0 7.0 8.0
    afl 82800 20.0 1.4 1.231174 0.0 0.0 1.0 2.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 45072.50 40.589732 44965.0 45049.75 45074.0 45103.50 45132.0
    afl 82800 20.0 44819.15 239.798441 44247.0 44658.00 44821.0 45021.25 45164.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.75 0.786398 1.0 1.0 2.0 2.0 3.0
    afl_edges 82800 20.0 1.35 0.587143 1.0 1.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 38219.65 100.842908 37960.0 38178.5 38247.5 38282.0 38360.0
    afl_edges 82800 20.0 37784.55 130.471806 37603.0 37684.5 37773.5 37842.5 38121.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 374.00 0.000000 374.0 374.0 374.0 374.0 374.0
    afl_edges 82800 20.0 371.55 0.510418 371.0 371.0 372.0 372.0 372.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 8.6 1.095445 7.0 8.0 9.0 9.25 10.0
    afl_edges 82800 20.0 7.0 1.025978 5.0 6.0 7.0 8.00 9.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_edges 82800 20.0 2539.9 137.992334 2285.0 2409.5 2564.5 2602.00 2794.0
    afl 82800 20.0 2492.6 99.079660 2412.0 2418.5 2428.0 2602.25 2668.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

tpm2_tpm2_execute_command_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 5556.7 539.949423 4417.0 5172.75 5852.5 5953.75 6122.0
    afl_edges 82800 20.0 5402.3 465.658572 4647.0 4903.50 5465.5 5787.50 5999.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    afl_edges 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 13726.0 350.119228 13368.0 13481.25 13541.5 14145.0 14321.0
    afl_edges 82800 20.0 13527.7 260.181738 13296.0 13410.50 13431.5 13474.5 14207.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.20 0.410391 1.0 1.0 1.0 1.0 2.0
    afl_edges 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 471729.70 15569.756544 425420.0 467580.25 473985.5 484118.0 486926.0
    afl_edges 82800 20.0 461088.85 14287.432900 407001.0 458499.75 463324.5 468674.5 476015.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    afl_edges 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 9790.20 358.063152 8680.0 9637.0 9917.0 10028.25 10123.0
    afl_edges 82800 20.0 9428.35 451.899532 8530.0 9232.0 9432.0 9814.50 10010.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 45668bb09ccc1f7ab7f2cdebee11a18b9d8d4e27

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 45668bb09ccc1f7ab7f2cdebee11a18b9d8d4e27
# Download the internal config file.
curl https://storage.googleapis.com/edges/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data