FuzzBench: trimming report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_no_trim 99.25
afl 88.81
By avg. rank
average rank
fuzzer
afl_no_trim 1.38
afl 1.62
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_no_trim afl
    FuzzerMedian 96.78 95.89
    FuzzerMean 95.26 94.44
    arrow_parquet-arrow-fuzz 96.78 95.36
    aspell_aspell_fuzzer 99.83 99.75
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan
    file_magic_fuzzer 98.38 96.28
    grok_grk_decompress_fuzzer 93.60 92.56
    libarchive_libarchive_fuzzer 85.16 81.80
    libgit2_objects_fuzzer 99.80 99.75
    libhevc_hevc_dec_fuzzer 79.73 79.84
    libhtp_fuzz_htp 99.92 99.95
    libxml2_libxml2_xml_reader_for_file_fuzzer 93.31 90.96
    matio_matio_fuzzer 98.13 96.43
    mruby-2018-05-23 92.76 87.65
    muparser_set_eval_fuzzer 96.30 96.30
    njs_njs_process_script_fuzzer 96.78 96.80
    openh264_decoder_fuzzer 99.85 99.44
    php_php-fuzz-execute 94.32 94.27
    php_php-fuzz-parser-2020-07-25 99.70 99.35
    poppler_pdf_fuzzer 98.21 96.61
    proj4_standard_fuzzer 100.00 100.00
    quickjs_eval-2020-01-05 93.14 95.23
    stb_stbi_read_fuzzer 92.03 92.28
    systemd_fuzz-varlink 88.48 88.48
    usrsctp_fuzzer_connect 94.41 94.16
    wireshark_fuzzshark_ip 97.07 95.49
    zstd_stream_decompress 98.49 97.74
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_no_trim afl
    FuzzerMedian 68.33 54.17
    FuzzerMean 57.34 51.95
    arrow_parquet-arrow-fuzz 91.49 87.23
    aspell_aspell_fuzzer 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer 70.00 70.00
    file_magic_fuzzer 0.00 0.00
    grok_grk_decompress_fuzzer 25.00 12.50
    libarchive_libarchive_fuzzer nan nan
    libgit2_objects_fuzzer 33.33 33.33
    libhevc_hevc_dec_fuzzer 85.94 87.50
    libhtp_fuzz_htp 80.00 80.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 15.38 15.38
    matio_matio_fuzzer 84.62 76.92
    mruby-2018-05-23 66.67 33.33
    muparser_set_eval_fuzzer nan nan
    njs_njs_process_script_fuzzer 33.33 33.33
    openh264_decoder_fuzzer 75.00 50.00
    php_php-fuzz-execute 33.33 33.33
    php_php-fuzz-parser-2020-07-25 40.00 30.00
    poppler_pdf_fuzzer 89.58 58.33
    proj4_standard_fuzzer nan nan
    quickjs_eval-2020-01-05 100.00 100.00
    stb_stbi_read_fuzzer 79.55 75.00
    systemd_fuzz-varlink 100.00 100.00
    usrsctp_fuzzer_connect 0.00 0.00
    wireshark_fuzzshark_ip 58.33 66.67
    zstd_stream_decompress 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl_no_trim afl
    FuzzerSum 328 286 277
    arrow_parquet-arrow-fuzz 69 63 65
    aspell_aspell_fuzzer 2 2 2
    ffmpeg_ffmpeg_demuxer_fuzzer 31 23 25
    file_magic_fuzzer 1 1 1
    grok_grk_decompress_fuzzer 9 9 8
    libarchive_libarchive_fuzzer 0 0 0
    libgit2_objects_fuzzer 3 3 3
    libhevc_hevc_dec_fuzzer 35 35 34
    libhtp_fuzz_htp 5 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 13 2 13
    matio_matio_fuzzer 20 19 15
    mruby-2018-05-23 10 10 8
    muparser_set_eval_fuzzer 0 0 0
    njs_njs_process_script_fuzzer 4 4 3
    openh264_decoder_fuzzer 8 8 5
    php_php-fuzz-execute 7 7 1
    php_php-fuzz-parser-2020-07-25 11 8 10
    poppler_pdf_fuzzer 60 49 41
    proj4_standard_fuzzer 0 0 0
    quickjs_eval-2020-01-05 1 1 1
    stb_stbi_read_fuzzer 26 25 25
    systemd_fuzz-varlink 1 1 1
    usrsctp_fuzzer_connect 1 1 1
    wireshark_fuzzshark_ip 10 9 9
    zstd_stream_decompress 1 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 11.0 43.000000 2.792848 38.0 42.0 43.0 45.0 47.0
    afl 82800 11.0 38.454545 9.709414 10.0 40.0 41.0 42.5 44.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 11.0 5183.727273 90.536281 5060.0 5117.5 5168.0 5216.5 5340.0
    afl 82800 11.0 5099.272727 56.505028 5022.0 5063.0 5092.0 5120.5 5233.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_no_trim 82800 18.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 18.0 5538.777778 4.066442 5533.0 5535.25 5538.5 5541.75 5547.0
    afl 82800 16.0 5524.750000 20.381364 5493.0 5499.00 5534.0 5538.00 5548.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 6.823529 1.704233 4.0 5.0 7.0 8.0 10.0
    afl_no_trim 82800 19.0 6.578947 1.835326 3.0 5.5 7.0 8.0 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_trim 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 11.0 0.272727 0.467099 0.0 0.0 0.0 0.50 1.0
    afl_no_trim 82800 18.0 0.277778 0.460889 0.0 0.0 0.0 0.75 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 18.0 3767.000000 54.953456 3676.0 3712.5 3786.5 3806.0 3849.0
    afl 82800 11.0 3736.090909 62.221306 3666.0 3684.0 3706.0 3795.0 3822.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 19.0 2.473684 2.010208 1.0 1.0 2.0 2.5 8.0
    afl 82800 18.0 1.333333 1.495090 0.0 1.0 1.0 1.0 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 19.0 8997.473684 235.289271 8842.0 8892.00 8915.0 8926.0 9525.0
    afl 82800 18.0 8853.944444 150.235295 8779.0 8802.25 8816.0 8844.5 9447.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_trim 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 18.0 5910.666667 415.926888 5253.0 5817.50 5892.5 6024.5 6919.0
    afl 82800 12.0 5733.083333 444.325323 5187.0 5379.25 5660.0 6118.0 6401.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 1.421053 0.606977 1.0 1.0 1.0 2.0 3.0
    afl_no_trim 82800 19.0 1.210526 0.418854 1.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 19.0 2439.210526 1.812884 2438.0 2438.0 2439.0 2439.0 2444.0
    afl 82800 19.0 2438.842105 1.424514 2438.0 2438.0 2438.0 2439.0 2444.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 23.842105 10.264627 1.0 26.0 28.0 29.00 31.0
    afl_no_trim 82800 16.0 22.812500 11.034605 1.0 25.0 27.5 28.25 32.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 8500.0000 3467.060346 953.0 8749.00 9535.0 10653.5 11311.0
    afl_no_trim 82800 16.0 8522.8125 3884.133009 953.0 9172.75 9521.5 11300.0 11942.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3.470588 1.067570 1.0 3.0 4.0 4.00 5.0
    afl_no_trim 82800 14.0 4.142857 0.662994 3.0 4.0 4.0 4.75 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 6576.588235 2.399448 6570.0 6576.00 6577.0 6578.00 6580.0
    afl_no_trim 82800 14.0 6575.000000 2.601775 6570.0 6573.25 6575.0 6576.75 6580.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 2.785714 2.939874 2.0 2.0 2.0 2.0 13.0
    afl_no_trim 82800 15.0 2.000000 0.000000 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 15.0 21206.133333 795.218197 19844.0 20669.5 20923.0 22042.50 22423.0
    afl 82800 14.0 20468.142857 679.825867 19151.0 20271.0 20396.0 20678.75 22267.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl_no_trim.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 12.0 10.833333 0.834847 10.0 10.0 11.0 11.25 12.0
    afl 82800 18.0 10.055556 1.392088 7.0 9.0 10.0 11.00 12.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 12.0 2807.250000 26.765395 2762.0 2793.25 2803.5 2818.75 2857.0
    afl 82800 18.0 2751.666667 28.354894 2708.0 2730.00 2755.0 2764.00 2817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby-2018-05-23 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 56700 6.0 4.000000 1.414214 2.0 3.25 4.0 4.75 6.0
    afl 56700 13.0 2.230769 1.235168 1.0 1.00 2.0 3.00 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 56700 6.0 17034.000000 466.810454 16650.0 16695.25 16872.5 17208.0 17853.0
    afl 56700 13.0 16130.615385 445.420314 15607.0 15779.00 15944.0 16436.0 17061.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_trim 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 935.333333 1.414214 930.0 935.00 936.0 936.0 936.0
    afl_no_trim 82800 16.0 938.875000 9.646243 934.0 935.75 936.0 936.0 972.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 1.444444 0.615699 1.0 1.0 1.0 2.0 3.0
    afl_no_trim 82800 16.0 1.437500 0.512348 1.0 1.0 1.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 7261.722222 101.293868 7092.0 7190.25 7271.0 7345.5 7400.0
    afl_no_trim 82800 16.0 7276.062500 129.146670 6999.0 7213.00 7269.5 7358.0 7511.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 16.0 4.5625 0.813941 3.0 4.0 4.5 5.0 6.0
    afl 82800 13.0 3.0000 1.154701 1.0 2.0 3.0 4.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 16.0 14384.875000 10.493649 14367.0 14375.75 14386.5 14390.5 14406.0
    afl 82800 13.0 14306.230769 47.000273 14240.0 14268.00 14328.0 14338.0 14375.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 0.928571 0.267261 0.0 1.0 1.0 1.0 1.0
    afl_no_trim 82800 11.0 1.454545 0.687552 1.0 1.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 11.0 179834.818182 4091.014809 172201.0 177617.00 178919.0 182521.00 185901.0
    afl 82800 14.0 177239.142857 5819.063867 160901.0 174667.75 178808.0 180493.25 184957.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 13.0 2.384615 1.609268 0.0 1.0 2.0 4.00 5.0
    afl 82800 16.0 1.687500 1.138347 0.0 1.0 1.5 2.25 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 13.0 45099.692308 154.044033 44772.0 45109.00 45147.0 45183.0 45281.0
    afl 82800 16.0 44904.250000 230.450718 44411.0 44746.25 44988.0 45077.5 45229.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl_no_trim.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 8.0 20.500000 2.777460 17.0 17.75 21.5 22.25 24.0
    afl 82800 17.0 13.235294 3.579969 4.0 13.00 14.0 15.00 16.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 8.0 39562.750000 383.351666 39162.0 39262.5 39412.0 39908.25 40129.0
    afl 82800 17.0 38753.764706 137.002523 38510.0 38692.0 38767.0 38850.00 38981.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_trim 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_no_trim 82800 18.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

quickjs_eval-2020-01-05 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 0.933333 0.258199 0.0 1.0 1.0 1.0 1.0
    afl_no_trim 82800 17.0 0.941176 0.242536 0.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 15386.733333 575.196919 14398.0 14972.0 15526.0 15670.5 16304.0
    afl_no_trim 82800 17.0 15178.294118 543.947696 13920.0 14827.0 15185.0 15557.0 15946.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 16.0 18.0000 2.875181 12.0 17.0 17.5 20.0 22.0
    afl 82800 16.0 16.5625 2.920474 10.0 15.0 16.5 19.0 21.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2544.8125 112.443301 2402.0 2422.25 2600.5 2649.75 2661.0
    afl_no_trim 82800 16.0 2557.0000 120.044992 2403.0 2448.00 2593.5 2636.75 2818.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.157895 0.374634 0.0 0.0 0.0 0.0 1.0
    afl_no_trim 82800 17.0 0.058824 0.242536 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 17.0 13649.764706 301.721214 13380.0 13503.0 13556.0 13581.0 14298.0
    afl 82800 19.0 13621.789474 384.972521 12843.0 13473.0 13519.0 13601.5 14358.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 3.533333 1.302013 0.0 3.0 4.0 4.00 6.0
    afl_no_trim 82800 14.0 3.785714 0.892582 3.0 3.0 3.5 4.75 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 14.0 484591.928571 11680.702294 461014.0 478805.5 487070.5 491759.25 501784.0
    afl 82800 15.0 475860.533333 16351.675755 437403.0 472753.5 479164.0 487987.00 493402.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 0.300000 0.483046 0.0 0.0 0.0 0.75 1.0
    afl_no_trim 82800 14.0 0.142857 0.363137 0.0 0.0 0.0 0.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_trim 82800 14.0 9864.928571 286.590693 9384.0 9594.25 10007.0 10046.5 10124.0
    afl 82800 10.0 9761.000000 425.564462 8780.0 9623.50 9930.5 10069.5 10109.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 34571b40ef58a9a9a016111322adfde68f019249

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 34571b40ef58a9a9a016111322adfde68f019249
# Download the internal config file.
curl https://storage.googleapis.com/trimming/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data