FuzzBench: culling report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_no_favfactor 90.14
afl 87.00
afl_no_favored 81.94
By avg. rank
average rank
fuzzer
afl_no_favfactor 1.84
afl 2.00
afl_no_favored 2.16
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_no_favfactor afl afl_no_favored
    FuzzerMedian 95.70 96.42 94.70
    FuzzerMean 95.46 94.96 94.18
    arrow_parquet-arrow-fuzz 97.35 96.60 93.27
    aspell_aspell_fuzzer 99.77 99.75 99.66
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan nan
    file_magic_fuzzer 99.06 96.61 93.53
    grok_grk_decompress_fuzzer 93.36 93.32 92.34
    libarchive_libarchive_fuzzer 89.79 84.74 89.83
    libgit2_objects_fuzzer 99.71 99.67 99.75
    libhevc_hevc_dec_fuzzer 90.94 83.92 81.32
    libhtp_fuzz_htp 99.92 99.91 99.89
    libxml2_libxml2_xml_reader_for_file_fuzzer 90.84 88.88 87.76
    matio_matio_fuzzer 97.41 97.66 95.75
    mruby-2018-05-23 94.08 91.84 85.22
    muparser_set_eval_fuzzer 97.70 97.70 97.70
    njs_njs_process_script_fuzzer 95.55 96.23 95.17
    openh264_decoder_fuzzer 99.73 99.65 97.73
    php_php-fuzz-execute 90.79 90.78 97.42
    php_php-fuzz-parser-2020-07-25 99.52 99.38 98.81
    poppler_pdf_fuzzer 98.76 98.77 98.98
    proj4_standard_fuzzer 100.00 100.00 100.00
    quickjs_eval-2020-01-05 93.28 95.23 89.10
    stb_stbi_read_fuzzer 92.19 97.65 90.78
    systemd_fuzz-varlink 88.48 88.48 94.24
    usrsctp_fuzzer_connect 94.30 94.15 93.91
    wireshark_fuzzshark_ip 95.85 95.32 96.08
    zstd_stream_decompress 92.75 92.76 92.18
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_no_favfactor afl afl_no_favored
    FuzzerMedian 63.33 56.82 52.50
    FuzzerMean 53.37 51.23 49.16
    arrow_parquet-arrow-fuzz 86.46 85.42 79.17
    aspell_aspell_fuzzer 100.00 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer 60.00 70.00 55.00
    file_magic_fuzzer 0.00 0.00 0.00
    grok_grk_decompress_fuzzer 14.29 14.29 14.29
    libarchive_libarchive_fuzzer nan nan nan
    libgit2_objects_fuzzer 66.67 33.33 33.33
    libhevc_hevc_dec_fuzzer 87.50 87.50 84.38
    libhtp_fuzz_htp 66.67 66.67 58.33
    libxml2_libxml2_xml_reader_for_file_fuzzer 15.38 15.38 15.38
    matio_matio_fuzzer 71.43 71.43 78.57
    mruby-2018-05-23 50.00 33.33 0.00
    muparser_set_eval_fuzzer nan nan nan
    njs_njs_process_script_fuzzer 20.00 20.00 40.00
    openh264_decoder_fuzzer 66.67 50.00 33.33
    php_php-fuzz-execute 33.33 33.33 66.67
    php_php-fuzz-parser-2020-07-25 25.00 37.50 12.50
    poppler_pdf_fuzzer 75.00 63.64 79.55
    proj4_standard_fuzzer nan nan nan
    quickjs_eval-2020-01-05 100.00 100.00 100.00
    stb_stbi_read_fuzzer 85.71 78.57 80.95
    systemd_fuzz-varlink 100.00 100.00 100.00
    usrsctp_fuzzer_connect 0.00 0.00 0.00
    wireshark_fuzzshark_ip 50.00 66.67 50.00
    zstd_stream_decompress 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl_no_favfactor afl afl_no_favored
    FuzzerSum 346 282 277 264
    arrow_parquet-arrow-fuzz 71 65 65 53
    aspell_aspell_fuzzer 2 2 2 2
    ffmpeg_ffmpeg_demuxer_fuzzer 30 20 25 22
    file_magic_fuzzer 1 1 1 1
    grok_grk_decompress_fuzzer 9 7 8 2
    libarchive_libarchive_fuzzer 0 0 0 0
    libgit2_objects_fuzzer 3 3 3 3
    libhevc_hevc_dec_fuzzer 34 34 34 33
    libhtp_fuzz_htp 6 6 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 17 15 13 11
    matio_matio_fuzzer 22 19 15 18
    mruby-2018-05-23 9 8 8 2
    muparser_set_eval_fuzzer 0 0 0 0
    njs_njs_process_script_fuzzer 6 5 3 5
    openh264_decoder_fuzzer 7 7 5 3
    php_php-fuzz-execute 19 5 1 16
    php_php-fuzz-parser-2020-07-25 10 5 10 6
    poppler_pdf_fuzzer 59 44 41 45
    proj4_standard_fuzzer 0 0 0 0
    quickjs_eval-2020-01-05 1 1 1 1
    stb_stbi_read_fuzzer 25 25 25 25
    systemd_fuzz-varlink 1 1 1 1
    usrsctp_fuzzer_connect 1 1 1 1
    wireshark_fuzzshark_ip 12 7 9 8
    zstd_stream_decompress 1 1 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 16.0 38.000000 9.366607 12.0 37.75 41.5 43.25 46.0
    afl 82800 11.0 38.454545 9.709414 10.0 40.00 41.0 42.50 44.0
    afl_no_favored 82800 12.0 36.666667 6.005048 21.0 36.00 38.0 39.50 44.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 16.0 5132.875000 72.298340 5008.0 5096.50 5131.5 5171.25 5271.0
    afl 82800 11.0 5099.272727 56.505028 5022.0 5063.00 5092.0 5120.50 5233.0
    afl_no_favored 82800 12.0 4919.916667 28.053385 4887.0 4901.75 4916.0 4931.25 4993.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_no_favfactor 82800 19.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_no_favored 82800 16.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 19.0 5528.157895 18.580944 5491.0 5528.0 5535.0 5538.00 5547.0
    afl 82800 16.0 5524.750000 20.381364 5493.0 5499.0 5534.0 5538.00 5548.0
    afl_no_favored 82800 16.0 5515.937500 20.410679 5492.0 5494.0 5529.0 5532.25 5544.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 6.823529 1.704233 4.0 5.0 7.0 8.00 10.0
    afl_no_favfactor 82800 15.0 6.666667 2.193063 3.0 5.5 6.0 8.00 10.0
    afl_no_favored 82800 14.0 5.857143 1.511858 3.0 5.0 5.5 6.75 9.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favfactor 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favored 82800 14.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 11.0 0.272727 0.467099 0.0 0.0 0.0 0.5 1.0
    afl_no_favfactor 82800 18.0 0.388889 0.501631 0.0 0.0 0.0 1.0 1.0
    afl_no_favored 82800 13.0 0.230769 0.438529 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 18.0 3789.888889 49.773474 3607.0 3778.75 3800.0 3814.5 3836.0
    afl 82800 11.0 3736.090909 62.221306 3666.0 3684.00 3706.0 3795.0 3822.0
    afl_no_favored 82800 13.0 3369.384615 397.750663 2949.0 2967.00 3588.0 3715.0 3820.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 1.333333 1.495090 0.0 1.0 1.0 1.00 7.0
    afl_no_favfactor 82800 18.0 1.555556 1.381484 0.0 1.0 1.0 1.75 6.0
    afl_no_favored 82800 15.0 0.933333 0.457738 0.0 1.0 1.0 1.00 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 18.0 8876.722222 184.503086 8753.0 8797.25 8820.0 8838.25 9423.0
    afl 82800 18.0 8853.944444 150.235295 8779.0 8802.25 8816.0 8844.50 9447.0
    afl_no_favored 82800 15.0 8712.400000 65.701489 8598.0 8665.00 8723.0 8745.00 8835.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favfactor 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favored 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 19.0 6094.631579 338.652757 5239.0 5933.50 6000.0 6306.00 6679.0
    afl_no_favfactor 82800 18.0 5989.555556 381.456994 5307.0 5770.25 5997.0 6294.75 6576.0
    afl 82800 12.0 5733.083333 444.325323 5187.0 5379.25 5660.0 6118.00 6401.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 17.0 1.705882 0.771744 1.0 1.0 2.0 2.0 3.0
    afl 82800 19.0 1.421053 0.606977 1.0 1.0 1.0 2.0 3.0
    afl_no_favored 82800 17.0 1.470588 0.514496 1.0 1.0 1.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 17.0 2439.764706 1.888432 2438.0 2439.0 2440.0 2440.0 2446.0
    afl_no_favfactor 82800 17.0 2439.000000 1.457738 2438.0 2438.0 2439.0 2439.0 2444.0
    afl 82800 19.0 2438.842105 1.424514 2438.0 2438.0 2438.0 2439.0 2444.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 23.842105 10.264627 1.0 26.00 28.0 29.00 31.0
    afl_no_favfactor 82800 18.0 25.111111 8.956686 1.0 26.25 28.0 29.00 32.0
    afl_no_favored 82800 18.0 22.666667 10.278476 1.0 23.25 27.0 28.75 31.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 18.0 9277.833333 3136.909780 953.0 9427.00 10333.0 11043.25 11362.0
    afl 82800 19.0 8500.000000 3467.060346 953.0 8749.00 9535.0 10653.50 11311.0
    afl_no_favored 82800 18.0 8204.777778 3435.381502 953.0 8683.75 9239.5 10012.75 11247.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3.470588 1.067570 1.0 3.00 4.0 4.0 5.0
    afl_no_favfactor 82800 18.0 3.833333 1.043185 2.0 3.00 4.0 4.0 6.0
    afl_no_favored 82800 16.0 3.687500 1.302242 2.0 2.75 3.5 5.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 18.0 6577.000000 2.950573 6572.0 6576.25 6577.5 6578.0 6583.0
    afl 82800 17.0 6576.588235 2.399448 6570.0 6576.00 6577.0 6578.0 6580.0
    afl_no_favored 82800 16.0 6576.500000 1.788854 6574.0 6575.75 6576.0 6577.0 6581.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 2.785714 2.939874 2.0 2.0 2.0 2.00 13.0
    afl_no_favfactor 82800 12.0 4.583333 3.895413 2.0 2.0 2.0 8.25 11.0
    afl_no_favored 82800 16.0 2.500000 2.000000 2.0 2.0 2.0 2.00 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 12.0 21124.500000 991.095124 19525.0 20473.0 20845.5 22079.25 22947.0
    afl 82800 14.0 20468.142857 679.825867 19151.0 20271.0 20396.0 20678.75 22267.0
    afl_no_favored 82800 16.0 20136.062500 588.412040 19073.0 19792.0 20138.0 20505.75 21071.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 17.0 10.529412 1.462773 8.0 10.0 11.0 11.0 14.0
    afl 82800 18.0 10.055556 1.392088 7.0 9.0 10.0 11.0 12.0
    afl_no_favfactor 82800 19.0 9.947368 0.848115 8.0 9.5 10.0 10.5 11.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 2751.666667 28.354894 2708.0 2730.0 2755.0 2764.0 2817.0
    afl_no_favfactor 82800 19.0 2754.421053 31.172131 2705.0 2733.5 2748.0 2761.5 2821.0
    afl_no_favored 82800 17.0 2701.588235 19.513005 2656.0 2694.0 2701.0 2715.0 2738.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby-2018-05-23 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 63000 9.0 3.222222 1.563472 1.0 3.00 3.0 4.00 6.0
    afl 63000 10.0 2.200000 1.229273 1.0 1.25 2.0 2.75 5.0
    afl_no_favored 63000 8.0 0.375000 0.517549 0.0 0.00 0.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 63000 9.0 16470.333333 414.459588 15963.0 16284.00 16380.0 16481.00 17259.0
    afl 63000 10.0 16054.600000 297.550444 15708.0 15877.25 15989.0 16156.75 16772.0
    afl_no_favored 63000 8.0 14914.875000 336.329531 14516.0 14691.00 14837.5 15097.00 15456.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favfactor 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favored 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 935.333333 1.414214 930.0 935.0 936.0 936.0 936.0
    afl_no_favfactor 82800 18.0 937.500000 6.080344 933.0 935.0 936.0 936.0 958.0
    afl_no_favored 82800 19.0 935.789474 2.572913 930.0 935.0 936.0 936.0 944.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 19.0 1.684211 0.820070 1.0 1.0 2.0 2.0 4.0
    afl 82800 18.0 1.444444 0.615699 1.0 1.0 1.0 2.0 3.0
    afl_no_favfactor 82800 17.0 1.823529 1.185079 1.0 1.0 1.0 2.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 7261.722222 101.293868 7092.0 7190.25 7271.0 7345.5 7400.0
    afl_no_favfactor 82800 17.0 7230.117647 130.026383 7012.0 7122.00 7220.0 7339.0 7452.0
    afl_no_favored 82800 19.0 7246.473684 156.973235 7058.0 7147.00 7191.0 7359.0 7556.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 14.0 3.714286 1.266647 2.0 3.0 4.0 4.75 6.0
    afl 82800 13.0 3.000000 1.154701 1.0 2.0 3.0 4.00 5.0
    afl_no_favored 82800 15.0 1.666667 0.617213 1.0 1.0 2.0 2.00 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 14.0 14307.500000 69.951357 14129.0 14261.5 14339.5 14359.25 14379.0
    afl 82800 13.0 14306.230769 47.000273 14240.0 14268.0 14328.0 14338.00 14375.0
    afl_no_favored 82800 15.0 14045.333333 41.628745 13980.0 14014.5 14053.0 14077.00 14112.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 15.0 2.000000 0.845154 1.0 1.0 2.0 3.0 3.0
    afl 82800 14.0 0.928571 0.267261 0.0 1.0 1.0 1.0 1.0
    afl_no_favfactor 82800 14.0 1.142857 0.363137 1.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 15.0 192415.733333 2294.435364 189453.0 190737.50 191876.0 193597.50 196967.0
    afl_no_favfactor 82800 14.0 178219.142857 2950.273021 172270.0 178062.00 178835.5 179441.75 183845.0
    afl 82800 14.0 177239.142857 5819.063867 160901.0 174667.75 178808.0 180493.25 184957.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 1.687500 1.138347 0.0 1.0 1.5 2.25 4.0
    afl_no_favfactor 82800 12.0 0.833333 0.937437 0.0 0.0 1.0 1.00 3.0
    afl_no_favored 82800 14.0 0.857143 1.099450 0.0 0.0 0.5 1.00 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 12.0 44989.833333 242.372678 44615.0 44717.75 45051.5 45186.25 45270.0
    afl 82800 16.0 44904.250000 230.450718 44411.0 44746.25 44988.0 45077.50 45229.0
    afl_no_favored 82800 14.0 44647.928571 257.494874 44082.0 44468.25 44732.5 44823.25 44980.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 16.0 17.187500 3.506542 11.0 14.75 17.5 20.0 22.0
    afl_no_favfactor 82800 14.0 16.357143 1.945691 13.0 15.00 16.5 18.0 20.0
    afl 82800 17.0 13.235294 3.579969 4.0 13.00 14.0 15.0 16.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 16.0 38862.937500 126.826637 38570.0 38805.5 38849.5 38977.50 39086.0
    afl 82800 17.0 38753.764706 137.002523 38510.0 38692.0 38767.0 38850.00 38981.0
    afl_no_favfactor 82800 14.0 38772.571429 170.987050 38531.0 38665.0 38762.5 38796.75 39249.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favfactor 82800 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_no_favored 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_no_favfactor 82800 12.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_no_favored 82800 16.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

quickjs_eval-2020-01-05 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 0.933333 0.258199 0.0 1.0 1.0 1.0 1.0
    afl_no_favfactor 82800 19.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    afl_no_favored 82800 15.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 15386.733333 575.196919 14398.0 14972.0 15526.0 15670.5 16304.0
    afl_no_favfactor 82800 19.0 15101.000000 478.309059 13853.0 14885.0 15209.0 15443.5 15728.0
    afl_no_favored 82800 15.0 14561.133333 373.336667 14046.0 14227.5 14527.0 14855.5 15084.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 17.0 17.882353 1.798692 14.0 17.0 18.0 19.00 21.0
    afl_no_favored 82800 16.0 16.750000 2.265686 14.0 15.0 17.0 18.25 21.0
    afl 82800 16.0 16.562500 2.920474 10.0 15.0 16.5 19.00 21.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2544.812500 112.443301 2402.0 2422.25 2600.5 2649.75 2661.0
    afl_no_favfactor 82800 17.0 2517.058824 100.178260 2403.0 2435.00 2455.0 2619.00 2660.0
    afl_no_favored 82800 16.0 2451.375000 76.861672 2396.0 2410.00 2417.5 2449.00 2663.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.157895 0.374634 0.0 0.0 0.0 0.00 1.0
    afl_no_favfactor 82800 18.0 0.277778 0.460889 0.0 0.0 0.0 0.75 1.0
    afl_no_favored 82800 19.0 0.105263 0.315302 0.0 0.0 0.0 0.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favfactor 82800 18.0 13575.055556 328.040792 12868.0 13440.75 13540.0 13609.75 14359.0
    afl 82800 19.0 13621.789474 384.972521 12843.0 13473.00 13519.0 13601.50 14358.0
    afl_no_favored 82800 19.0 13539.526316 200.566799 13341.0 13419.50 13485.0 13569.00 14272.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 3.533333 1.302013 0.0 3.0 4.0 4.00 6.0
    afl_no_favfactor 82800 16.0 2.875000 1.204159 0.0 3.0 3.0 3.25 4.0
    afl_no_favored 82800 11.0 3.272727 1.555050 0.0 3.0 3.0 4.00 6.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_no_favored 82800 11.0 484167.636364 4650.813634 476972.0 481270.50 482995.0 487166.0 491604.0
    afl_no_favfactor 82800 16.0 477713.187500 14823.852842 429576.0 477768.25 481815.0 482823.0 497499.0
    afl 82800 15.0 475860.533333 16351.675755 437403.0 472753.50 479164.0 487987.0 493402.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 0.300000 0.483046 0.0 0.0 0.0 0.75 1.0
    afl_no_favfactor 82800 12.0 0.083333 0.288675 0.0 0.0 0.0 0.00 1.0
    afl_no_favored 82800 14.0 0.357143 0.497245 0.0 0.0 0.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 9761.000000 425.564462 8780.0 9623.50 9930.5 10069.50 10109.0
    afl_no_favfactor 82800 12.0 9881.083333 400.723541 9393.0 9479.75 9930.0 10105.50 10706.0
    afl_no_favored 82800 14.0 9760.571429 320.453216 9027.0 9733.00 9868.5 9962.25 10111.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 34571b40ef58a9a9a016111322adfde68f019249

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 34571b40ef58a9a9a016111322adfde68f019249
# Download the internal config file.
curl https://storage.googleapis.com/culling/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data