FuzzBench: score report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_score_random 93.52
afl 90.51
afl_score_max 88.88
afl_score_no_novel_prioritization 87.63
afl_score_min 81.06
By avg. rank
average rank
fuzzer
afl_score_random 2.82
afl_score_max 2.92
afl 3.02
afl_score_no_novel_prioritization 3.10
afl_score_min 3.14
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_score_random afl_score_no_novel_prioritization afl afl_score_max afl_score_min
    FuzzerMedian 96.43 96.74 96.87 95.21 94.82
    FuzzerMean 95.41 94.93 94.89 94.68 93.46
    arrow_parquet-arrow-fuzz 96.82 96.86 97.31 97.70 94.39
    aspell_aspell_fuzzer 99.73 99.69 99.69 99.77 99.71
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan nan nan nan
    file_magic_fuzzer 96.47 98.49 96.44 95.50 95.71
    grok_grk_decompress_fuzzer 95.77 92.73 92.73 93.55 92.53
    libarchive_libarchive_fuzzer 88.80 88.29 85.22 90.14 83.84
    libgit2_objects_fuzzer 99.75 99.80 99.75 99.80 99.75
    libhevc_hevc_dec_fuzzer 85.47 80.72 78.87 85.95 76.58
    libhtp_fuzz_htp 99.98 99.93 99.95 99.97 99.91
    libxml2_libxml2_xml_reader_for_file_fuzzer 92.44 92.61 90.48 91.73 90.37
    matio_matio_fuzzer 96.66 97.62 97.80 94.57 95.07
    mruby-2018-05-23 96.39 94.65 95.08 95.04 92.13
    muparser_set_eval_fuzzer 97.40 97.35 97.40 97.29 97.40
    njs_njs_process_script_fuzzer 95.73 96.63 97.64 95.38 94.57
    openh264_decoder_fuzzer 99.78 99.60 99.67 99.83 99.39
    php_php-fuzz-execute 93.33 90.61 91.33 97.50 81.12
    php_php-fuzz-parser-2020-07-25 98.57 99.45 99.32 99.40 96.62
    poppler_pdf_fuzzer 98.09 97.87 97.96 98.27 98.62
    proj4_standard_fuzzer 100.00 100.00 100.00 100.00 100.00
    quickjs_eval-2020-01-05 89.29 92.06 92.64 87.23 92.47
    stb_stbi_read_fuzzer 91.10 86.86 93.34 86.86 86.61
    systemd_fuzz-varlink 100.00 88.48 88.48 88.48 100.00
    usrsctp_fuzzer_connect 93.95 94.38 94.06 94.50 94.11
    wireshark_fuzzshark_ip 92.57 96.56 94.43 88.92 85.21
    zstd_stream_decompress 91.65 97.16 97.74 94.94 96.92
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_score_random afl afl_score_max afl_score_no_novel_prioritization afl_score_min
    FuzzerMedian 50.00 50.00 50.00 50.00 41.67
    FuzzerMean 48.87 47.83 46.95 46.28 44.06
    arrow_parquet-arrow-fuzz 91.49 87.23 90.43 89.36 81.91
    aspell_aspell_fuzzer 100.00 100.00 100.00 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer 41.67 58.33 58.33 50.00 25.00
    file_magic_fuzzer 0.00 0.00 0.00 0.00 0.00
    grok_grk_decompress_fuzzer 31.25 12.50 25.00 12.50 12.50
    libarchive_libarchive_fuzzer nan nan nan nan nan
    libgit2_objects_fuzzer 33.33 33.33 33.33 33.33 33.33
    libhevc_hevc_dec_fuzzer 89.06 87.50 90.62 84.38 76.56
    libhtp_fuzz_htp 80.00 80.00 80.00 60.00 80.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 15.38 15.38 15.38 15.38 15.38
    matio_matio_fuzzer 76.92 76.92 76.92 76.92 76.92
    mruby-2018-05-23 60.00 60.00 20.00 60.00 20.00
    muparser_set_eval_fuzzer nan nan nan nan nan
    njs_njs_process_script_fuzzer 33.33 33.33 33.33 33.33 33.33
    openh264_decoder_fuzzer 50.00 50.00 50.00 33.33 58.33
    php_php-fuzz-execute 50.00 25.00 25.00 25.00 25.00
    php_php-fuzz-parser-2020-07-25 25.00 37.50 25.00 37.50 0.00
    poppler_pdf_fuzzer 71.43 66.67 66.67 76.19 83.33
    proj4_standard_fuzzer nan nan nan nan nan
    quickjs_eval-2020-01-05 50.00 50.00 50.00 50.00 50.00
    stb_stbi_read_fuzzer 76.19 78.57 76.19 80.95 80.95
    systemd_fuzz-varlink 50.00 50.00 50.00 50.00 50.00
    usrsctp_fuzzer_connect 0.00 0.00 0.00 0.00 0.00
    wireshark_fuzzshark_ip 50.00 50.00 66.67 50.00 66.67
    zstd_stream_decompress 0.00 0.00 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl afl_score_random afl_score_max afl_score_no_novel_prioritization afl_score_min
    FuzzerSum 360 277 271 266 266 244
    arrow_parquet-arrow-fuzz 74 65 60 65 60 59
    aspell_aspell_fuzzer 2 2 2 2 2 2
    ffmpeg_ffmpeg_demuxer_fuzzer 35 25 20 19 22 10
    file_magic_fuzzer 1 1 1 1 1 1
    grok_grk_decompress_fuzzer 11 8 8 9 3 5
    libarchive_libarchive_fuzzer 0 0 0 0 0 0
    libgit2_objects_fuzzer 3 3 2 2 2 2
    libhevc_hevc_dec_fuzzer 34 34 34 34 34 34
    libhtp_fuzz_htp 5 5 5 5 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 17 13 12 12 13 7
    matio_matio_fuzzer 25 15 19 16 17 20
    mruby-2018-05-23 8 8 8 4 7 5
    muparser_set_eval_fuzzer 0 0 0 0 0 0
    njs_njs_process_script_fuzzer 4 3 3 2 4 3
    openh264_decoder_fuzzer 6 5 5 6 5 6
    php_php-fuzz-execute 19 1 9 9 7 5
    php_php-fuzz-parser-2020-07-25 11 10 7 6 7 3
    poppler_pdf_fuzzer 61 41 43 40 40 37
    proj4_standard_fuzzer 0 0 0 0 0 0
    quickjs_eval-2020-01-05 3 1 1 1 1 3
    stb_stbi_read_fuzzer 26 25 23 25 26 24
    systemd_fuzz-varlink 2 1 2 1 1 1
    usrsctp_fuzzer_connect 1 1 0 0 1 1
    wireshark_fuzzshark_ip 10 9 6 6 7 10
    zstd_stream_decompress 2 1 1 1 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 82800 8.0 42.375000 1.767767 39.0 41.75 43.0 43.00 45.0
    afl_score_max 82800 14.0 42.214286 2.833279 37.0 40.25 42.5 44.00 47.0
    afl_score_no_novel_prioritization 82800 12.0 41.416667 2.998737 34.0 40.50 42.0 43.25 45.0
    afl 82800 11.0 38.454545 9.709414 10.0 40.00 41.0 42.50 44.0
    afl_score_min 82800 14.0 38.785714 1.672335 35.0 38.00 38.5 39.75 42.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 14.0 5105.071429 38.210881 5022.0 5089.00 5112.5 5135.25 5152.0
    afl 82800 11.0 5099.272727 56.505028 5022.0 5063.00 5092.0 5120.50 5233.0
    afl_score_no_novel_prioritization 82800 12.0 5061.250000 42.229083 4994.0 5034.00 5068.5 5081.75 5136.0
    afl_score_random 82800 8.0 5088.625000 45.204101 5044.0 5055.75 5066.5 5127.75 5162.0
    afl_score_min 82800 14.0 4939.357143 26.187511 4890.0 4923.75 4939.5 4945.50 4986.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_score_max 82800 18.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_score_min 82800 19.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_score_no_novel_prioritization 82800 17.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_score_random 82800 18.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 18.0 5539.444444 5.316149 5533.0 5536.0 5538.0 5544.0 5550.0
    afl_score_random 82800 18.0 5537.055556 7.099894 5517.0 5534.0 5536.0 5540.5 5551.0
    afl_score_min 82800 19.0 5535.736842 3.106304 5532.0 5533.0 5535.0 5537.0 5543.0
    afl 82800 16.0 5524.750000 20.381364 5493.0 5499.0 5534.0 5538.0 5548.0
    afl_score_no_novel_prioritization 82800 17.0 5529.294118 15.975625 5495.0 5532.0 5534.0 5535.0 5548.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 6.823529 1.704233 4.0 5.00 7.0 8.0 10.0
    afl_score_max 82800 15.0 7.066667 2.218966 3.0 5.00 7.0 8.0 12.0
    afl_score_no_novel_prioritization 82800 19.0 5.421053 1.643701 2.0 5.00 6.0 6.0 8.0
    afl_score_random 82800 16.0 5.750000 1.732051 4.0 4.75 5.0 7.0 9.0
    afl_score_min 82800 19.0 3.263158 0.871914 2.0 3.00 3.0 4.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_max 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_min 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_no_novel_prioritization 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_random 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 11.0 0.272727 0.467099 0.0 0.0 0.0 0.5 1.0
    afl_score_max 82800 19.0 0.157895 0.374634 0.0 0.0 0.0 0.0 1.0
    afl_score_min 82800 16.0 0.187500 0.403113 0.0 0.0 0.0 0.0 1.0
    afl_score_no_novel_prioritization 82800 19.0 0.210526 0.418854 0.0 0.0 0.0 0.0 1.0
    afl_score_random 82800 18.0 0.111111 0.323381 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_no_novel_prioritization 82800 19.0 3735.157895 193.584395 2967.0 3733.00 3785.0 3823.00 3843.0
    afl_score_random 82800 18.0 3711.166667 70.559320 3603.0 3652.25 3707.5 3771.00 3828.0
    afl 82800 11.0 3736.090909 62.221306 3666.0 3684.00 3706.0 3795.00 3822.0
    afl_score_min 82800 16.0 3603.562500 231.411889 2960.0 3590.25 3678.0 3733.75 3798.0
    afl_score_max 82800 19.0 3654.736842 175.698239 2977.0 3648.00 3670.0 3706.50 3822.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 82800 6.0 3.000000 2.280351 1.0 1.0 2.5 4.75 6.0
    afl_score_max 82800 16.0 2.937500 2.080665 1.0 2.0 2.0 2.75 8.0
    afl 82800 18.0 1.333333 1.495090 0.0 1.0 1.0 1.00 7.0
    afl_score_min 82800 17.0 1.411765 0.795206 1.0 1.0 1.0 2.00 4.0
    afl_score_no_novel_prioritization 82800 18.0 1.111111 0.471405 0.0 1.0 1.0 1.00 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 82800 6.0 9140.833333 333.698317 8821.0 8841.00 9104.5 9446.75 9501.0
    afl_score_max 82800 16.0 9041.437500 275.000841 8813.0 8876.25 8894.0 9199.25 9507.0
    afl 82800 18.0 8853.944444 150.235295 8779.0 8802.25 8816.0 8844.50 9447.0
    afl_score_no_novel_prioritization 82800 18.0 8821.500000 40.594479 8752.0 8799.00 8815.5 8830.75 8911.0
    afl_score_min 82800 17.0 8836.705882 141.706018 8763.0 8781.00 8797.0 8818.00 9369.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_max 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_min 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_no_novel_prioritization 82800 14.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_random 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 19.0 5927.473684 462.586132 5135.0 5675.50 5987.0 6299.00 6642.0
    afl_score_random 82800 15.0 5959.133333 318.325949 5235.0 5800.00 5898.0 6208.00 6429.0
    afl_score_no_novel_prioritization 82800 14.0 5877.071429 391.643768 5130.0 5760.75 5864.0 5954.50 6543.0
    afl 82800 12.0 5733.083333 444.325323 5187.0 5379.25 5660.0 6118.00 6401.0
    afl_score_min 82800 18.0 5539.222222 311.610533 5028.0 5357.25 5568.5 5744.75 6068.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 1.421053 0.606977 1.0 1.0 1.0 2.0 3.0
    afl_score_max 82800 19.0 1.157895 0.374634 1.0 1.0 1.0 1.0 2.0
    afl_score_min 82800 17.0 1.058824 0.242536 1.0 1.0 1.0 1.0 2.0
    afl_score_no_novel_prioritization 82800 20.0 1.350000 0.489360 1.0 1.0 1.0 2.0 2.0
    afl_score_random 82800 14.0 1.142857 0.363137 1.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 19.0 2438.789474 1.182227 2438.0 2438.0 2439.0 2439.0 2443.0
    afl_score_no_novel_prioritization 82800 20.0 2438.650000 0.489360 2438.0 2438.0 2439.0 2439.0 2439.0
    afl 82800 19.0 2438.842105 1.424514 2438.0 2438.0 2438.0 2439.0 2444.0
    afl_score_min 82800 17.0 2438.294118 1.263166 2437.0 2438.0 2438.0 2438.0 2443.0
    afl_score_random 82800 14.0 2438.428571 0.513553 2438.0 2438.0 2438.0 2439.0 2439.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 18.0 25.944444 9.136045 1.0 28.00 29.0 30.00 31.0
    afl_score_random 82800 14.0 25.357143 9.270359 2.0 27.00 28.5 30.00 32.0
    afl 82800 19.0 23.842105 10.264627 1.0 26.00 28.0 29.00 31.0
    afl_score_no_novel_prioritization 82800 19.0 19.894737 13.353712 1.0 1.00 27.0 30.00 32.0
    afl_score_min 82800 14.0 20.285714 10.622390 1.0 22.25 24.5 26.75 29.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 18.0 9519.444444 3222.626317 953.0 9571.0 10391.0 11017.25 12090.0
    afl_score_random 82800 14.0 9099.642857 3454.987668 1022.0 9409.0 10333.5 11145.50 11397.0
    afl_score_no_novel_prioritization 82800 19.0 7394.578947 4535.457656 953.0 953.5 9759.0 10610.00 11378.0
    afl 82800 19.0 8500.000000 3467.060346 953.0 8749.0 9535.0 10653.50 11311.0
    afl_score_min 82800 14.0 7761.071429 3727.762011 953.0 9030.5 9258.5 9603.25 10903.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3.470588 1.067570 1.0 3.0 4.0 4.0 5.0
    afl_score_max 82800 19.0 4.000000 0.745356 2.0 4.0 4.0 4.0 5.0
    afl_score_min 82800 9.0 3.777778 1.563472 0.0 4.0 4.0 5.0 5.0
    afl_score_random 82800 10.0 3.400000 1.173788 1.0 3.0 4.0 4.0 5.0
    afl_score_no_novel_prioritization 82800 16.0 2.937500 1.436141 0.0 2.0 3.0 4.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 82800 10.0 6578.200000 1.751190 6574.0 6578.0 6578.5 6579.0 6580.0
    afl_score_max 82800 19.0 6577.473684 1.711673 6574.0 6576.5 6578.0 6578.5 6580.0
    afl 82800 17.0 6576.588235 2.399448 6570.0 6576.0 6577.0 6578.0 6580.0
    afl_score_no_novel_prioritization 82800 16.0 6576.250000 2.144761 6572.0 6575.0 6575.5 6578.0 6580.0
    afl_score_min 82800 9.0 6574.000000 3.708099 6568.0 6572.0 6574.0 6577.0 6578.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 2.785714 2.939874 2.0 2.0 2.0 2.00 13.0
    afl_score_max 82800 11.0 2.727273 2.760105 1.0 2.0 2.0 2.00 11.0
    afl_score_min 82800 14.0 2.285714 1.069045 2.0 2.0 2.0 2.00 6.0
    afl_score_no_novel_prioritization 82800 11.0 2.727273 1.678744 2.0 2.0 2.0 2.00 7.0
    afl_score_random 82800 12.0 3.583333 2.937480 2.0 2.0 2.0 3.25 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_no_novel_prioritization 82800 11.0 20874.818182 545.087666 20051.0 20650.00 20878.0 20955.50 22257.0
    afl_score_random 82800 12.0 20917.250000 574.252103 20130.0 20471.75 20839.5 21304.00 21822.0
    afl_score_max 82800 11.0 20800.818182 772.340316 19637.0 20434.50 20679.0 20932.00 22543.0
    afl 82800 14.0 20468.142857 679.825867 19151.0 20271.00 20396.0 20678.75 22267.0
    afl_score_min 82800 14.0 20343.642857 454.926470 19343.0 20154.75 20372.5 20701.00 21046.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 10.055556 1.392088 7.0 9.0 10.0 11.0 12.0
    afl_score_max 82800 11.0 10.090909 1.375103 8.0 9.0 10.0 11.0 13.0
    afl_score_min 82800 19.0 9.578947 1.017393 8.0 9.0 10.0 10.0 11.0
    afl_score_no_novel_prioritization 82800 12.0 10.166667 1.337116 8.0 9.0 10.0 11.0 13.0
    afl_score_random 82800 9.0 10.333333 0.866025 9.0 10.0 10.0 11.0 12.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 2751.666667 28.354894 2708.0 2730.0 2755.0 2764.0 2817.0
    afl_score_no_novel_prioritization 82800 12.0 2752.750000 11.553236 2738.0 2744.0 2750.0 2762.0 2777.0
    afl_score_random 82800 9.0 2712.333333 18.754999 2683.0 2700.0 2723.0 2727.0 2733.0
    afl_score_min 82800 19.0 2678.368421 33.749421 2610.0 2662.5 2678.0 2704.0 2730.0
    afl_score_max 82800 11.0 2649.636364 44.782302 2573.0 2622.5 2664.0 2683.0 2696.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby-2018-05-23 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 81900 3.0 2.666667 0.577350 2.0 2.5 3.0 3.0 3.0
    afl_score_no_novel_prioritization 81900 13.0 2.923077 1.187542 1.0 2.0 3.0 4.0 5.0
    afl_score_random 81900 3.0 3.333333 0.577350 3.0 3.0 3.0 3.5 4.0
    afl_score_max 81900 11.0 0.727273 0.786245 0.0 0.0 1.0 1.0 2.0
    afl_score_min 81900 18.0 1.666667 0.907485 1.0 1.0 1.0 2.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 81900 3.0 16721.666667 256.718393 16544.0 16574.5 16605.0 16810.5 17016.0
    afl 81900 3.0 16247.000000 303.362819 15900.0 16139.5 16379.0 16420.5 16462.0
    afl_score_max 81900 11.0 16340.090909 362.099559 15848.0 16107.0 16373.0 16478.5 16956.0
    afl_score_no_novel_prioritization 81900 13.0 16358.923077 446.016342 15804.0 16005.0 16306.0 16570.0 17227.0
    afl_score_min 81900 18.0 15788.388889 227.821948 15329.0 15638.5 15870.5 15929.0 16121.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_max 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_min 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_no_novel_prioritization 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_random 82800 14.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 935.333333 1.414214 930.0 935.00 936.0 936.0 936.0
    afl_score_min 82800 18.0 936.777778 4.583289 935.0 935.00 936.0 936.0 955.0
    afl_score_random 82800 14.0 935.714286 0.468807 935.0 935.25 936.0 936.0 936.0
    afl_score_no_novel_prioritization 82800 18.0 935.500000 0.514496 935.0 935.00 935.5 936.0 936.0
    afl_score_max 82800 17.0 935.176471 0.951006 932.0 935.00 935.0 936.0 936.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 1.444444 0.615699 1.0 1.0 1.0 2.00 3.0
    afl_score_max 82800 15.0 1.466667 0.516398 1.0 1.0 1.0 2.00 2.0
    afl_score_min 82800 16.0 1.250000 0.577350 1.0 1.0 1.0 1.00 3.0
    afl_score_no_novel_prioritization 82800 17.0 1.411765 0.507300 1.0 1.0 1.0 2.00 2.0
    afl_score_random 82800 18.0 1.333333 0.594089 1.0 1.0 1.0 1.75 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 7261.722222 101.293868 7092.0 7190.25 7271.0 7345.50 7400.0
    afl_score_no_novel_prioritization 82800 17.0 7176.764706 122.122648 6913.0 7092.00 7196.0 7246.00 7417.0
    afl_score_random 82800 18.0 7159.277778 124.073890 6939.0 7068.75 7129.0 7259.25 7383.0
    afl_score_max 82800 15.0 7133.933333 122.643191 6910.0 7058.00 7103.0 7199.00 7361.0
    afl_score_min 82800 16.0 7102.250000 171.158990 6888.0 6998.25 7042.5 7181.00 7447.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_min 82800 16.0 3.437500 0.963933 2.0 3.0 3.5 4.0 5.0
    afl 82800 13.0 3.000000 1.154701 1.0 2.0 3.0 4.0 5.0
    afl_score_max 82800 9.0 3.000000 1.581139 1.0 2.0 3.0 4.0 6.0
    afl_score_random 82800 7.0 3.000000 1.527525 1.0 2.0 3.0 4.0 5.0
    afl_score_no_novel_prioritization 82800 9.0 2.666667 1.500000 1.0 2.0 2.0 3.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 9.0 14330.444444 47.552895 14258.0 14274.00 14351.0 14368.00 14370.0
    afl_score_random 82800 7.0 14339.285714 32.050555 14276.0 14332.50 14343.0 14358.50 14374.0
    afl 82800 13.0 14306.230769 47.000273 14240.0 14268.00 14328.0 14338.00 14375.0
    afl_score_no_novel_prioritization 82800 9.0 14293.444444 52.299883 14186.0 14254.00 14317.0 14331.00 14347.0
    afl_score_min 82800 16.0 14274.687500 62.691540 14140.0 14245.25 14287.5 14324.25 14350.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_random 81000 6.0 1.833333 0.752773 1.0 1.25 2.0 2.0 3.0
    afl 81000 14.0 0.928571 0.267261 0.0 1.00 1.0 1.0 1.0
    afl_score_max 81000 7.0 1.714286 1.112697 1.0 1.00 1.0 2.0 4.0
    afl_score_min 81000 9.0 1.111111 0.781736 0.0 1.00 1.0 1.0 3.0
    afl_score_no_novel_prioritization 81000 11.0 1.363636 0.674200 1.0 1.00 1.0 1.5 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 81000 7.0 191022.142857 2232.186180 189051.0 189548.00 190836.0 191260.00 195652.0
    afl_score_random 81000 6.0 182546.000000 2717.803598 178024.0 181875.75 182669.0 183776.50 186157.0
    afl 81000 14.0 177858.714286 6742.314901 160630.0 174602.25 178762.0 181139.25 189683.0
    afl_score_no_novel_prioritization 81000 11.0 176745.363636 2698.421549 172708.0 174777.00 177337.0 178174.50 181585.0
    afl_score_min 81000 9.0 156997.666667 3769.557700 150027.0 157875.00 158779.0 159207.00 159760.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 1.687500 1.138347 0.0 1.0 1.5 2.25 4.0
    afl_score_no_novel_prioritization 82800 8.0 1.625000 0.744024 1.0 1.0 1.5 2.00 3.0
    afl_score_max 82800 9.0 1.333333 0.866025 0.0 1.0 1.0 2.00 3.0
    afl_score_random 82800 7.0 1.142857 1.345185 0.0 0.0 1.0 2.00 3.0
    afl_score_min 82800 11.0 0.454545 0.522233 0.0 0.0 0.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_no_novel_prioritization 82800 8.0 45035.500000 76.761225 44921.0 44974.00 45047.5 45073.0 45154.0
    afl_score_max 82800 9.0 44962.777778 166.151872 44756.0 44776.00 45025.0 45113.0 45151.0
    afl 82800 16.0 44904.250000 230.450718 44411.0 44746.25 44988.0 45077.5 45229.0
    afl_score_random 82800 7.0 44609.285714 205.693716 44287.0 44491.50 44648.0 44721.5 44904.0
    afl_score_min 82800 11.0 43793.636364 362.485385 43289.0 43521.00 43768.0 44014.5 44521.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_min 82800 10.0 17.400000 2.836273 12.0 16.25 17.5 19.75 21.0
    afl_score_no_novel_prioritization 82800 10.0 15.500000 1.433721 13.0 14.25 16.0 16.75 17.0
    afl_score_random 82800 7.0 15.571429 1.812654 14.0 14.00 15.0 17.00 18.0
    afl 82800 17.0 13.235294 3.579969 4.0 13.00 14.0 15.00 16.0
    afl_score_max 82800 7.0 14.857143 2.193063 12.0 14.00 14.0 15.50 19.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_min 82800 10.0 39042.300000 271.420563 38649.0 38886.00 39030.0 39184.75 39576.0
    afl_score_max 82800 7.0 38911.285714 131.246224 38733.0 38824.00 38893.0 39012.50 39080.0
    afl_score_random 82800 7.0 38861.428571 162.976335 38666.0 38745.00 38821.0 38968.50 39116.0
    afl 82800 17.0 38753.764706 137.002523 38510.0 38692.00 38767.0 38850.00 38981.0
    afl_score_no_novel_prioritization 82800 10.0 38748.300000 84.417020 38650.0 38679.25 38733.0 38798.00 38896.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_max 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_min 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_no_novel_prioritization 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_score_random 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_score_max 82800 16.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_score_min 82800 19.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_score_no_novel_prioritization 82800 15.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0
    afl_score_random 82800 16.0 374.0 0.0 374.0 374.0 374.0 374.0 374.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

quickjs_eval-2020-01-05 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 0.933333 0.258199 0.0 1.0 1.0 1.0 1.0
    afl_score_max 82800 18.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    afl_score_min 82800 16.0 1.125000 0.341565 1.0 1.0 1.0 1.0 2.0
    afl_score_no_novel_prioritization 82800 13.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    afl_score_random 82800 11.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 15386.733333 575.196919 14398.0 14972.0 15526.0 15670.50 16304.0
    afl_score_min 82800 16.0 15447.187500 332.472399 14772.0 15196.0 15497.5 15664.00 15994.0
    afl_score_no_novel_prioritization 82800 13.0 15477.692308 617.298062 14445.0 15274.0 15429.0 15704.00 16759.0
    afl_score_random 82800 11.0 14835.636364 407.919422 14036.0 14547.5 14964.0 15130.50 15313.0
    afl_score_max 82800 18.0 14643.388889 368.053727 13661.0 14484.5 14619.5 14914.75 15282.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_min 82800 18.0 16.222222 2.510110 9.0 16.00 17.0 18.0 19.0
    afl_score_no_novel_prioritization 82800 12.0 16.250000 2.767506 11.0 15.25 17.0 18.0 20.0
    afl 82800 16.0 16.562500 2.920474 10.0 15.00 16.5 19.0 21.0
    afl_score_max 82800 17.0 15.705882 2.468925 10.0 15.00 16.0 17.0 19.0
    afl_score_random 82800 10.0 14.900000 5.363457 2.0 12.75 16.0 18.5 20.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 2544.812500 112.443301 2402.0 2422.25 2600.5 2649.75 2661.0
    afl_score_random 82800 10.0 2517.100000 115.025070 2373.0 2406.25 2538.0 2621.75 2647.0
    afl_score_max 82800 17.0 2496.000000 102.491463 2383.0 2411.00 2420.0 2598.00 2650.0
    afl_score_no_novel_prioritization 82800 12.0 2485.666667 103.956576 2394.0 2414.75 2420.0 2622.25 2628.0
    afl_score_min 82800 18.0 2465.500000 97.892047 2338.0 2405.25 2413.0 2557.00 2641.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.157895 0.374634 0.0 0.0 0.0 0.0 1.0
    afl_score_max 82800 15.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    afl_score_min 82800 18.0 0.055556 0.235702 0.0 0.0 0.0 0.0 1.0
    afl_score_no_novel_prioritization 82800 17.0 0.058824 0.242536 0.0 0.0 0.0 0.0 1.0
    afl_score_random 82800 12.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 82800 15.0 13606.400000 247.817271 13345.0 13475.50 13582.0 13603.50 14245.0
    afl_score_no_novel_prioritization 82800 17.0 13578.588235 229.986700 13334.0 13424.00 13565.0 13637.00 14373.0
    afl_score_min 82800 18.0 13516.944444 126.500626 13297.0 13448.75 13527.0 13596.75 13726.0
    afl 82800 19.0 13621.789474 384.972521 12843.0 13473.00 13519.0 13601.50 14358.0
    afl_score_random 82800 12.0 13494.416667 63.675967 13396.0 13438.00 13503.5 13551.00 13591.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_max 68400 14.0 3.642857 0.633324 3.0 3.0 4.0 4.0 5.0
    afl_score_min 68400 6.0 3.333333 1.032796 2.0 2.5 4.0 4.0 4.0
    afl 68400 15.0 3.600000 0.828079 3.0 3.0 3.0 4.0 6.0
    afl_score_no_novel_prioritization 68400 9.0 3.444444 0.726483 3.0 3.0 3.0 4.0 5.0
    afl_score_random 68400 5.0 3.400000 0.547723 3.0 3.0 3.0 4.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_score_no_novel_prioritization 68400 9.0 480130.888889 9146.495195 466260.0 471496.00 481760.0 486206.00 492746.0
    afl 68400 15.0 470195.133333 16003.888509 433740.0 466365.50 471128.0 483629.50 486205.0
    afl_score_random 68400 5.0 453059.400000 19064.757166 420100.0 453132.00 461869.0 464914.00 465282.0
    afl_score_max 68400 14.0 441211.000000 11841.164262 420308.0 431986.50 443646.5 449698.00 461783.0
    afl_score_min 68400 6.0 424454.000000 12641.748819 406582.0 419569.25 425147.5 426471.75 445224.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 0.300000 0.483046 0.0 0.0 0.0 0.75 1.0
    afl_score_max 82800 12.0 0.083333 0.288675 0.0 0.0 0.0 0.00 1.0
    afl_score_min 82800 18.0 0.166667 0.383482 0.0 0.0 0.0 0.00 1.0
    afl_score_no_novel_prioritization 82800 5.0 0.000000 0.000000 0.0 0.0 0.0 0.00 0.0
    afl_score_random 82800 10.0 0.000000 0.000000 0.0 0.0 0.0 0.00 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 9761.000000 425.564462 8780.0 9623.50 9930.5 10069.50 10109.0
    afl_score_no_novel_prioritization 82800 5.0 9753.600000 315.940342 9348.0 9492.00 9871.0 10002.00 10055.0
    afl_score_min 82800 18.0 9629.277778 448.818817 8424.0 9469.50 9847.0 9907.25 10033.0
    afl_score_max 82800 12.0 9477.750000 518.482598 8059.0 9303.50 9645.5 9846.00 9881.0
    afl_score_random 82800 10.0 9423.600000 385.909661 8659.0 9253.25 9312.0 9768.75 9902.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 34571b40ef58a9a9a016111322adfde68f019249

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 34571b40ef58a9a9a016111322adfde68f019249
# Download the internal config file.
curl https://storage.googleapis.com/score/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data