FuzzBench: scheduling report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_scheduling_lifo 90.33
afl 82.94
afl_scheduling_random 80.58
By avg. rank
average rank
fuzzer
afl_scheduling_lifo 1.86
afl 2.02
afl_scheduling_random 2.12
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_scheduling_random afl afl_scheduling_lifo
    FuzzerMedian 94.57 94.48 95.06
    FuzzerMean 94.60 94.22 93.28
    arrow_parquet-arrow-fuzz 95.88 98.00 95.64
    aspell_aspell_fuzzer 99.75 99.64 98.98
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan nan
    file_magic_fuzzer 96.03 97.29 97.25
    grok_grk_decompress_fuzzer 89.70 89.61 94.80
    libarchive_libarchive_fuzzer 93.11 90.11 93.44
    libgit2_objects_fuzzer 99.52 99.52 99.52
    libhevc_hevc_dec_fuzzer 91.13 80.98 92.34
    libhtp_fuzz_htp 99.38 99.85 99.35
    libxml2_libxml2_xml_reader_for_file_fuzzer 89.04 91.43 89.15
    matio_matio_fuzzer 87.64 92.21 74.67
    mruby-2018-05-23 89.80 96.45 91.66
    muparser_set_eval_fuzzer 97.70 97.56 97.56
    njs_njs_process_script_fuzzer 93.93 93.41 96.62
    openh264_decoder_fuzzer 98.18 99.18 96.12
    php_php-fuzz-execute 96.16 87.39 90.07
    php_php-fuzz-parser-2020-07-25 99.57 99.11 99.67
    poppler_pdf_fuzzer 97.08 98.60 93.14
    proj4_standard_fuzzer 99.24 100.00 99.24
    quickjs_eval-2020-01-05 91.74 87.76 87.16
    stb_stbi_read_fuzzer 90.19 91.59 98.03
    systemd_fuzz-varlink 94.44 94.44 94.44
    usrsctp_fuzzer_connect 93.50 94.16 93.10
    wireshark_fuzzshark_ip 93.12 88.41 71.47
    zstd_stream_decompress 94.70 94.51 95.32
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_scheduling_lifo afl afl_scheduling_random
    FuzzerMedian 52.78 52.08 53.57
    FuzzerMean 51.32 48.49 47.57
    arrow_parquet-arrow-fuzz 89.13 89.13 89.13
    aspell_aspell_fuzzer 100.00 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer 66.67 54.17 58.33
    file_magic_fuzzer 0.00 0.00 0.00
    grok_grk_decompress_fuzzer 55.56 11.11 11.11
    libarchive_libarchive_fuzzer nan nan nan
    libgit2_objects_fuzzer 66.67 33.33 66.67
    libhevc_hevc_dec_fuzzer 89.39 86.36 87.88
    libhtp_fuzz_htp 60.00 70.00 60.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 18.18 18.18 18.18
    matio_matio_fuzzer 69.23 80.77 76.92
    mruby-2018-05-23 16.67 58.33 16.67
    muparser_set_eval_fuzzer nan nan nan
    njs_njs_process_script_fuzzer 40.00 30.00 20.00
    openh264_decoder_fuzzer 50.00 58.33 50.00
    php_php-fuzz-execute 28.57 14.29 14.29
    php_php-fuzz-parser-2020-07-25 60.00 20.00 25.00
    poppler_pdf_fuzzer 35.71 64.29 61.90
    proj4_standard_fuzzer nan nan nan
    quickjs_eval-2020-01-05 50.00 50.00 50.00
    stb_stbi_read_fuzzer 90.48 85.71 83.33
    systemd_fuzz-varlink 100.00 100.00 100.00
    usrsctp_fuzzer_connect 0.00 0.00 0.00
    wireshark_fuzzshark_ip 42.86 42.86 57.14
    zstd_stream_decompress 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl afl_scheduling_lifo afl_scheduling_random
    FuzzerSum 367 287 284 275
    arrow_parquet-arrow-fuzz 72 63 66 63
    aspell_aspell_fuzzer 2 2 2 2
    ffmpeg_ffmpeg_demuxer_fuzzer 38 23 26 23
    file_magic_fuzzer 1 1 1 1
    grok_grk_decompress_fuzzer 13 7 13 4
    libarchive_libarchive_fuzzer 0 0 0 0
    libgit2_objects_fuzzer 3 3 2 3
    libhevc_hevc_dec_fuzzer 34 34 34 33
    libhtp_fuzz_htp 5 5 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 11 11 2 2
    matio_matio_fuzzer 25 22 21 16
    mruby-2018-05-23 7 6 4 5
    muparser_set_eval_fuzzer 0 0 0 0
    njs_njs_process_script_fuzzer 7 5 5 5
    openh264_decoder_fuzzer 6 6 6 6
    php_php-fuzz-execute 24 8 15 10
    php_php-fuzz-parser-2020-07-25 15 9 15 11
    poppler_pdf_fuzzer 59 43 27 49
    proj4_standard_fuzzer 0 0 0 0
    quickjs_eval-2020-01-05 4 1 4 2
    stb_stbi_read_fuzzer 26 26 26 25
    systemd_fuzz-varlink 1 1 1 1
    usrsctp_fuzzer_connect 1 1 1 1
    wireshark_fuzzshark_ip 11 9 7 6
    zstd_stream_decompress 2 1 1 2
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 41.294118 2.687115 37.0 40.0 41.0 43.0 46.0
    afl_scheduling_lifo 82800 17.0 39.941176 3.071453 34.0 38.0 41.0 42.0 44.0
    afl_scheduling_random 82800 16.0 41.000000 2.129163 37.0 41.0 41.0 42.0 45.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3236.647059 37.634660 3172.0 3210.00 3236.0 3260.00 3302.0
    afl_scheduling_random 82800 16.0 3153.000000 51.360166 3052.0 3131.25 3166.0 3187.25 3230.0
    afl_scheduling_lifo 82800 17.0 3157.823529 63.536638 2994.0 3137.00 3158.0 3180.00 3291.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.0 0.000000 2.0 2.0 2.0 2.0 2.0
    afl_scheduling_lifo 82800 20.0 1.2 1.005249 0.0 0.0 2.0 2.0 2.0
    afl_scheduling_random 82800 20.0 2.0 0.000000 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 20.0 3332.0 8.006576 3310.0 3329.75 3333.5 3337.00 3342.0
    afl 82800 20.0 3327.5 9.185228 3312.0 3320.50 3330.0 3333.25 3341.0
    afl_scheduling_lifo 82800 20.0 3252.7 84.471172 3138.0 3164.50 3308.0 3329.25 3337.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 7.75 2.149051 4.0 6.0 8.0 9.00 12.0
    afl_scheduling_random 82800 20.0 7.10 2.100125 3.0 6.0 7.0 9.00 11.0
    afl 82800 20.0 6.45 1.503505 4.0 5.0 6.5 7.25 9.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_lifo 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_random 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.3 0.470162 0.0 0.0 0.0 1.0 1.0
    afl_scheduling_lifo 82800 20.0 0.4 0.502625 0.0 0.0 0.0 1.0 1.0
    afl_scheduling_random 82800 20.0 0.2 0.410391 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2316.35 40.320133 2248.0 2278.50 2315.5 2347.50 2380.0
    afl_scheduling_lifo 82800 20.0 2304.45 41.578683 2229.0 2272.00 2314.5 2336.75 2373.0
    afl_scheduling_random 82800 20.0 2284.65 43.281545 2223.0 2245.25 2285.5 2320.00 2350.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 4.70 2.939746 0.0 2.0 5.0 7.25 9.0
    afl 82800 20.0 1.30 1.080935 0.0 1.0 1.0 1.00 5.0
    afl_scheduling_random 82800 20.0 1.15 0.587143 0.0 1.0 1.0 1.25 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 5876.5 276.793006 5529.0 5628.25 5878.5 6144.75 6201.0
    afl_scheduling_random 82800 20.0 5581.8 70.827143 5507.0 5554.00 5562.0 5592.00 5860.0
    afl 82800 20.0 5580.2 100.498023 5509.0 5546.25 5556.5 5571.00 5993.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_lifo 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_random 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 3359.0 92.450271 3078.0 3333.25 3354.5 3417.00 3534.0
    afl_scheduling_random 82800 20.0 3337.9 141.213873 2965.0 3316.75 3342.5 3395.75 3580.0
    afl 82800 20.0 3237.6 183.976371 2886.0 3131.25 3235.0 3358.75 3590.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 1.75 0.444262 1.0 1.75 2.0 2.0 2.0
    afl_scheduling_random 82800 20.0 1.90 0.447214 1.0 2.00 2.0 2.0 3.0
    afl 82800 20.0 1.55 0.686333 1.0 1.00 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 835.15 1.631112 833.0 834.0 835.0 836.0 839.0
    afl_scheduling_lifo 82800 20.0 834.95 1.468081 833.0 834.0 835.0 836.0 838.0
    afl_scheduling_random 82800 20.0 834.00 2.339591 827.0 834.0 835.0 835.0 836.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 28.25 6.568465 1.0 28.0 29.5 31.00 32.0
    afl_scheduling_random 82800 20.0 28.10 5.159559 7.0 28.0 29.0 30.25 31.0
    afl 82800 20.0 24.85 10.424036 1.0 27.0 28.5 30.00 33.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 6717.05 1601.634082 555.0 6841.50 7204.5 7380.75 7802.0
    afl_scheduling_random 82800 20.0 6646.10 1535.175351 801.0 6445.00 7110.0 7373.75 7751.0
    afl 82800 20.0 5541.95 2221.840429 555.0 5699.75 6318.0 6809.50 7308.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3.35 1.308877 0.0 2.75 3.5 4.0 5.0
    afl_scheduling_lifo 82800 20.0 2.65 2.207046 0.0 0.00 3.0 5.0 5.0
    afl_scheduling_random 82800 20.0 2.55 2.089447 0.0 0.00 3.0 5.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2987.50 2.762531 2981.0 2985.75 2987.5 2989.25 2992.0
    afl_scheduling_random 82800 20.0 2973.15 2.560325 2968.0 2971.75 2973.5 2975.25 2977.0
    afl_scheduling_lifo 82800 20.0 2972.70 3.130495 2966.0 2971.00 2972.5 2973.75 2978.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.90 2.403944 2.0 2.0 2.0 2.0 11.0
    afl_scheduling_lifo 82800 20.0 1.85 0.366348 1.0 2.0 2.0 2.0 2.0
    afl_scheduling_random 82800 20.0 1.90 0.307794 1.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 13891.3 596.884464 12601.0 13628.00 13795.0 13873.50 15088.0
    afl_scheduling_lifo 82800 20.0 13304.2 445.091309 12240.0 13163.50 13450.5 13635.50 13754.0
    afl_scheduling_random 82800 20.0 13327.8 628.694102 11963.0 13105.75 13434.0 13628.25 14449.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 10.45 1.356272 8.0 9.00 10.5 11.25 13.0
    afl_scheduling_random 82800 20.0 9.90 1.071153 8.0 9.75 10.0 10.25 12.0
    afl_scheduling_lifo 82800 20.0 8.95 1.605091 6.0 8.00 9.0 10.00 11.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 4839.70 127.054526 4606.0 4746.75 4859.5 4919.75 5087.0
    afl_scheduling_random 82800 20.0 4627.60 364.392703 3902.0 4469.50 4618.5 4881.75 5240.0
    afl_scheduling_lifo 82800 20.0 4014.45 571.368392 3209.0 3545.00 3935.0 4389.25 5270.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby-2018-05-23 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, afl_scheduling_lifo.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 71100 8.0 3.375 0.744024 2.0 3.0 3.5 4.0 4.0
    afl_scheduling_lifo 71100 2.0 1.000 1.414214 0.0 0.5 1.0 1.5 2.0
    afl_scheduling_random 71100 20.0 0.850 0.875094 0.0 0.0 1.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 71100 8.0 7796.875 206.873624 7440.0 7738.50 7816.5 7916.50 8092.0
    afl_scheduling_lifo 71100 2.0 7428.500 201.525433 7286.0 7357.25 7428.5 7499.75 7571.0
    afl_scheduling_random 71100 20.0 7332.600 221.533269 7036.0 7176.00 7277.5 7445.25 7843.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_lifo 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_random 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 20.0 723.00 5.211324 719.0 721.0 722.0 722.0 739.0
    afl 82800 20.0 721.35 1.039990 720.0 721.0 721.0 722.0 723.0
    afl_scheduling_lifo 82800 20.0 722.30 3.867544 719.0 721.0 721.0 723.0 738.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 2.30 1.260743 1.0 1.0 2.0 3.0 5.0
    afl 82800 20.0 1.65 0.745160 1.0 1.0 1.5 2.0 3.0
    afl_scheduling_random 82800 20.0 1.75 1.208522 1.0 1.0 1.0 2.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 3657.15 87.301218 3491.0 3587.25 3675.5 3703.50 3804.0
    afl_scheduling_random 82800 20.0 3581.70 56.311259 3506.0 3531.25 3573.0 3622.25 3710.0
    afl 82800 20.0 3543.00 81.931936 3294.0 3500.25 3553.5 3594.75 3664.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3.45 0.887041 2.0 3.00 3.5 4.0 5.0
    afl_scheduling_lifo 82800 20.0 2.65 1.694418 0.0 1.75 3.0 4.0 5.0
    afl_scheduling_random 82800 20.0 3.20 1.281447 1.0 2.00 3.0 4.0 6.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 7757.40 49.102899 7683.0 7711.25 7777.5 7791.25 7842.0
    afl_scheduling_random 82800 20.0 7688.95 75.622661 7546.0 7634.75 7699.0 7748.75 7793.0
    afl_scheduling_lifo 82800 20.0 7552.90 72.004313 7437.0 7510.00 7537.5 7591.25 7720.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 2.30 1.809333 0.0 1.0 2.0 3.0 7.0
    afl 82800 20.0 1.35 0.587143 1.0 1.0 1.0 2.0 3.0
    afl_scheduling_random 82800 20.0 1.60 0.882580 1.0 1.0 1.0 2.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 20.0 61165.80 1789.470536 56801.0 60616.25 61641.0 62348.25 63901.0
    afl_scheduling_lifo 82800 20.0 54766.30 7448.950276 44173.0 48040.25 57741.0 61529.00 64104.0
    afl 82800 20.0 56398.05 2005.125996 52748.0 55269.25 56017.5 58216.75 59959.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 5.80 2.284962 1.0 4.0 6.0 7.0 10.0
    afl_scheduling_random 82800 20.0 2.45 1.234376 0.0 2.0 2.5 3.0 5.0
    afl 82800 20.0 1.65 1.089423 0.0 1.0 2.0 2.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 17159.2 36.624122 17077.0 17144.00 17162.0 17181.00 17218.0
    afl_scheduling_random 82800 20.0 17148.0 20.619357 17101.0 17137.00 17144.0 17159.75 17184.0
    afl 82800 20.0 17049.5 44.921569 16945.0 17031.75 17064.5 17077.25 17109.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 14.5 2.704772 11.0 12.75 13.5 16.25 21.0
    afl_scheduling_random 82800 20.0 13.4 3.439706 7.0 11.00 13.0 15.00 19.0
    afl_scheduling_lifo 82800 20.0 8.0 2.554665 3.0 6.00 7.5 10.00 12.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 19416.40 75.509079 19298.0 19365.25 19401.0 19472.50 19575.0
    afl_scheduling_random 82800 20.0 19183.10 254.728111 18794.0 19046.50 19101.0 19304.75 19676.0
    afl_scheduling_lifo 82800 20.0 18335.95 173.729819 17757.0 18279.50 18325.5 18443.75 18585.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_lifo 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_scheduling_random 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 132.0 0.0 132.0 132.0 132.0 132.0 132.0
    afl_scheduling_lifo 82800 20.0 131.0 0.0 131.0 131.0 131.0 131.0 131.0
    afl_scheduling_random 82800 20.0 131.0 0.0 131.0 131.0 131.0 131.0 131.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

quickjs_eval-2020-01-05 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    afl_scheduling_lifo 82800 20.0 0.95 0.604805 0.0 1.0 1.0 1.0 2.0
    afl_scheduling_random 82800 20.0 0.85 0.489360 0.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 20.0 8689.65 561.979048 7645.0 8422.25 8829.0 9085.50 9521.0
    afl 82800 20.0 8480.75 433.260330 7577.0 8238.75 8446.0 8795.75 9624.0
    afl_scheduling_lifo 82800 20.0 8524.95 514.557805 7829.0 8213.00 8388.5 8765.25 9595.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 18.40 2.161871 13.0 17.75 19.0 20.00 21.0
    afl 82800 20.0 17.25 3.242400 8.0 16.75 18.0 18.25 21.0
    afl_scheduling_random 82800 20.0 17.05 2.818641 11.0 15.50 17.5 19.00 21.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 1588.50 75.132515 1483.0 1496.50 1638.0 1646.75 1657.0
    afl 82800 20.0 1568.00 75.074700 1475.0 1506.00 1530.5 1647.00 1671.0
    afl_scheduling_random 82800 20.0 1558.55 72.054420 1486.0 1495.75 1507.0 1641.50 1651.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    afl_scheduling_lifo 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    afl_scheduling_random 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 5220.25 127.482868 5070.0 5149.00 5185.5 5214.75 5507.0
    afl_scheduling_random 82800 20.0 5241.30 164.468554 5072.0 5121.50 5149.0 5436.50 5504.0
    afl_scheduling_lifo 82800 20.0 5149.70 117.910358 5008.0 5100.75 5127.0 5162.50 5470.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 19.0 3.526316 0.512989 3.0 3.0 4.0 4.0 4.0
    afl 82800 19.0 3.736842 0.933459 3.0 3.0 3.0 4.0 6.0
    afl_scheduling_lifo 82800 19.0 3.421053 0.961237 3.0 3.0 3.0 3.5 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_random 82800 19.0 77788.526316 9767.267959 50949.0 79565.0 81345.0 82209.5 87355.0
    afl 82800 19.0 75422.210526 6068.834993 57132.0 73207.5 77229.0 79559.0 82160.0
    afl_scheduling_lifo 82800 19.0 61396.263158 12344.044708 39632.0 49555.5 62433.0 72537.5 78883.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    afl_scheduling_lifo 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    afl_scheduling_random 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_scheduling_lifo 82800 20.0 4978.05 159.070483 4649.0 4950.5 5056.5 5083.25 5103.0
    afl_scheduling_random 82800 20.0 4910.60 211.642300 4420.0 4715.0 5024.0 5068.00 5091.0
    afl 82800 20.0 4930.75 218.111482 4353.0 4893.0 5014.0 5040.25 5305.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 889718617d09ff81540a5d73abce74223861851c

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 889718617d09ff81540a5d73abce74223861851c
# Download the internal config file.
curl https://storage.googleapis.com/scheduling/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data