FuzzBench: fitness report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 83.32
afl_fitness 83.08
afl_fitness_only 70.17
By avg. rank
average rank
fuzzer
afl 1.94
afl_fitness 1.96
afl_fitness_only 2.10
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl afl_fitness afl_fitness_only
    FuzzerMedian 96.84 95.84 74.66
    FuzzerMean 90.60 90.02 67.65
    arrow_parquet-arrow-fuzz 96.72 95.77 84.39
    aspell_aspell_fuzzer 99.64 99.66 97.85
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan nan
    file_magic_fuzzer 98.10 97.63 64.22
    grok_grk_decompress_fuzzer 93.14 92.94 88.26
    libarchive_libarchive_fuzzer 89.02 88.71 29.54
    libgit2_objects_fuzzer 99.92 99.92 96.84
    libhevc_hevc_dec_fuzzer 81.93 80.46 7.14
    libhtp_fuzz_htp 99.92 99.93 94.44
    libxml2_libxml2_xml_reader_for_file_fuzzer 90.91 89.64 58.52
    matio_matio_fuzzer 97.80 97.77 83.71
    muparser_set_eval_fuzzer 97.96 97.91 79.79
    ndpi_fuzz_ndpi_reader 1.12 1.12 0.90
    njs_njs_process_script_fuzzer 96.52 95.84 50.06
    openh264_decoder_fuzzer 99.26 99.34 93.69
    php_php-fuzz-execute 95.50 95.83 74.84
    php_php-fuzz-parser-2020-07-25 99.13 99.04 90.10
    poppler_pdf_fuzzer 98.99 99.11 93.02
    proj4_standard_fuzzer 71.37 71.37 70.99
    stb_stbi_read_fuzzer 87.47 87.40 74.49
    systemd_fuzz-varlink 100.00 88.48 88.48
    tpm2_tpm2_execute_command_fuzzer 89.98 93.21 21.31
    usrsctp_fuzzer_connect 94.89 94.68 68.18
    wireshark_fuzzshark_ip 96.95 96.12 66.49
    zstd_stream_decompress 98.09 98.52 46.45
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl afl_fitness afl_fitness_only
    FuzzerMedian 25.00 25.00 16.67
    FuzzerMean 30.04 29.91 24.07
    arrow_parquet-arrow-fuzz 86.67 86.67 77.78
    aspell_aspell_fuzzer nan nan nan
    ffmpeg_ffmpeg_demuxer_fuzzer 40.00 40.00 0.00
    file_magic_fuzzer nan nan nan
    grok_grk_decompress_fuzzer 25.00 25.00 50.00
    libarchive_libarchive_fuzzer nan nan nan
    libgit2_objects_fuzzer 33.33 33.33 33.33
    libhevc_hevc_dec_fuzzer 0.00 0.00 0.00
    libhtp_fuzz_htp 60.00 70.00 0.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 16.67 16.67 16.67
    matio_matio_fuzzer 83.33 83.33 70.83
    muparser_set_eval_fuzzer nan nan nan
    ndpi_fuzz_ndpi_reader 0.00 0.00 0.00
    njs_njs_process_script_fuzzer 0.00 0.00 0.00
    openh264_decoder_fuzzer 43.75 37.50 50.00
    php_php-fuzz-execute 0.00 0.00 40.00
    php_php-fuzz-parser-2020-07-25 18.75 25.00 31.25
    poppler_pdf_fuzzer 50.00 37.50 37.50
    proj4_standard_fuzzer nan nan nan
    stb_stbi_read_fuzzer 80.00 80.00 50.00
    systemd_fuzz-varlink 0.00 0.00 0.00
    tpm2_tpm2_execute_command_fuzzer nan nan nan
    usrsctp_fuzzer_connect 0.00 0.00 0.00
    wireshark_fuzzshark_ip 33.33 33.33 0.00
    zstd_stream_decompress 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl_fitness afl afl_fitness_only
    FuzzerSum 235 180 179 147
    arrow_parquet-arrow-fuzz 107 88 85 75
    aspell_aspell_fuzzer 0 0 0 0
    ffmpeg_ffmpeg_demuxer_fuzzer 6 5 6 1
    file_magic_fuzzer 0 0 0 0
    grok_grk_decompress_fuzzer 4 3 4 2
    libarchive_libarchive_fuzzer 0 0 0 0
    libgit2_objects_fuzzer 3 3 3 1
    libhevc_hevc_dec_fuzzer 3 3 1 0
    libhtp_fuzz_htp 5 5 5 3
    libxml2_libxml2_xml_reader_for_file_fuzzer 13 11 13 2
    matio_matio_fuzzer 24 19 17 17
    muparser_set_eval_fuzzer 0 0 0 0
    ndpi_fuzz_ndpi_reader 4 4 0 0
    njs_njs_process_script_fuzzer 2 2 1 0
    openh264_decoder_fuzzer 9 5 7 8
    php_php-fuzz-execute 15 2 4 12
    php_php-fuzz-parser-2020-07-25 13 9 8 12
    poppler_pdf_fuzzer 9 4 7 5
    proj4_standard_fuzzer 0 0 0 0
    stb_stbi_read_fuzzer 11 11 11 8
    systemd_fuzz-varlink 1 0 1 0
    tpm2_tpm2_execute_command_fuzzer 0 0 0 0
    usrsctp_fuzzer_connect 1 1 1 0
    wireshark_fuzzshark_ip 4 4 4 0
    zstd_stream_decompress 1 1 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 13.0 38.538462 2.904462 34.0 37.0 39.0 40.00 43.0
    afl_fitness 82800 19.0 39.000000 2.081666 36.0 38.0 39.0 40.00 45.0
    afl_fitness_only 82800 20.0 35.800000 2.820974 31.0 34.0 35.0 37.25 41.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 13.0 5179.769231 75.053485 5011.0 5123.0 5188.0 5232.0 5298.0
    afl_fitness 82800 19.0 5158.105263 64.960582 5051.0 5111.5 5137.0 5221.0 5265.0
    afl_fitness_only 82800 20.0 4523.150000 35.302117 4451.0 4512.5 4526.5 4548.0 4589.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 5544.10 20.227392 5507.0 5537.50 5551.0 5559.00 5570.0
    afl 82800 20.0 5547.55 13.823988 5508.0 5548.75 5550.0 5551.50 5560.0
    afl_fitness_only 82800 20.0 5452.15 10.158092 5436.0 5448.75 5450.0 5453.75 5483.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.25 0.850696 1.0 2.00 2.0 3.0 4.0
    afl_fitness 82800 20.0 2.20 1.196486 0.0 1.75 2.0 3.0 5.0
    afl_fitness_only 82800 20.0 0.05 0.223607 0.0 0.00 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3702.75 184.380863 2956.0 3685.00 3748.5 3787.00 3821.0
    afl_fitness 82800 20.0 3662.45 245.333602 2964.0 3673.50 3730.5 3791.75 3819.0
    afl_fitness_only 82800 20.0 2387.70 147.202367 2181.0 2240.25 2454.0 2513.25 2556.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness_only 82800 20.0 1.65 0.489360 1.0 1.00 2.0 2.0 2.0
    afl 82800 20.0 1.20 0.894427 0.0 1.00 1.0 1.0 4.0
    afl_fitness 82800 20.0 0.90 0.640723 0.0 0.75 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 8898.30 233.123211 8747.0 8779.75 8821.5 8850.50 9451.0
    afl_fitness 82800 20.0 8829.05 153.196631 8744.0 8778.00 8802.0 8810.25 9471.0
    afl_fitness_only 82800 20.0 8364.40 24.336241 8313.0 8351.25 8359.5 8375.75 8425.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 5910.3 441.683633 5215.0 5510.50 5920.0 6108.75 6650.0
    afl_fitness 82800 20.0 5851.3 377.024514 5253.0 5493.00 5899.0 6072.50 6435.0
    afl_fitness_only 82800 20.0 2118.6 295.647977 1897.0 1940.75 1964.5 2148.25 2650.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.1 0.307794 1.0 1.0 1.0 1.0 2.0
    afl_fitness 82800 20.0 1.5 0.606977 1.0 1.0 1.0 2.0 3.0
    afl_fitness_only 82800 20.0 1.0 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2438.40 0.753937 2437.0 2438.00 2438.0 2439.00 2440.0
    afl_fitness 82800 20.0 2437.35 3.869925 2421.0 2438.00 2438.0 2438.00 2439.0
    afl_fitness_only 82800 20.0 2367.50 13.136931 2351.0 2355.75 2363.0 2381.25 2384.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.3 0.470162 0.0 0.0 0.0 1.0 1.0
    afl_fitness 82800 20.0 0.5 0.688247 0.0 0.0 0.0 1.0 2.0
    afl_fitness_only 82800 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 8712.60 4012.893737 878.0 9008.00 10029.5 11346.25 12207.0
    afl_fitness 82800 20.0 9270.55 3717.714811 878.0 9336.75 9849.0 11839.50 12241.0
    afl_fitness_only 82800 20.0 1220.55 1544.878925 871.0 873.75 874.0 877.00 7784.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 3.55 0.998683 2.0 3.0 3.5 4.0 5.0
    afl 82800 20.0 3.00 1.521772 0.0 2.0 3.0 4.0 5.0
    afl_fitness_only 82800 20.0 0.35 0.812728 0.0 0.0 0.0 0.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 6579.00 2.176429 6573.0 6577.0 6579.5 6581.00 6582.0
    afl 82800 20.0 6579.10 2.245463 6575.0 6578.0 6579.0 6581.00 6584.0
    afl_fitness_only 82800 20.0 6010.45 270.604290 5675.0 5731.5 6218.0 6249.25 6298.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 3.00 2.635786 2.0 2.0 2.0 2.0 12.0
    afl_fitness 82800 20.0 2.35 2.058998 1.0 2.0 2.0 2.0 11.0
    afl_fitness_only 82800 20.0 1.45 0.686333 0.0 1.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 20637.70 747.666271 19175.0 20454.25 20680.0 20963.25 22145.0
    afl_fitness 82800 20.0 20536.65 786.038989 19323.0 20151.25 20390.5 20730.75 22747.0
    afl_fitness_only 82800 20.0 12766.65 1736.683811 8268.0 12062.00 13312.0 13846.25 14918.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 9.8 0.894427 7.0 9.75 10.0 10.0 11.0
    afl_fitness 82800 20.0 10.0 1.213954 8.0 9.00 10.0 11.0 12.0
    afl_fitness_only 82800 20.0 8.4 1.187656 7.0 7.00 8.5 9.0 11.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2757.05 21.624730 2715.0 2745.75 2759.0 2767.25 2813.0
    afl_fitness 82800 20.0 2765.35 26.055154 2717.0 2752.75 2758.0 2779.75 2821.0
    afl_fitness_only 82800 20.0 2330.80 95.384320 2176.0 2247.75 2361.5 2402.50 2477.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 935.85 5.214100 924.0 935.0 935.5 936.0 955.0
    afl_fitness 82800 20.0 932.80 3.914884 925.0 932.5 935.0 935.0 936.0
    afl_fitness_only 82800 20.0 760.80 23.246052 716.0 741.5 762.0 773.5 810.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ndpi_fuzz_ndpi_reader summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.2 0.894427 0.0 0.0 0.0 0.0 4.0
    afl_fitness_only 82800 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 91.50 7.192833 77.0 95.0 95.0 95.0 95.0
    afl_fitness 82800 20.0 510.45 1885.270593 77.0 77.0 95.0 95.0 8520.0
    afl_fitness_only 82800 20.0 77.00 0.000000 77.0 77.0 77.0 77.0 77.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.3 0.470162 0.0 0.0 0.0 1.0 1.0
    afl_fitness 82800 20.0 0.2 0.410391 0.0 0.0 0.0 0.0 1.0
    afl_fitness_only 82800 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 7104.75 91.016988 6885.0 7048.50 7121.5 7137.75 7283.0
    afl_fitness 82800 20.0 7095.60 152.675232 6728.0 6999.50 7071.0 7214.25 7378.0
    afl_fitness_only 82800 20.0 3735.70 194.074348 3530.0 3565.25 3693.5 3858.00 4132.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness_only 82800 20.0 4.35 0.988087 4.0 4.00 4.0 4.0 8.0
    afl 82800 20.0 3.50 1.317893 1.0 3.00 3.5 4.0 7.0
    afl_fitness 82800 20.0 3.20 1.196486 1.0 2.75 3.0 4.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 14259.20 78.712401 14069.0 14201.00 14263.5 14323.50 14353.0
    afl 82800 20.0 14268.05 58.345139 14123.0 14237.25 14251.5 14324.50 14358.0
    afl_fitness_only 82800 20.0 13462.40 46.236691 13387.0 13434.00 13452.0 13496.25 13542.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness_only 82800 20.0 2.25 1.069924 1.0 1.75 2.0 3.0 5.0
    afl 82800 20.0 0.20 0.410391 0.0 0.00 0.0 0.0 1.0
    afl_fitness 82800 20.0 0.10 0.307794 0.0 0.00 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 173698.05 2868.354204 168583.0 171505.75 173662.5 174705.75 181213.0
    afl 82800 20.0 173432.75 3314.968046 167978.0 171228.00 173066.5 175025.50 181103.0
    afl_fitness_only 82800 20.0 135639.35 101.950594 135455.0 135566.00 135618.5 135714.00 135852.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness_only 82800 20.0 2.95 1.820208 1.0 2.0 2.5 3.25 8.0
    afl_fitness 82800 20.0 2.20 1.399248 0.0 1.0 2.0 3.00 5.0
    afl 82800 20.0 1.75 1.332785 0.0 1.0 1.5 2.00 6.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 44880.75 202.323835 44539.0 44733.5 44908.0 45033.50 45194.0
    afl_fitness 82800 20.0 44834.05 330.650471 43854.0 44671.0 44868.0 45064.25 45302.0
    afl_fitness_only 82800 20.0 40834.85 295.176178 40538.0 40622.5 40819.0 40951.25 41748.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.90 0.911910 1.0 1.0 2.0 2.25 4.0
    afl_fitness 82800 20.0 1.55 0.604805 1.0 1.0 1.5 2.00 3.0
    afl_fitness_only 82800 20.0 1.60 0.680557 1.0 1.0 1.5 2.00 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 38231.0 126.301644 37909.0 38162.00 38246.0 38296.50 38444.0
    afl 82800 20.0 38224.8 124.305905 38056.0 38164.00 38199.5 38274.25 38589.0
    afl_fitness_only 82800 20.0 35909.3 90.161317 35744.0 35857.75 35894.5 35964.25 36136.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 381.50 33.54102 374.0 374.0 374.0 374.0 524.0
    afl_fitness 82800 20.0 374.00 0.00000 374.0 374.0 374.0 374.0 374.0
    afl_fitness_only 82800 20.0 372.35 1.03999 371.0 372.0 372.0 372.5 374.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 8.2 0.767772 7.0 8.0 8.0 9.0 9.0
    afl_fitness 82800 20.0 7.9 0.967906 7.0 7.0 8.0 8.0 10.0
    afl_fitness_only 82800 20.0 4.8 0.951453 3.0 4.0 5.0 5.0 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2501.45 94.975052 2411.0 2419.0 2432.5 2601.00 2651.0
    afl_fitness 82800 20.0 2513.85 128.039581 2390.0 2414.0 2430.5 2631.75 2781.0
    afl_fitness_only 82800 20.0 2070.25 44.192015 1947.0 2061.0 2071.5 2087.00 2156.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

tpm2_tpm2_execute_command_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_fitness_only 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 5590.800000 603.156487 4455.0 5436.75 5888.0 5982.75 6115.0
    afl 82800 18.0 5491.000000 555.014997 4387.0 4932.75 5684.0 5946.50 6094.0
    afl_fitness_only 82800 19.0 1367.210526 139.475437 1166.0 1279.50 1346.0 1420.00 1704.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    afl_fitness 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    afl_fitness_only 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 13604.80 226.134473 13409.0 13519.25 13561.5 13593.25 14292.0
    afl_fitness 82800 20.0 13580.65 228.650797 13357.0 13451.00 13531.5 13587.00 14260.0
    afl_fitness_only 82800 20.0 9684.30 310.604013 8908.0 9620.25 9745.0 9910.50 10006.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.20 0.523148 1.0 1.0 1.0 1.0 3.0
    afl_fitness 82800 20.0 1.25 0.550120 1.0 1.0 1.0 1.0 3.0
    afl_fitness_only 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 471084.55 12772.524481 432859.0 464909.75 473292.5 481591.25 488161.0
    afl_fitness 82800 20.0 465562.75 9289.074782 448047.0 458081.25 469200.0 472657.00 479159.0
    afl_fitness_only 82800 20.0 324822.00 837.506449 323709.0 324107.50 324602.5 325610.50 326280.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    afl_fitness 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    afl_fitness_only 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_fitness 82800 20.0 9782.65 309.012566 9129.0 9621.25 9926.0 9993.50 10059.0
    afl 82800 20.0 9798.90 267.014271 9291.0 9692.75 9882.5 9993.75 10075.0
    afl_fitness_only 82800 20.0 4805.90 256.059183 4601.0 4648.00 4679.5 4870.00 5580.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: cfbeee00f22d790a9e1cfe9b47a0ab52062f42c8

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout cfbeee00f22d790a9e1cfe9b47a0ab52062f42c8
# Download the internal config file.
curl https://storage.googleapis.com/fitness/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data