FuzzBench: splicing report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl_splicing_mutation 97.15
afl 94.66
By avg. rank
average rank
fuzzer
afl_splicing_mutation 1.48
afl 1.52
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_splicing_mutation afl
    FuzzerMedian 95.65 94.95
    FuzzerMean 94.64 93.96
    arrow_parquet-arrow-fuzz 95.00 95.36
    aspell_aspell_fuzzer 99.85 99.82
    ffmpeg_ffmpeg_demuxer_fuzzer nan nan
    file_magic_fuzzer 97.86 97.88
    grok_grk_decompress_fuzzer 91.83 92.15
    libarchive_libarchive_fuzzer 89.07 89.28
    libgit2_objects_fuzzer 99.29 99.29
    libhevc_hevc_dec_fuzzer 75.11 75.51
    libhtp_fuzz_htp 99.80 99.83
    libxml2_libxml2_xml_reader_for_file_fuzzer 94.34 90.83
    matio_matio_fuzzer 95.40 94.54
    mruby-2018-05-23 95.90 89.63
    muparser_set_eval_fuzzer 97.96 97.96
    njs_njs_process_script_fuzzer 96.08 96.37
    openh264_decoder_fuzzer 99.23 99.16
    php_php-fuzz-execute 92.30 88.85
    php_php-fuzz-parser-2020-07-25 99.57 99.33
    poppler_pdf_fuzzer 98.34 98.41
    proj4_standard_fuzzer 100.00 100.00
    quickjs_eval-2020-01-05 93.41 93.58
    stb_stbi_read_fuzzer 89.92 84.74
    systemd_fuzz-varlink 94.44 94.44
    usrsctp_fuzzer_connect 92.70 92.21
    wireshark_fuzzshark_ip 84.89 87.77
    zstd_stream_decompress 99.14 98.15
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    afl_splicing_mutation afl
    FuzzerMedian 66.67 60.00
    FuzzerMean 55.21 53.42
    arrow_parquet-arrow-fuzz 89.13 91.30
    aspell_aspell_fuzzer 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer 66.67 66.67
    file_magic_fuzzer 0.00 0.00
    grok_grk_decompress_fuzzer 20.00 20.00
    libarchive_libarchive_fuzzer nan nan
    libgit2_objects_fuzzer 50.00 50.00
    libhevc_hevc_dec_fuzzer 87.10 88.71
    libhtp_fuzz_htp 80.00 80.00
    libxml2_libxml2_xml_reader_for_file_fuzzer 20.00 20.00
    matio_matio_fuzzer 71.43 71.43
    mruby-2018-05-23 75.00 75.00
    muparser_set_eval_fuzzer 0.00 0.00
    njs_njs_process_script_fuzzer 25.00 50.00
    openh264_decoder_fuzzer 66.67 50.00
    php_php-fuzz-execute 33.33 33.33
    php_php-fuzz-parser-2020-07-25 66.67 25.00
    poppler_pdf_fuzzer 72.50 70.00
    proj4_standard_fuzzer nan nan
    quickjs_eval-2020-01-05 100.00 100.00
    stb_stbi_read_fuzzer 86.36 77.27
    systemd_fuzz-varlink 100.00 100.00
    usrsctp_fuzzer_connect 0.00 0.00
    wireshark_fuzzshark_ip 60.00 60.00
    zstd_stream_decompress 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
    Total afl_splicing_mutation afl
    FuzzerSum 311 287 259
    arrow_parquet-arrow-fuzz 70 63 66
    aspell_aspell_fuzzer 2 2 2
    ffmpeg_ffmpeg_demuxer_fuzzer 24 22 18
    file_magic_fuzzer 1 1 1
    grok_grk_decompress_fuzzer 6 4 6
    libarchive_libarchive_fuzzer 0 0 0
    libgit2_objects_fuzzer 3 3 3
    libhevc_hevc_dec_fuzzer 34 34 34
    libhtp_fuzz_htp 5 5 5
    libxml2_libxml2_xml_reader_for_file_fuzzer 10 10 2
    matio_matio_fuzzer 20 17 18
    mruby-2018-05-23 8 7 7
    muparser_set_eval_fuzzer 1 0 1
    njs_njs_process_script_fuzzer 5 4 5
    openh264_decoder_fuzzer 7 7 6
    php_php-fuzz-execute 11 9 5
    php_php-fuzz-parser-2020-07-25 14 13 7
    poppler_pdf_fuzzer 52 49 37
    proj4_standard_fuzzer 0 0 0
    quickjs_eval-2020-01-05 1 1 1
    stb_stbi_read_fuzzer 25 25 25
    systemd_fuzz-varlink 1 1 1
    usrsctp_fuzzer_connect 1 1 1
    wireshark_fuzzshark_ip 9 9 7
    zstd_stream_decompress 1 0 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 41.647059 2.644361 36.0 40.0 42.0 44.0 46.0
    afl_splicing_mutation 82800 17.0 40.941176 2.192903 37.0 39.0 41.0 42.0 44.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3207.117647 68.584694 3070.0 3155.0 3187.0 3256.0 3342.0
    afl_splicing_mutation 82800 17.0 3166.882353 67.740204 3001.0 3128.0 3175.0 3213.0 3280.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

aspell_aspell_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0
    afl_splicing_mutation 82800 18.0 2.0 0.0 2.0 2.0 2.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 18.0 3327.666667 10.128585 3310.0 3317.75 3333.0 3334.75 3337.0
    afl 82800 17.0 3330.352941 6.092425 3310.0 3328.00 3332.0 3333.00 3338.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

ffmpeg_ffmpeg_demuxer_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 5.368421 1.300022 3.0 4.5 6.0 6.0 7.0
    afl_splicing_mutation 82800 19.0 6.315789 1.335525 4.0 5.5 6.0 7.0 9.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_splicing_mutation 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

file_magic_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.300000 0.470162 0.0 0.0 0.0 1.0 1.0
    afl_splicing_mutation 82800 19.0 0.421053 0.507257 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2327.850000 35.683219 2269.0 2305.25 2327.5 2357.0 2378.0
    afl_splicing_mutation 82800 19.0 2303.263158 101.006514 1909.0 2294.50 2327.0 2348.5 2377.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

grok_grk_decompress_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 1.368421 1.065130 0.0 1.0 1.0 2.0 5.0
    afl_splicing_mutation 82800 18.0 1.000000 0.685994 0.0 1.0 1.0 1.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 5583.947368 113.293853 5504.0 5538.0 5553.0 5579.00 6026.0
    afl_splicing_mutation 82800 18.0 5557.333333 78.021867 5507.0 5521.0 5533.5 5561.25 5853.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libarchive_libarchive_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_splicing_mutation 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 3251.500000 207.418092 2922.0 3110.0 3219.5 3413.0 3606.0
    afl_splicing_mutation 82800 17.0 3190.588235 184.197265 2781.0 3099.0 3212.0 3307.0 3541.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libgit2_objects_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 1.421053 0.507257 1.0 1.0 1.0 2.0 2.0
    afl_splicing_mutation 82800 19.0 1.315789 0.477567 1.0 1.0 1.0 2.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 835.210526 1.474937 833.0 834.0 835.0 836.0 839.0
    afl_splicing_mutation 82800 19.0 835.684211 2.334586 833.0 834.0 835.0 836.5 841.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 24.125000 9.337202 1.0 24.75 27.5 29.0 31.0
    afl_splicing_mutation 82800 18.0 20.555556 11.273197 1.0 11.25 27.0 29.0 30.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 16.0 5507.625 2067.219288 555.0 5615.25 5776.0 6808.00 7649.0
    afl_splicing_mutation 82800 18.0 4609.500 2748.693342 555.0 1593.00 5745.0 6929.75 7499.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhtp_fuzz_htp summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3.823529 1.236694 1.0 3.0 4.0 5.0 5.0
    afl_splicing_mutation 82800 19.0 3.578947 1.304513 1.0 3.0 4.0 5.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 2986.470588 1.940285 2984.0 2985.0 2986.0 2988.0 2990.0
    afl_splicing_mutation 82800 19.0 2985.526316 2.674331 2982.0 2984.0 2985.0 2987.5 2991.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_libxml2_xml_reader_for_file_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 2.000000 0.000000 2.0 2.0 2.0 2.0 2.0
    afl_splicing_mutation 82800 15.0 2.733333 2.153624 2.0 2.0 2.0 2.0 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 15.0 14270.133333 426.842371 13458.0 14042.0 14183.0 14708.0 15034.0
    afl 82800 15.0 13556.866667 284.231763 12859.0 13401.5 13656.0 13788.0 13948.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 9.947368 1.268143 7.0 9.0 10.0 11.0 12.0
    afl_splicing_mutation 82800 17.0 9.941176 1.819017 7.0 9.0 10.0 11.0 14.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 17.0 4713.882353 93.910784 4516.0 4653.0 4733.0 4771.0 4935.0
    afl 82800 19.0 4703.473684 111.246657 4553.0 4635.0 4690.0 4732.5 4961.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby-2018-05-23 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 69300 5.0 2.4 0.894427 1.0 2.0 3.0 3.0 3.0
    afl_splicing_mutation 69300 15.0 3.0 1.069045 1.0 2.5 3.0 4.0 4.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 69300 15.0 8173.2 132.846958 7933.0 8108.5 8188.0 8242.0 8472.0
    afl 69300 5.0 7634.8 143.363873 7441.0 7573.0 7653.0 7675.0 7832.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

muparser_set_eval_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.055556 0.235702 0.0 0.0 0.0 0.0 1.0
    afl_splicing_mutation 82800 18.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 722.222222 2.602161 720.0 721.25 722.0 722.0 732.0
    afl_splicing_mutation 82800 18.0 722.611111 3.759520 720.0 721.00 722.0 723.0 737.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

njs_njs_process_script_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 2.000000 0.766965 1.0 2.0 2.0 2.0 4.0
    afl_splicing_mutation 82800 18.0 1.444444 0.615699 1.0 1.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 3538.555556 55.449205 3459.0 3492.50 3539.5 3564.75 3633.0
    afl_splicing_mutation 82800 18.0 3538.555556 55.539305 3441.0 3506.25 3529.0 3569.75 3673.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 19.0 3.894737 1.048530 1.0 3.5 4.0 4.0 6.0
    afl 82800 17.0 3.352941 1.169464 1.0 3.0 3.0 4.0 6.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 19.0 7758.526316 48.749209 7644.0 7741.5 7777.0 7789.0 7837.0
    afl 82800 17.0 7758.117647 36.202352 7694.0 7730.0 7771.0 7788.0 7805.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 1.263158 0.653376 0.0 1.0 1.0 1.5 3.0
    afl_splicing_mutation 82800 20.0 1.450000 0.686333 1.0 1.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 20.0 57852.250000 3730.026780 44778.0 56890.5 57636.0 59500.25 62444.0
    afl 82800 19.0 55140.105263 2695.360806 45607.0 54495.0 55484.0 56184.00 59094.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 17.0 3.941176 1.248529 2.0 3.00 4.0 5.00 6.0
    afl 82800 18.0 1.722222 1.637989 0.0 0.25 1.5 2.75 6.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 17.0 17049.470588 80.028993 16801.0 17023.0 17082.0 17097.00 17155.0
    afl 82800 18.0 17035.611111 58.657670 16917.0 17014.0 17039.5 17079.75 17122.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

poppler_pdf_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 18.0 14.888889 3.103866 11.0 12.25 14.5 16.75 20.0
    afl 82800 17.0 13.941176 1.560072 10.0 13.00 14.0 15.00 17.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 19409.235294 101.794234 19251.0 19369.00 19393.0 19431.00 19707.0
    afl_splicing_mutation 82800 18.0 19398.055556 86.065278 19290.0 19360.75 19379.0 19404.25 19645.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_standard_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    afl_splicing_mutation 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 132.0 0.0 132.0 132.0 132.0 132.0 132.0
    afl_splicing_mutation 82800 18.0 132.0 0.0 132.0 132.0 132.0 132.0 132.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

quickjs_eval-2020-01-05 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    afl_splicing_mutation 82800 18.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 8468.600000 341.893216 7877.0 8179.00 8602.0 8730.50 8872.0
    afl_splicing_mutation 82800 18.0 8574.111111 336.412368 8061.0 8304.25 8586.5 8771.75 9192.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 18.0 17.722222 3.083267 11.0 16.25 19.0 20.0 22.0
    afl 82800 19.0 17.368421 1.801559 13.0 16.50 17.0 19.0 21.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 18.0 1590.333333 84.100080 1487.0 1509.75 1615.0 1644.0 1796.0
    afl 82800 19.0 1562.263158 74.785502 1487.0 1496.00 1522.0 1641.5 1670.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

usrsctp_fuzzer_connect summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.052632 0.229416 0.0 0.0 0.0 0.0 1.0
    afl_splicing_mutation 82800 18.0 0.333333 0.485071 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 18.0 5280.500000 188.964749 5091.0 5152.75 5198.5 5418.75 5608.0
    afl 82800 19.0 5235.842105 178.865332 4909.0 5128.50 5171.0 5334.50 5546.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark_ip summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 3.470588 0.624264 3.0 3.0 3.0 4.0 5.0
    afl_splicing_mutation 82800 17.0 3.705882 0.848875 3.0 3.0 3.0 4.0 5.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 75024.823529 4088.044325 68599.0 72748.0 73792.0 75787.0 84071.0
    afl_splicing_mutation 82800 17.0 71322.117647 4656.518145 64916.0 67364.0 71368.0 74691.0 79061.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zstd_stream_decompress summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 18.0 0.055556 0.235702 0.0 0.0 0.0 0.0 1.0
    afl_splicing_mutation 82800 19.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl_splicing_mutation 82800 19.0 5052.052632 24.541289 4997.0 5036.50 5056.0 5066.50 5100.0
    afl 82800 18.0 4915.222222 181.524121 4565.0 4744.25 5005.5 5035.25 5090.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 40b2e384dd7d950c4ceb3b4f51eb08c8318e2a83

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout 40b2e384dd7d950c4ceb3b4f51eb08c8318e2a83
# Download the internal config file.
curl https://storage.googleapis.com/splicing/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data