<div dir="ltr"><span id="gmail-docs-internal-guid-03dd8d26-24e6-a951-453b-88ef375b2baa"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">I don't think it's an apples-to-oranges comparison to compare these fuzzers against the Cyber Grand Challenge test set (<a href="https://github.com/trailofbits/cb-multios">https://github.com/trailofbits/cb-multios</a>). In fact, the CGC test set is a perfect shooting gallery. The test set is entirely comprised of network services that implement protocols that represent real world software. DECREE has no knowledge of file systems or files at all. The protocols are frequently simplified, but over the 241 challenges you can make plenty of fair comparisons.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">There are challenges that implement:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Nonces</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">* <a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/NoHiC">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/NoHiC</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">* <a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/Griswold">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/Griswold</a>)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Checksums</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">* <a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/ValveChecks">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/ValveChecks</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">* <a href="https://github.com/trailofbits/cb-multios/blob/master/original-challenges/Packet_Receiver">https://github.com/trailofbits/cb-multios/blob/master/original-challenges/Packet_Receiver</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">RSA-like authentication</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">* <a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/FASTLANE">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/FASTLANE</a> </span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Several challenges are "in spirit" re-implementations of vulnerabilities that were publicly exploited, for example:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">• Morris Worm (<a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_1--Hat_Trick--Morris_Worm">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_1--Hat_Trick--Morris_Worm</a>)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">• Crackaddr (<a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_2--Mail_Server--Crackaddr">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_2--Mail_Server--Crackaddr</a>)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">• Stuxnet LNK (<a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_5--File_Explorer--LNK_Bug">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_5--File_Explorer--LNK_Bug</a>)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">• Heartbleed (<a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_6--Secure_Server--Heartbleed">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_6--Secure_Server--Heartbleed</a>)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">• Linux FUTEX / TowelRoot (<a href="https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REDPILL">https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REDPILL</a>)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">It's true that CGC's ABI was simplified to make this analysis more feasible, but it's still expressive enough to support complicated real-world vulnerabilities. Many of these challenges were solved in competition, although we're still waiting for someone to do the analysis and present it publicly. Anecdotally, the Trail of Bits CRS can generate crashing inputs for ~50% of these challenges, including ones with nonces and checksums.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">In the next few months, I will compare both static and dynamic analysis tools using this software repository. We (Trail of Bits) hope to release repeatable DevOps-style test results. You're absolutely right about optimizing fuzzers for your target and I hope to have statistics soon to back up that claim. Some of my initial results will be presented at <a href="http://inbot.xyz">inbot.xyz</a> at the end of September.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Cheers,</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Ryan</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">@withzombies</span></p><br></span></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 13, 2016 at 11:33 AM, Dave Aitel <span dir="ltr"><<a href="mailto:dave.aitel@gmail.com" target="_blank">dave.aitel@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So let's take a quick break from thinking about how messed up Wassenaar is or what random annoying thing the EFF or ACLU said about 0day today and talk about fuzzers. AFL has everyone's mind share, but I you have to point out that it is still a VERY specialized tool. <div><br></div><div>The process of taking a file, sending it into some processing unit, and then figuring out if it crashes, sounds easy and generic. But in practice you have to carefully optimize how you do it to get any kind of speed and effectiveness out of it. </div><div><br></div><div>This is another thing about the Cyber Grand Challenge: I think they optimized the problem set in a way using that limited system call VM for AFL-like fuzzers. I'm just going to assume none of the problem sets were a complex RPC-like protocol, because we would have seen zero people solve them and DARPA knows that.</div><div><br></div><div>What I mean is this: It is very hard to optimize the block-based fuzzing technique for automation. But they solve two completely different types of problems. </div><div><br></div><div>AFL-like fuzzers excel at files for one reason: Files don't do computation. SPIKE-like fuzzers excel at protocols because they are there to handle challenge responses, size-fields, checksums, encryption, and other things common in network protocols. There's also minor differences in how they handle mutation. And of course, in many cases a SPIKE-like fuzzer is EASIER to set up and use than something like AFL, with less problem-optimization needed for valuable results.</div><div><br></div><div>But still, no comparison of a file-fuzzer to a block-based or protocol fuzzer (PEACH/SPIKE/CODENOMICON) is going to be apples to apples. It's more like apples to dragons.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-dave</div><div> </div></font></span></div>
<br>______________________________<wbr>_________________<br>
Dailydave mailing list<br>
<a href="mailto:Dailydave@lists.immunityinc.com">Dailydave@lists.immunityinc.<wbr>com</a><br>
<a href="https://lists.immunityinc.com/mailman/listinfo/dailydave" rel="noreferrer" target="_blank">https://lists.immunityinc.com/<wbr>mailman/listinfo/dailydave</a><br>
<br></blockquote></div><br></div>