<div dir="ltr">I figured I'd chime in as someone who builds security machine learning models as part of his day job. A few hopefully not-too-incongruous observations:<div><br></div><div>1) Most security problems are not machine learning problems. Like encryption, dual-factor authentication, taint analysis, or hand-crafted IOCs, machine learning is just one of many security tools. But somehow people outside of machine learning seem to think a) machine learning can be applied everywhere and replace every other approach or b) machine learning can be applied nowhere, always underperforms, and is marketing snake oil. The people who believe a) are bound to be disappointed and the people who believe b) are bound to be blindsided when they wake up and realize machine learning has become an important ingredient in the network defense landscape.</div><div><br></div><div>2) For a working security data scientist, much of the ingenuity to developing a successful machine learning product is in picking problems that <i>are</i> good machine learning problems and not going down the rabbit hole of problems that aren't. Unsupervised clustering of malware to help identify new malware families or link threat actors -- that's a good problem, and systems that do this are currently deployed to good effect, but can probably be improved upon. Detecting and classifying malware is another good one that's already been productized but merits continued research. Setting firewall policy or predicting which users on a network will commit treason or sell your trade secrets is not a good machine learning problem and probably won't be in the foreseeable future, even though I'll bet there are products on the market that claim to do these things.</div><div><br></div><div>3) For a problem to be a good security machine learning problem you need a continuously replenished source of good data, because security models go out of date as adversaries evolve if the models don't evolve along with them. If you don't have good data at scale (and this includes <i>ground truth</i> with respect to this data) machine learning is the wrong approach. For example, because we don't have thousands of examples of employees going rogue and selling trade secrets (at least I don't) a machine learning approach to detecting such employees doesn't make sense.</div><div><br></div><div>4) To echo what Sven said, custom modeling for a given security application, which involves mostly either feature engineering or custom crafting of deep learning models that automate a portion of the feature engineering process, is the main work of a security data scientist. In my experience, wholesale adoption of approaches from other fields never works. For one, the statistics of the problem are totally different: in the detection use case, we tend only to care about the performance of a model in the extremely low false positive rate region, which changes the modeling goals from many non-security applications. And secondly, security is just different from computer vision, text mining, etc., and in my experience requires custom solutions to perform well. </div><div><br></div><div>Best,</div><div>Josh</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 1, 2016 at 9:59 PM, <span dir="ltr"><<a href="mailto:Robin.Lowe@forces.gc.ca" target="_blank">Robin.Lowe@forces.gc.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Good day all,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Just a couple things I thought of while reading the earlier discussion on AI and this follow-up email. Just some, as Chris so eloquently put it earlier, conversation
fodder.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I think one thing we have to keep in mind is that the underlying framework behind machine learning is still a machine. An issue I can see about this is who
is accountable for if it fails? If we’re talking about national security, what’s the risk that someone will be willing to take on in order to prove that their new machine learning intrusion detection system works 100% of the time? The number of hours that
would be required to amass the amount of data needed to seed the system would be substantial, even on its own.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">There’s also the possibility of false positives being generated by erroneous data. Sure, an listening meterpreter shell on port 4444 is pretty damn obvious,
but what about, say, Cobalt Strike’s Beacon system? Will the people developing the IDS need to spend thousands of dollars throwing all of these expensive network auditing programs at it in order to generate the data necessary to make it accurate even 90% of
the time?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Also, the budget just for personnel would be pretty high. You’d need people in R&D, maintenance, actually checking flagged intrusion attempts, etc.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">One last thing before I start in on the possible positives is that the machine itself might be prone to exploitation. Similar to how getting into domain controllers
and hypervisors are pretty much endgame states, what if you broke into the IDS itself and started messing with its signatures? Seems like a few things to think about.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">However, some cost-reducing factors are that it’s always looking. And faster than a person can. Sure, there are some blue teams that are basically machines
at this point, I can definitely see a time where machines can take over that facet of security.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">You don’t have to pay it a salary, just keep the machine happy with electricity and known behaviours and it’ll chug along.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kind of starting to sound like an antivirus program but one that looks at networks instead of files.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">New to this sort of thing so sorry if I mentioned something that would be considered common knowledge or just plain nonsense.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">Leading Seaman/Matelot de 1re classe Robin Lowe<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">Naval Communicator, HMCS EDMONTON<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">Department of National Defence / Government of Canada<u></u><u></u></span></p>
<p class="MsoNormal"><u><span lang="FR-CA" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue"><a href="mailto:Robin.Lowe@forces.gc.ca" target="_blank">Robin.Lowe@forces.gc.ca</a></span></u><span lang="FR-CA" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">
/ Tel: <a href="tel:250-363-7940" value="+12503637940" target="_blank">250-363-7940</a><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CA" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="FR-CA" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">Communicateur Naval, NCSM EDMONTON<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CA" style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#0d0d0d">Ministère de la Défense nationale / Gouvernement du Canada<u></u><u></u></span></p>
<p class="MsoNormal"><u><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:blue"><a href="mailto:Robin.Lowe@forces.gc.ca" target="_blank">Robin.Lowe@forces.gc.ca</a></span></u><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0d0d0d">
/ Tel: <a href="tel:250-363-7940" value="+12503637940" target="_blank">250-363-7940</a></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><br>
</span><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#4f81bd">“</span></i><i><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#4f81bd">The quieter you are, the more you are able to hear.”</span></i><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></i></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:dailydave-bounces@lists.immunityinc.com" target="_blank">dailydave-bounces@lists.immunityinc.com</a> [mailto:<a href="mailto:dailydave-bounces@lists.immunityinc.com" target="_blank">dailydave-bounces@lists.immunityinc.com</a>]
<b>On Behalf Of </b>Dave Aitel<br>
<b>Sent:</b> April-01-16 11:36 AM<br>
<b>To:</b> <a href="mailto:dailydave@lists.immunityinc.com" target="_blank">dailydave@lists.immunityinc.com</a><br>
<b>Subject:</b> [Dailydave] Assymetry<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">One possible long-lasting cause of the "asymmetry" everyone talks about is that US defenders get quite high salaries compared to Chinese attackers (I assume, not being a Chinese attacker it's hard to know for sure).<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just in pure "dollars spent vs dollars spent" it seems like it would be three times cheaper to be a Chinese attacker at that rate?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">But I think it's still a question whether or not machine learning techniques make surveillance cheaper than intrusion as a rule. What if it does? What would that change about our national strategy? (And if it DOESN'T then why bother?)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-dave<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Dailydave mailing list<br>
<a href="mailto:Dailydave@lists.immunityinc.com">Dailydave@lists.immunityinc.com</a><br>
<a href="https://lists.immunityinc.com/mailman/listinfo/dailydave" rel="noreferrer" target="_blank">https://lists.immunityinc.com/mailman/listinfo/dailydave</a><br>
<br></blockquote></div><br></div>