[Dailydave] CVSS is the worst compression algorithm ever
Adrian Sanabria
adrian.sanabria at gmail.com
Fri Jan 11 05:23:24 UTC 2019
Everywhere I've ever pentested, we've used a low/medium/high or
low/medium/high/critical scale - this is my first encounter with DREAD.
What you describe though - clients attempting to negotiate down the
severity of vulns on the report - was common regardless of the scoring
system used. I don't see DREAD being unique in that respect.
Reflecting, it's probably what pushed me towards the binary system I ended
up using. No score - just exploitable or not. Then, if it's someone that
just refuses to have a critical finding on their report, they'll argue over
the likelihood of it being exploited. Or over whether the finding was in
scope. Then I consider firing that client.
I try to explain - when you're hiring a pentester, you're paying for their
opinion as they perform a point-in-time assessment. The client is free to
debate the findings and they don't have to agree with the consultant, but
the client can't change/rewrite the report without destroying the integrity
of the report.
--Adrian
On Thu, Jan 10, 2019 at 5:21 PM Adam Shostack <adam at shostack.org> wrote:
> Okay, I'll respond generally about DREAD. The issue comes up when
> people say "We'll treat a DREAD rating of >= 8 as critical." Then
> someone looks at your discoverability of 7, and says "hmm, if this
> were a 6, then DREAD would be 7.9...can we change it?" Lacking any
> guidance on the difference, it's hard to say no.
>
> Really, it's often "You're being unreasonable by making
> discoverability a 7 here. It's not that high!"
>
> Adam
>
>
>
> On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote:
> > I probably shouldn't have brought it up - I'm not involved much on the
> > pentesting side. I know we've discussed replacing it, but finding little
> out
> > there to replace it with.
> >
> > In my own work, I find most of my pentesting results come down to a
> binary
> > value (hackable, not hackable) and some sense of likelihood of it getting
> > exploited by a malicious party. Highs/mediums/lows all seem pointless
> when
> > emulating the attacker perspective.
> >
> > Looking at DREAD, I honestly can't say I find anything fatally wrong
> with it.
> > Perhaps it's because I've never known pentesting to be terribly
> consistent
> > across tests or consultants in my career, so the bar is set pretty low
> in my
> > mind?
> >
> > --Adrian
> >
> > On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam at shostack.org> wrote:
> >
> > On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
> > > Our pentesters use DREAD, which I think most people have moved on
> from,
> > but at
> > > least the scoring is clear and consistent.
> >
> > I'm sorry, but I need to rant a little.
> >
> > A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
> > Microsoft. If you are getting consistent scoring out of DREAD, you
> > are not using DREAD (as described in Writing Secure Code 1, which I
> > think is the first public description).
> >
> > You are using some derivitive that adds tools to provide for
> > that consistency. Those tools may be as simple as a set of examples
> > of each of the constiuents and what constitutes a 7 or a 3.
> >
> > I care about this because I one of the biggest things that I see
> > making threat modeling hard is everyone calls their specific
> > collection of techniques 'lightweight threat modeling' or 'agile
> > threat modeling' and people trying to learn get confused because
> > there's 6 contradictory descriptions that have been labeled "agile
> > tm". People writing down process so their engineers can do it
> > consistently get confused in the same way they'd get confused if we
> > all said "oh yeah. we're writing code, and you can assign variables
> > with either = or <=". We name our languages, we version them. We
> need
> > to start doing the same for threat modeling constructs.
> >
> > If you say "We're using DREADNOP 1.0" that's cool. Alternately,
> maybe
> > you're using DREAD 1.0 in its raw form, in which case, how are you
> > getting consistent scores?
> >
> > Adam
> >
> >
> > > In addition to CVE being wrong on critical details, I've found
> that most
> > of
> > > ExploitDB isn't exploits. Many are vulnerability checks and almost
> all
> > are
> > > incorrectly entered. PrivEsc will be labeled RCE and RCE will be
> labeled
> > DoS.
> > > It's all a mess. If I had the resources to burn it all down and
> start
> > from
> > > scratch, I'd do it.
> > >
> > > --Adrian
> > >
> > > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson at 126.com
> wrote:
> > >
> > > > They use a ton of big words in the paper to call CVSS out
> and give
> > it a
> > > shellacking. Like most of you, we have extensive use of CVSS
> in our
> > > consulting practice and I've seen this stuff first hand. CVSS
> is of
> > course
> > > just a buggy compression algorithm for taking complex
> qualitative
> > data and
> > > then putting it on a number line.
> > >
> > >
> > > Over the years I've worked at a few different consultancies
> and at
> > least
> > > originally basically no one used any sort of standardized
> metric, the
> > > reports were generally humorous from a technical standpoint as
> the
> > numbers
> > > were basically just made up and didn't adhere to even basic
> > statistics
> > > methodologies-- we take the X and multiple it by Y and add the
> Z and
> > > there's your score! Some even plotted them along cartoon
> looking
> > graphs and
> > > plots and my suspicion was that they were really included to
> give
> > depth to
> > > the reports and break up the monotony of text on text on text.
> Oddly,
> > I've
> > > never even worked at a place that described the methodology as
> > outlined in
> > > their reports to their employees leaving some question as to
> how a
> > > methodology was to be implemented if only the client ever
> heard about
> > it.
> > >
> > > In that sense, CVSS et al make some amount of sense, if
> nothing else
> > it
> > > standardizes to a common metric and isn't what the sales guy or
> > operations
> > > manager made up. Additionally, what a strange word--
> shellacking,
> > there is
> > > no 'k' in the word until its made into a present participle.
> > >
> > > > The paper has three angles here:
> > > > Qualitative mappings into quantitative numbers are a silly
> thing to
> > do,
> > > like people trying to do "social science" by using
> SurveyMonkey.
> > >
> > > Which is what most people are or were selling.
> > >
> > > > It's fine to have a lossy compression algorithm that
> emphasizes
> > certain
> > > aspects of the input signal over others, of course, but an
> additional
> > CERT/
> > > CC critique is we have no reason to think CVSS does this in any
> > useful way.
> > >
> > > Well there 's a missing line here, you can see it from the way
> that
> > > client-side attacks perverted the concept of remote and so
> they made
> > them
> > > remote also instead of adding the new line to the plot.
> Because of
> > stuff
> > > like this. everything is remote now which limits its
> usefulness. This
> > > doesn't even touch on the fact that most of the CVE database is
> > basically
> > > wrong from submissions including very limited data, id est
> "memory
> > > corruption results in a DoS".
> > >
> > > Nathaniel
> > >
> > > 在 2019-01-09 00:14:00,"Dave Aitel" <dave.aitel at cyxtera.com>
> 写道:
> > >
> > >
> > > I wanted to take a few minutes and do a quick highlight of
> a
> > paper from
> > > CMU-CERT which I think most people have missed out
> on: https://
> > > resources.sei.cmu.edu/asset_files/WhitePaper/
> > 2018_019_001_538372.pdf
> > >
> > > Towards Improving CVSS - resources.sei.cmu.edu
> > > resources.sei.cmu.edu
> > > SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
> > > REV-03.18.2016.0 Distribution Statement A: Approved for
> Public
> > Release;
> > > Distribution Is Unlimited TOWARDS IMPROVING CVSS
> > > It's almost as funny a read as their previous best work on
> how "
> > > clientless HTTPS VPNs are insanely dumb what were you
> thinking
> > omg?"
> > >
> > > They use a ton of big words in the paper to call CVSS out
> and
> > give it a
> > > shellacking. Like most of you, we have extensive use of
> CVSS in
> > our
> > > consulting practice and I've seen this stuff first hand.
> CVSS is
> > of
> > > course just a buggy compression algorithm for taking
> complex
> > > qualitative data and then putting it on a number line. The
> paper
> > has
> > > three angles here:
> > > 1. Qualitative mappings into quantitative numbers are a
> silly
> > thing to
> > > do, like people trying to do "social science" by using
> > > SurveyMonkey.
> > > 2. We're pretty sure that the compression algorithm is
> not, in
> > fact,
> > > putting higher risk items as bigger numbers, which is
> the
> > whole
> > > point of the thing.
> > > 3. Nobody is applying this in any sort of consistent way
> (which
> > is
> > > probably impossible) which is ALSO the whole point of
> the
> > thing.
> > >
> > > It's fine to have a lossy compression algorithm that
> emphasizes
> > certain
> > > aspects of the input signal over others, of course, but an
> > additional
> > > CERT/CC critique is we have no reason to think CVSS does
> this in
> > any
> > > useful way.
> > >
> > >
> > > There's definitely people in the CVSS process (who I will
> avoid
> > calling
> > > out by name) who think ANY quantization is good. But read
> the
> > paper and
> > > decide for yourself - because these are probably serious
> issues
> > that
> > > are turning your entire risk org into a
> Garbage-In-Garbage-Out
> > org...
> > >
> > >
> > > -dave
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Dailydave mailing list
> > > Dailydave at lists.immunityinc.com
> > > https://lists.immunityinc.com/mailman/listinfo/dailydave
> > >
> >
> > > _______________________________________________
> > > Dailydave mailing list
> > > Dailydave at lists.immunityinc.com
> > > https://lists.immunityinc.com/mailman/listinfo/dailydave
> >
> >
> > --
> > Adam Shostack
> > President, Shostack & Associates
> > https://associates.shostack.org • +1 917 391 2168
> >
> > Join my very quiet annnoucement list:
> https://adam.shostack.org/newthing
> >
> >
>
> --
> Adam Shostack
> President, Shostack & Associates
> https://associates.shostack.org • +1 917 391 2168
>
> Join my very quiet annnoucement list: https://adam.shostack.org/newthing
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.immunityinc.com/pipermail/dailydave/attachments/20190111/6c450e1f/attachment-0001.html>
More information about the Dailydave
mailing list