[Dailydave] CVSS is the worst compression algorithm ever
Adam Shostack
adam at shostack.org
Thu Jan 10 22:21:55 UTC 2019
Okay, I'll respond generally about DREAD. The issue comes up when
people say "We'll treat a DREAD rating of >= 8 as critical." Then
someone looks at your discoverability of 7, and says "hmm, if this
were a 6, then DREAD would be 7.9...can we change it?" Lacking any
guidance on the difference, it's hard to say no.
Really, it's often "You're being unreasonable by making
discoverability a 7 here. It's not that high!"
Adam
On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote:
> I probably shouldn't have brought it up - I'm not involved much on the
> pentesting side. I know we've discussed replacing it, but finding little out
> there to replace it with.
>
> In my own work, I find most of my pentesting results come down to a binary
> value (hackable, not hackable) and some sense of likelihood of it getting
> exploited by a malicious party. Highs/mediums/lows all seem pointless when
> emulating the attacker perspective.
>
> Looking at DREAD, I honestly can't say I find anything fatally wrong with it.
> Perhaps it's because I've never known pentesting to be terribly consistent
> across tests or consultants in my career, so the bar is set pretty low in my
> mind?
>
> --Adrian
>
> On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam at shostack.org> wrote:
>
> On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
> > Our pentesters use DREAD, which I think most people have moved on from,
> but at
> > least the scoring is clear and consistent.
>
> I'm sorry, but I need to rant a little.
>
> A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
> Microsoft. If you are getting consistent scoring out of DREAD, you
> are not using DREAD (as described in Writing Secure Code 1, which I
> think is the first public description).
>
> You are using some derivitive that adds tools to provide for
> that consistency. Those tools may be as simple as a set of examples
> of each of the constiuents and what constitutes a 7 or a 3.
>
> I care about this because I one of the biggest things that I see
> making threat modeling hard is everyone calls their specific
> collection of techniques 'lightweight threat modeling' or 'agile
> threat modeling' and people trying to learn get confused because
> there's 6 contradictory descriptions that have been labeled "agile
> tm". People writing down process so their engineers can do it
> consistently get confused in the same way they'd get confused if we
> all said "oh yeah. we're writing code, and you can assign variables
> with either = or <=". We name our languages, we version them. We need
> to start doing the same for threat modeling constructs.
>
> If you say "We're using DREADNOP 1.0" that's cool. Alternately, maybe
> you're using DREAD 1.0 in its raw form, in which case, how are you
> getting consistent scores?
>
> Adam
>
>
> > In addition to CVE being wrong on critical details, I've found that most
> of
> > ExploitDB isn't exploits. Many are vulnerability checks and almost all
> are
> > incorrectly entered. PrivEsc will be labeled RCE and RCE will be labeled
> DoS.
> > It's all a mess. If I had the resources to burn it all down and start
> from
> > scratch, I'd do it.
> >
> > --Adrian
> >
> > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson at 126.com wrote:
> >
> > > They use a ton of big words in the paper to call CVSS out and give
> it a
> > shellacking. Like most of you, we have extensive use of CVSS in our
> > consulting practice and I've seen this stuff first hand. CVSS is of
> course
> > just a buggy compression algorithm for taking complex qualitative
> data and
> > then putting it on a number line.
> >
> >
> > Over the years I've worked at a few different consultancies and at
> least
> > originally basically no one used any sort of standardized metric, the
> > reports were generally humorous from a technical standpoint as the
> numbers
> > were basically just made up and didn't adhere to even basic
> statistics
> > methodologies-- we take the X and multiple it by Y and add the Z and
> > there's your score! Some even plotted them along cartoon looking
> graphs and
> > plots and my suspicion was that they were really included to give
> depth to
> > the reports and break up the monotony of text on text on text. Oddly,
> I've
> > never even worked at a place that described the methodology as
> outlined in
> > their reports to their employees leaving some question as to how a
> > methodology was to be implemented if only the client ever heard about
> it.
> >
> > In that sense, CVSS et al make some amount of sense, if nothing else
> it
> > standardizes to a common metric and isn't what the sales guy or
> operations
> > manager made up. Additionally, what a strange word-- shellacking,
> there is
> > no 'k' in the word until its made into a present participle.
> >
> > > The paper has three angles here:
> > > Qualitative mappings into quantitative numbers are a silly thing to
> do,
> > like people trying to do "social science" by using SurveyMonkey.
> >
> > Which is what most people are or were selling.
> >
> > > It's fine to have a lossy compression algorithm that emphasizes
> certain
> > aspects of the input signal over others, of course, but an additional
> CERT/
> > CC critique is we have no reason to think CVSS does this in any
> useful way.
> >
> > Well there 's a missing line here, you can see it from the way that
> > client-side attacks perverted the concept of remote and so they made
> them
> > remote also instead of adding the new line to the plot. Because of
> stuff
> > like this. everything is remote now which limits its usefulness. This
> > doesn't even touch on the fact that most of the CVE database is
> basically
> > wrong from submissions including very limited data, id est "memory
> > corruption results in a DoS".
> >
> > Nathaniel
> >
> > 在 2019-01-09 00:14:00,"Dave Aitel" <dave.aitel at cyxtera.com> 写道:
> >
> >
> > I wanted to take a few minutes and do a quick highlight of a
> paper from
> > CMU-CERT which I think most people have missed out on: https://
> > resources.sei.cmu.edu/asset_files/WhitePaper/
> 2018_019_001_538372.pdf
> >
> > Towards Improving CVSS - resources.sei.cmu.edu
> > resources.sei.cmu.edu
> > SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
> > REV-03.18.2016.0 Distribution Statement A: Approved for Public
> Release;
> > Distribution Is Unlimited TOWARDS IMPROVING CVSS
> > It's almost as funny a read as their previous best work on how "
> > clientless HTTPS VPNs are insanely dumb what were you thinking
> omg?"
> >
> > They use a ton of big words in the paper to call CVSS out and
> give it a
> > shellacking. Like most of you, we have extensive use of CVSS in
> our
> > consulting practice and I've seen this stuff first hand. CVSS is
> of
> > course just a buggy compression algorithm for taking complex
> > qualitative data and then putting it on a number line. The paper
> has
> > three angles here:
> > 1. Qualitative mappings into quantitative numbers are a silly
> thing to
> > do, like people trying to do "social science" by using
> > SurveyMonkey.
> > 2. We're pretty sure that the compression algorithm is not, in
> fact,
> > putting higher risk items as bigger numbers, which is the
> whole
> > point of the thing.
> > 3. Nobody is applying this in any sort of consistent way (which
> is
> > probably impossible) which is ALSO the whole point of the
> thing.
> >
> > It's fine to have a lossy compression algorithm that emphasizes
> certain
> > aspects of the input signal over others, of course, but an
> additional
> > CERT/CC critique is we have no reason to think CVSS does this in
> any
> > useful way.
> >
> >
> > There's definitely people in the CVSS process (who I will avoid
> calling
> > out by name) who think ANY quantization is good. But read the
> paper and
> > decide for yourself - because these are probably serious issues
> that
> > are turning your entire risk org into a Garbage-In-Garbage-Out
> org...
> >
> >
> > -dave
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Dailydave mailing list
> > Dailydave at lists.immunityinc.com
> > https://lists.immunityinc.com/mailman/listinfo/dailydave
> >
>
> > _______________________________________________
> > Dailydave mailing list
> > Dailydave at lists.immunityinc.com
> > https://lists.immunityinc.com/mailman/listinfo/dailydave
>
>
> --
> Adam Shostack
> President, Shostack & Associates
> https://associates.shostack.org • +1 917 391 2168
>
> Join my very quiet annnoucement list: https://adam.shostack.org/newthing
>
>
--
Adam Shostack
President, Shostack & Associates
https://associates.shostack.org • +1 917 391 2168
Join my very quiet annnoucement list: https://adam.shostack.org/newthing
More information about the Dailydave
mailing list