[Dailydave] CVSS is the worst compression algorithm ever

Adam Shostack adam at shostack.org
Thu Jan 10 22:21:55 UTC 2019


Okay, I'll respond generally about DREAD.  The issue comes up when
people say "We'll treat a DREAD rating of >= 8 as critical."  Then
someone looks at your discoverability of 7, and says "hmm, if this
were a 6, then DREAD would be 7.9...can we change it?"  Lacking any
guidance on the difference, it's hard to say no.

Really, it's often "You're being unreasonable by making
discoverability a 7 here.  It's not that high!"

Adam



On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote:
> I probably shouldn't have brought it up - I'm not involved much on the
> pentesting side. I know we've discussed replacing it, but finding little out
> there to replace it with.
> 
> In my own work, I find most of my pentesting results come down to a binary
> value (hackable, not hackable) and some sense of likelihood of it getting
> exploited by a malicious party. Highs/mediums/lows all seem pointless when
> emulating the attacker perspective.
> 
> Looking at DREAD, I honestly can't say I find anything fatally wrong with it.
> Perhaps it's because I've never known pentesting to be terribly consistent
> across tests or consultants in my career, so the bar is set pretty low in my
> mind?
> 
> --Adrian
> 
> On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam at shostack.org> wrote:
> 
>     On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
>     > Our pentesters use DREAD, which I think most people have moved on from,
>     but at
>     > least the scoring is clear and consistent. 
> 
>     I'm sorry, but I need to rant a little.
> 
>     A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
>     Microsoft.  If you are getting consistent scoring out of DREAD, you
>     are not using DREAD (as described in Writing Secure Code 1, which I
>     think is the first public description).
> 
>     You are using some derivitive that adds tools to provide for
>     that consistency.  Those tools may be as simple as a set of examples
>     of each of the constiuents and what constitutes a 7 or a 3.
> 
>     I care about this because I one of the biggest things that I see
>     making threat modeling hard is everyone calls their specific
>     collection of techniques 'lightweight threat modeling' or 'agile
>     threat modeling' and people trying to learn get confused because
>     there's 6 contradictory descriptions that have been labeled "agile
>     tm".  People writing down process so their engineers can do it
>     consistently get confused in the same way they'd get confused if we
>     all said "oh yeah. we're writing code, and you can assign variables
>     with either = or <=".  We name our languages, we version them. We need
>     to start doing the same for threat modeling constructs.
> 
>     If you say "We're using DREADNOP 1.0" that's cool.  Alternately, maybe
>     you're using DREAD 1.0 in its raw form, in which case, how are you
>     getting consistent scores?
> 
>     Adam
> 
> 
>     > In addition to CVE being wrong on critical details, I've found that most
>     of
>     > ExploitDB isn't exploits. Many are vulnerability checks and almost all
>     are
>     > incorrectly entered. PrivEsc will be labeled RCE and RCE will be labeled
>     DoS.
>     > It's all a mess. If I had the resources to burn it all down and start
>     from
>     > scratch, I'd do it. 
>     >
>     > --Adrian
>     >
>     > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson at 126.com wrote:
>     >
>     >     > They use a ton of big words in the paper to call CVSS out and give
>     it a
>     >     shellacking. Like most of you, we have extensive use of CVSS in our
>     >     consulting practice and I've seen this stuff first hand. CVSS is of
>     course
>     >     just a buggy compression algorithm for taking complex qualitative
>     data and
>     >     then putting it on a number line.
>     >
>     >
>     >     Over the years I've worked at a few different consultancies and at
>     least
>     >     originally basically no one used any sort of standardized metric, the
>     >     reports were generally humorous from a technical standpoint as the
>     numbers
>     >     were basically just made up and didn't adhere to even basic
>     statistics
>     >     methodologies-- we take the X and multiple it by Y and add the Z and
>     >     there's your score! Some even plotted them along cartoon looking
>     graphs and
>     >     plots and my suspicion was that they were really included to give
>     depth to
>     >     the reports and break up the monotony of text on text on text. Oddly,
>     I've
>     >     never even worked at a place that described the methodology as
>     outlined in
>     >     their reports to their employees leaving some question as to how a
>     >     methodology was to be implemented if only the client ever heard about
>     it.
>     >
>     >     In that sense, CVSS et al make some amount of sense, if nothing else
>     it
>     >     standardizes to a common metric and isn't what the sales guy or
>     operations
>     >     manager made up. Additionally, what a strange word-- shellacking,
>     there is
>     >     no 'k' in the word until its made into a present participle.
>     >
>     >     > The paper has three angles here:
>     >     > Qualitative mappings into quantitative numbers are a silly thing to
>     do,
>     >     like people trying to do "social science" by using SurveyMonkey.
>     >
>     >     Which is what most people are or were selling.
>     >
>     >     > It's fine to have a lossy compression algorithm that emphasizes
>     certain
>     >     aspects of the input signal over others, of course, but an additional
>     CERT/
>     >     CC critique is we have no reason to think CVSS does this in any
>     useful way.
>     >
>     >     Well there 's a missing line here, you can see it from the way that
>     >     client-side attacks perverted the concept of remote and so they made
>     them
>     >     remote also instead of adding the new line to the plot. Because of
>     stuff
>     >     like this. everything is remote now which limits its usefulness. This
>     >     doesn't even touch on the fact that most of the CVE database is
>     basically
>     >     wrong from submissions including very limited data, id est "memory
>     >     corruption results in a DoS".
>     >
>     >     Nathaniel
>     >
>     >     在 2019-01-09 00:14:00,"Dave Aitel" <dave.aitel at cyxtera.com> 写道:
>     >
>     >       
>     >         I wanted to take a few minutes and do a quick highlight of a
>     paper from
>     >         CMU-CERT which I think most people have missed out on: https://
>     >         resources.sei.cmu.edu/asset_files/WhitePaper/
>     2018_019_001_538372.pdf
>     >
>     >         Towards Improving CVSS - resources.sei.cmu.edu
>     >         resources.sei.cmu.edu
>     >         SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
>     >         REV-03.18.2016.0 Distribution Statement A: Approved for Public
>     Release;
>     >         Distribution Is Unlimited TOWARDS IMPROVING CVSS
>     >         It's almost as funny a read as their previous best work on how "
>     >         clientless HTTPS VPNs are insanely dumb what were you thinking
>     omg?"
>     >
>     >         They use a ton of big words in the paper to call CVSS out and
>     give it a
>     >         shellacking. Like most of you, we have extensive use of CVSS in
>     our
>     >         consulting practice and I've seen this stuff first hand. CVSS is
>     of
>     >         course just a buggy compression algorithm for taking complex
>     >         qualitative data and then putting it on a number line. The paper
>     has
>     >         three angles here: 
>     >          1. Qualitative mappings into quantitative numbers are a silly
>     thing to
>     >             do, like people trying to do "social science" by using
>     >             SurveyMonkey.
>     >          2. We're pretty sure that the compression algorithm is not, in
>     fact,
>     >             putting higher risk items as bigger numbers, which is the
>     whole
>     >             point of the thing.  
>     >          3. Nobody is applying this in any sort of consistent way (which
>     is
>     >             probably impossible) which is ALSO the whole point of the
>     thing.
>     >
>     >         It's fine to have a lossy compression algorithm that emphasizes
>     certain
>     >         aspects of the input signal over others, of course, but an
>     additional
>     >         CERT/CC critique is we have no reason to think CVSS does this in
>     any
>     >         useful way. 
>     >
>     >
>     >         There's definitely people in the CVSS process (who I will avoid
>     calling
>     >         out by name) who think ANY quantization is good. But read the
>     paper and
>     >         decide for yourself - because these are probably serious issues
>     that
>     >         are turning your entire risk org into a Garbage-In-Garbage-Out
>     org...
>     >
>     >
>     >         -dave
>     >
>     >
>     >
>     >
>     >
>     >      
>     >
>     >     _______________________________________________
>     >     Dailydave mailing list
>     >     Dailydave at lists.immunityinc.com
>     >     https://lists.immunityinc.com/mailman/listinfo/dailydave
>     >
> 
>     > _______________________________________________
>     > Dailydave mailing list
>     > Dailydave at lists.immunityinc.com
>     > https://lists.immunityinc.com/mailman/listinfo/dailydave
> 
> 
>     --
>     Adam Shostack
>     President, Shostack & Associates
>     https://associates.shostack.org • +1 917 391 2168
> 
>     Join my very quiet annnoucement list: https://adam.shostack.org/newthing
> 
> 

-- 
Adam Shostack
President, Shostack & Associates
https://associates.shostack.org • +1 917 391 2168

Join my very quiet annnoucement list: https://adam.shostack.org/newthing



More information about the Dailydave mailing list