[Dailydave] CVSS is the worst compression algorithm ever

Thu Jan 10 21:38:30 UTC 2019

I probably shouldn't have brought it up - I'm not involved much on the
pentesting side. I know we've discussed replacing it, but finding little
out there to replace it with.

In my own work, I find most of my pentesting results come down to a binary
value (hackable, not hackable) and some sense of likelihood of it getting
exploited by a malicious party. Highs/mediums/lows all seem pointless when
emulating the attacker perspective.

Looking at DREAD, I honestly can't say I find anything fatally wrong with
it. Perhaps it's because I've never known pentesting to be terribly
consistent across tests or consultants in my career, so the bar is set
pretty low in my mind?

--Adrian

On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam at shostack.org> wrote:

> On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
> > Our pentesters use DREAD, which I think most people have moved on from,
> but at
> > least the scoring is clear and consistent.
>
> I'm sorry, but I need to rant a little.
>
> A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
> Microsoft.  If you are getting consistent scoring out of DREAD, you
> are not using DREAD (as described in Writing Secure Code 1, which I
> think is the first public description).
>
> You are using some derivitive that adds tools to provide for
> that consistency.  Those tools may be as simple as a set of examples
> of each of the constiuents and what constitutes a 7 or a 3.
>
> I care about this because I one of the biggest things that I see
> making threat modeling hard is everyone calls their specific
> collection of techniques 'lightweight threat modeling' or 'agile
> threat modeling' and people trying to learn get confused because
> there's 6 contradictory descriptions that have been labeled "agile
> tm".  People writing down process so their engineers can do it
> consistently get confused in the same way they'd get confused if we
> all said "oh yeah. we're writing code, and you can assign variables
> with either = or <=".  We name our languages, we version them. We need
> to start doing the same for threat modeling constructs.
>
> If you say "We're using DREADNOP 1.0" that's cool.  Alternately, maybe
> you're using DREAD 1.0 in its raw form, in which case, how are you
> getting consistent scores?
>
> Adam
>
>
> > In addition to CVE being wrong on critical details, I've found that most
> of
> > ExploitDB isn't exploits. Many are vulnerability checks and almost all
> are
> > incorrectly entered. PrivEsc will be labeled RCE and RCE will be labeled
> DoS.
> > It's all a mess. If I had the resources to burn it all down and start
> from
> > scratch, I'd do it.
> >
> > --Adrian
> >
> > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson at 126.com wrote:
> >
> >     > They use a ton of big words in the paper to call CVSS out and give
> it a
> >     shellacking. Like most of you, we have extensive use of CVSS in our
> >     consulting practice and I've seen this stuff first hand. CVSS is of
> course
> >     just a buggy compression algorithm for taking complex qualitative
> data and
> >     then putting it on a number line.
> >
> >
> >     Over the years I've worked at a few different consultancies and at
> least
> >     originally basically no one used any sort of standardized metric, the
> >     reports were generally humorous from a technical standpoint as the
> numbers
> >     were basically just made up and didn't adhere to even basic
> statistics
> >     methodologies-- we take the X and multiple it by Y and add the Z and
> >     there's your score! Some even plotted them along cartoon looking
> graphs and
> >     plots and my suspicion was that they were really included to give
> depth to
> >     the reports and break up the monotony of text on text on text.
> Oddly, I've
> >     never even worked at a place that described the methodology as
> outlined in
> >     their reports to their employees leaving some question as to how a
> >     methodology was to be implemented if only the client ever heard
> about it.
> >
> >     In that sense, CVSS et al make some amount of sense, if nothing else
> it
> >     standardizes to a common metric and isn't what the sales guy or
> operations
> >     manager made up. Additionally, what a strange word-- shellacking,
> there is
> >     no 'k' in the word until its made into a present participle.
> >
> >     > The paper has three angles here:
> >     > Qualitative mappings into quantitative numbers are a silly thing
> to do,
> >     like people trying to do "social science" by using SurveyMonkey.
> >
> >     Which is what most people are or were selling.
> >
> >     > It's fine to have a lossy compression algorithm that emphasizes
> certain
> >     aspects of the input signal over others, of course, but an
> additional CERT/
> >     CC critique is we have no reason to think CVSS does this in any
> useful way.
> >
> >     Well there 's a missing line here, you can see it from the way that
> >     client-side attacks perverted the concept of remote and so they made
> them
> >     remote also instead of adding the new line to the plot. Because of
> stuff
> >     like this. everything is remote now which limits its usefulness. This
> >     doesn't even touch on the fact that most of the CVE database is
> basically
> >     wrong from submissions including very limited data, id est "memory
> >     corruption results in a DoS".
> >
> >     Nathaniel
> >
> >     在 2019-01-09 00:14:00，"Dave Aitel" <dave.aitel at cyxtera.com> 写道：
> >
> >
> >         I wanted to take a few minutes and do a quick highlight of a
> paper from
> >         CMU-CERT which I think most people have missed out on: https://
> >
> resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdf
> >
> >         Towards Improving CVSS - resources.sei.cmu.edu
> >         resources.sei.cmu.edu
> >         SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
> >         REV-03.18.2016.0 Distribution Statement A: Approved for Public
> Release;
> >         Distribution Is Unlimited TOWARDS IMPROVING CVSS
> >         It's almost as funny a read as their previous best work on how "
> >         clientless HTTPS VPNs are insanely dumb what were you thinking
> omg?"
> >
> >         They use a ton of big words in the paper to call CVSS out and
> give it a
> >         shellacking. Like most of you, we have extensive use of CVSS in
> our
> >         consulting practice and I've seen this stuff first hand. CVSS is
> of
> >         course just a buggy compression algorithm for taking complex
> >         qualitative data and then putting it on a number line. The paper
> has
> >         three angles here:
> >          1. Qualitative mappings into quantitative numbers are a silly
> thing to
> >             do, like people trying to do "social science" by using
> >             SurveyMonkey.
> >          2. We're pretty sure that the compression algorithm is not, in
> fact,
> >             putting higher risk items as bigger numbers, which is the
> whole
> >             point of the thing.
> >          3. Nobody is applying this in any sort of consistent way (which
> is
> >             probably impossible) which is ALSO the whole point of the
> thing.
> >
> >         It's fine to have a lossy compression algorithm that emphasizes
> certain
> >         aspects of the input signal over others, of course, but an
> additional
> >         CERT/CC critique is we have no reason to think CVSS does this in
> any
> >         useful way.
> >
> >
> >         There's definitely people in the CVSS process (who I will avoid
> calling
> >         out by name) who think ANY quantization is good. But read the
> paper and
> >         decide for yourself - because these are probably serious issues
> that
> >         are turning your entire risk org into a Garbage-In-Garbage-Out
> org...
> >
> >
> >         -dave
> >
> >
> >
> >
> >
> >
> >
> >     _______________________________________________
> >     Dailydave mailing list
> >     Dailydave at lists.immunityinc.com
> >     https://lists.immunityinc.com/mailman/listinfo/dailydave
> >
>
> > _______________________________________________
> > Dailydave mailing list
> > Dailydave at lists.immunityinc.com
> > https://lists.immunityinc.com/mailman/listinfo/dailydave
>
>
> --
> Adam Shostack
> President, Shostack & Associates
> https://associates.shostack.org • +1 917 391 2168
>
> Join my very quiet annnoucement list: https://adam.shostack.org/newthing
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.immunityinc.com/pipermail/dailydave/attachments/20190110/ee9f9da9/attachment-0001.html>