[Dailydave] CVSS is the worst compression algorithm ever

Thu Jan 10 19:15:52 UTC 2019

+1 Wim. You covered that perfectly.

--
Dennis Groves <http://about.me/dennis.groves>, MSc
dennis.groves at gmail.com

If you cannot think of three ways of abusing a tool,

you do not understand how to use it.

                                             -- Gerald M. Weinberg

On Thu, Jan 10, 2019 at 7:56 AM Wim Remes <wremes at gmail.com> wrote:

> Hi,
>
> Bruce really hits the nail on the head here. CVSS != Risk. To broaden that
> discussion and not waste too many words, I’ll reference FAIR (Factor
> Analysis of Information Risk, https://www.fairinstitute.org/what-is-fair)
> to indicate where “Vulnerability” contributes to an eventual quantitative
> risk valuation.
>
> I also always considered CVSS scoring to be qualitative instead of
> quantitative and the numbers to be ordinal. That makes them fine for
> ranking vulnerability, but horrible to perform math on (Jet Fuel x Peanut
> Butter = Shiny — hi Alex Hutton!).
>
> That said, it all boils down to a point I’ve been rapping on about for a
> long long time now. Organizations should not expect third party penetration
> testers to make an accurate assessment of risk. The data provided by a
> third party penetration tester should feed into your risk management
> framework, that is also fed with internally acquired business data, to
> produce (or adjust) a risk valuation. It would be helpful if we, as
> consultants, wouldn’t pretend that we (a) can come up with any form of
> credible risk score during such assessments and (b) are delivering scoring
> that can help with prioritization in a business context without additional
> effort on the client side. On the other hand, clients that have a risk
> management framework that can actually take vulnerability scores and use
> them to generate risk scores should be clear in what they expect from us.
> If you are asked, whether in an RFP or an SoW, to produce a risk score for
> your findings at the very least you should be returning a question for
> asset valuation and threat community descriptions.
>
> Cheers,
> Wim
>
>
>
> On 8 Jan 2019, at 18:33, Monroe, Bruce <bruce.monroe at intel.com> wrote:
>
> Hi Dave,
>
> I participate on the CVSS SIG being ran out of FIRST that is working on
> improvements to CVSS. So do a number of people out of CERT CC, NIST,
> MITRE along with a good representation of industry. A number of us
> provided feedback on this paper. CVSS is for scoring the severity of a
> vulnerability. CVSS does not = Risk.
>
> My understanding is there is a number of government entities that believe
> CVSS does = Risk and are using it in a vacuum for that purpose. While the
> CVSS score is a single component - you also must look at how the vulnerable
> component is deployed, controls in place, value of asset, patching windows,
> likelihood of exploit,ect…there is a lot that goes into determining risk.
>
> The fact that various USG entities is using CVSS wrong is an education
> issue imo. Yes CVSS has it’s issues with some of it’s elements being
> subjective eye of the beholder type items but that isn’t the reason for
> this paper…they’ve got USG people using it in a vacuum when it’s only a
> single element of determining your orgs risk due to a vulnerability. That
> isn’t a CVSS problem that’s a vulnerability management 101 problem.
>
> Regards,
> Bruce
> Intel PSIRT
>
> Opinions expressed are my own and may not reflect those of my employer.
> *From:* Dailydave <dailydave-bounces at lists.immunityinc.com> *On
> Behalf Of *Dave Aitel
> *Sent:* Tuesday, January 08, 2019 8:14 AM
> *To:* dailydave at lists.immunityinc.com
> *Subject:* [Dailydave] CVSS is the worst compression algorithm ever
>
>
> I wanted to take a few minutes and do a quick highlight of a paper from
> CMU-CERT which I think most people have missed out on:
> https://resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdf
>
> Towards Improving CVSS - resources.sei.cmu.edu
> <https://resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdf>
>
> resources.sei.cmu.edu
>
> SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
> REV-03.18.2016.0 Distribution Statement A: Approved for Public Release;
> Distribution Is Unlimited TOWARDS IMPROVING CVSS
> It's almost as funny a read as their previous best work on how "clientless
> HTTPS VPNs are insanely dumb <https://www.kb.cert.org/vuls/id/261869/> what were
> you thinking omg?"
>
> They use a ton of big words in the paper to call CVSS out and give it a
> shellacking. Like most of you, we have extensive use of CVSS in our
> consulting practice and I've seen this stuff first hand. CVSS is of course
> just a buggy compression algorithm for taking complex qualitative data and
> then putting it on a number line. The paper has three angles here:
>
>    1. Qualitative mappings into quantitative numbers are a silly thing to
>    do, like people trying to do "social science" by using SurveyMonkey.
>    2. We're pretty sure that the compression algorithm is not, in fact,
>    putting higher risk items as bigger numbers, which is the whole point of
>    the thing.
>    3. Nobody is applying this in any sort of consistent way (which is
>    probably impossible) which is ALSO the whole point of the thing.
>
>
> It's fine to have a lossy compression algorithm that emphasizes certain
> aspects of the input signal over others, of course, but an additional
> CERT/CC critique is we have no reason to think CVSS does this in any useful
> way.
>
>
>
> There's definitely people in the CVSS process (who I will avoid calling
> out by name) who think ANY quantization is good. But read the paper and
> decide for yourself - because these are probably serious issues that are
> turning your entire risk org into a Garbage-In-Garbage-Out org...
>
>
>
> -dave
>
>
> _______________________________________________
> Dailydave mailing list
> Dailydave at lists.immunityinc.com
> https://lists.immunityinc.com/mailman/listinfo/dailydave
>
>
> _______________________________________________
> Dailydave mailing list
> Dailydave at lists.immunityinc.com
> https://lists.immunityinc.com/mailman/listinfo/dailydave
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.immunityinc.com/pipermail/dailydave/attachments/20190110/02c4e2e4/attachment-0001.html>