<div dir="ltr">Hey everyone,<div><br></div><div>I have posted a full technical writeup and wrap-up for this bug. Check it out here: <a href="https://medium.com/@flyryan/postmortem-of-the-firefox-and-tor-certificate-pinning-vulnerability-rabbit-hole-bd507c1403b4#.oawicwift">https://medium.com/@flyryan/postmortem-of-the-firefox-and-tor-certificate-pinning-vulnerability-rabbit-hole-bd507c1403b4#.oawicwift</a></div><div><br></div><div>Thanks!</div><div><br></div><div>-Ryan</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 15, 2016 at 3:21 AM, Ryan Duff <span dir="ltr"><<a href="mailto:ry@nduff.com" target="_blank">ry@nduff.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<p class="MsoNormal">Hey everyone,<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">I spent a decent portion of my day looking into the claim by
the Tor-Fork developer that you could get cross-platform RCE on Tor Browser if
you're able to both MitM a connection and forge a single TLS certificate for
<a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a>. This is well within the capability of any decently
resourced nation-state. Definitely read @movrcx's write-up first to see his
claim. It's here:
<a href="https://hackernoon.com/tor-browser-exposed-anti-privacy-implantation-at-mass-scale-bd68e9eb1e95#.vh1a04yxy" target="_blank">https://hackernoon.com/tor-<wbr>browser-exposed-anti-privacy-<wbr>implantation-at-mass-scale-<wbr>bd68e9eb1e95#.vh1a04yxy</a><span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Instead of making you read a whole lot to find out the
issue, I'll give you the bottom line up front (BLUF): Firefox uses its own
static key pinning method for it's own Mozilla certs instead of using HPKP. The
enforcement of the static method appears to be much weaker than the HPKP method
and is flawed to the point that it is bypassable in this attack scenario. The
bug appears to be fixed as of the September 4th nightly build of Firefox but is
obviously still unpatched in both the current production versions of Firefox
and Tor Browser. I'm posting about this publically now instead of waiting
because @movrcx's post still shows how to take advantage of these flaws even if
it doesn't describe (and he didn't know at the time) exactly how and why it
bypassed certificate pinning.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Regarding my motivation for looking at this; I'm neither a
Tor-Fork OR Tor supporter. However, I've been a big fan of cross-platform RCE
for a very long time. We go way back. I also had concerns that anything
legitimate @movrcx may have found wouldn't be taken seriously because of the
inherent drama involved in going against Tor in such a public way. To his
credit, when I reached out to him after reading his blog post on it, he was
excellent to work with and didn't get defensive at all as I poked holes in his
claims. The result was us both learning a lot about how Firefox handles
certificate pinning and discovering that his attack works for reasons that
weren't obvious to either of us as we started research today. I also need to
caveat this by saying I only have a few hours of work into all of this and
while I know there is definitely a bug and I have a geat hypothesis on where
Mozilla's problem is rooted, I'm not positive about it. More lab work and
validation (no pun intended) is definitely needed. That's why I'm posting here
instead of staking more big claims in a blog. Any input from anyone who
understands Firefox certificate validation internals in detail would be very
appreciated. While I did my research, I obviously don't know what I don't know.
As you'll see, there is quite a bit of nuance here. I'm going to go through the
things @movrcx did and then explain my research into why they worked even
though they really shouldn't have.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">I want to give a quick overview of @movrcx's claims, address
some of the criticisms he has received on social media of them, and hopefully
get you believing that there is at least SOMETHING wrong happening here.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Here is a simple overview of @movrcx's attack (again, read
this first: <a href="https://hackernoon.com/tor-browser-exposed-anti-privacy-implantation-at-mass-scale-bd68e9eb1e95#.vh1a04yxy" target="_blank">https://hackernoon.com/tor-<wbr>browser-exposed-anti-privacy-<wbr>implantation-at-mass-scale-<wbr>bd68e9eb1e95#.vh1a04yxy</a>
); Firefox and Tor Browser update their extensions automatically by checking
<a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a> about every hour. Tor specifically has HTTPS Everywhere and
NoScript pre-installed. They both use the normal Firefox way of updating
themselves. To attack this, you just follow these steps. <span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">1) Write a malicious extension to be your payload and then
have it signed by Mozilla using their fully automated process.<span></span></p>
<p class="MsoNormal">2) Generate a forged certificate for <a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a> that
validates up through any CA built in to the Firefox certificate store (not an
easy task but definitely doable by a nation state and should be considered in
Tor's threat model).<span></span></p>
<p class="MsoNormal">3) Man-in-the-Middle traffic to <a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a> trying to
update NoScript or HTTPS Everywhere.<span></span></p>
<p class="MsoNormal">4) Serve your malicious extension instead of the requested
update to the target.<span></span></p>
<p class="MsoNormal">5) Win with no user interaction.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">The naysayers to this claim all mostly revolved around one
point (which I will disprove here). That point is "Mozilla pins the cert
for <a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a> and that pinning should prevent this attack. The only
reason it worked for @movrcx is because Firefox will bypass pinning when a
certificate validates through a user-added CA cert". This is wrong because,
unlike Firefox, Tor Browser strictly enforces certificate pinning. No matter
what cert @movrcx added, it still should have failed. Yet, he was able to get
it to work. That is what got me interested in this and drove me to dig in.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">There is actually quite a bit to unpack here and I'll start
with how Firefox handles certificate pinning enforcement levels. When a
certificate is pinned, a SHA-256 hash of the public cert is hardcoded into the
browser and depending on which cert is pinned (CA, Intermediate, End-Client
Cert) the site's TLS certificate must validate through or match the pinned
cert. However, Firefox actually has 4 enforcement settings that are applied
under the preference security.cert_pinning.<wbr>enforcement_level in about:config.
These are the enforcement levels:<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">0. Pinning disabled<span></span></p>
<p class="MsoNormal">1. Allow User MITM (pinning not enforced if the trust anchor
is a user inserted CA, default)<span></span></p>
<p class="MsoNormal">2. Strict. Pinning is always enforced.<span></span></p>
<p class="MsoNormal">3. Enforce test mode.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">The root of the naysayers revolved around the fact that Firefox's
default is enforcement level 1. This means that when a user adds their own CA
("like @movrcx did" but not really, as you’ll see) and the site's
certificate validates through that CA, Firefox will bypass certificate pinning.
They do this so companies can have their TLS stripping security appliances with
their root certs that need to be installed on client machines. By bypassing
pinning for those user added certs, the users will still be able to access
their GMail or other sites that would normally fail because of pinning. So,
game over, right? The attack should fail in practice according to the twitters
and the twitters are never wrong… Well... Tor actually changed the default
enforcement to level 2. You can see the ticket here: <a href="https://trac.torproject.org/projects/tor/ticket/16206" target="_blank">https://trac.torproject.org/<wbr>projects/tor/ticket/16206</a><span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Because of this, NO CA cert other than the one associated
with the pinned certificate should work. Even if you add your own CA cert to
the Firefox store, strict enforcement should mean that a connection to a site
with a pinned cert attempting to validate through that custom CA should fail. So,
double game over, right?! Nope. Somehow, @movrcx was able to get his extension
to load by validating it through his custom cert. despite strict pinning
enforcement. How is this possible? We must go deeper...<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">When @movrcx initially added his CA cert (which was the Burp
CA cert so he could perform the required MitM) to the certificate store, it DID
fail to validate. So, everything worked as expected and it's game over (x3)....
right!? Not quite. Lets looks at the debug output from the failure:<span></span>Deep down the certificate pinning rabbit hole of "Tor
Browser Exposed"</p>
<p class="MsoNormal"><span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">1473825046500addons.update-<wbr>checkerWARNRequest failed:
<a href="http://versioncheck-bg.addons.mozilla.org/update/Version" target="_blank">versioncheck-bg.addons.<wbr>mozilla.org/update/Version</a>…{<wbr>73a6fe31-595d-460b-a920-<wbr>fcc0f8843232}&version=2.9.0.<wbr>14&maxAppVersion=*&status=<wbr>userEnabled&appID={ec8030f7-<wbr>c20a-464f-9b0e-13a3a9e97384}&<wbr>appVersion=45.3.0&appOS=Linux&<wbr>appABI=x86_64-gcc3&locale=en-<wbr>US&currentAppVersion=45.3.0&<wbr>updateType=112&compatMode=<wbr>normal
- [Exception... "Certificate issuer is not built-in." nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame ::
resource://gre/modules/<wbr>CertUtils.jsm :: checkCert :: line 171" data: no]<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">See that exception? It doesn't say "failed to validate
pinned certificate" or something like that. Instead, it says,
"Certificate issuer is not built-in". Could it be that if it
validates through a cert that is built-in to the browser, it would work? Lets
find out. @movrx added the Burp certificate to certdata.txt (
<a href="https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt" target="_blank">https://hg.mozilla.org/<wbr>mozilla-central/raw-file/tip/<wbr>security/nss/lib/ckfw/<wbr>builtins/certdata.txt</a>
), compiled the TorBrowser with that in, retried his MitM, and it worked! Now,
there are actually TWO security failures here. The first one we discussed;
strict enforcement of pinning should have made this fail regardless of the new
certificate. But there is a second failure; adding a certificate this way
should not have made it recognized as “built in”. Firefox uses libNSS for
verification of built-in certificates. That’s done by calling a software module
called CKBI (which somehow is supposed to stand for "built-in root
certificates"). That module is implemented in libNSS as libnssckbi.so.
There is a special entitlement that libNSS looks for that can only be stored in
the read-only libnssckbi.so. Mozilla updates this certificate list and sends it
out with each new version of libNSS (which ships with Firefox). A brief rundown
of this policy can be found here: <a href="https://wiki.mozilla.org/NSS:Root_certs" target="_blank">https://wiki.mozilla.org/NSS:<wbr>Root_certs</a><span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">You actually can add your own CA manually and get this
entitlement, but it requires compiling a utility that is not normally included
with Mozilla's libNSS into your own custom copy of libNSS (instructions on how
to do that are here: <a href="http://wiki.cacert.org/NSSLib" target="_blank">http://wiki.cacert.org/NSSLib</a> ). You then must compile
your custom libNSS into your custom Firefox build. But, as I stated above, that
is not what @movrcx did.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Let's recap where we are at this point (I told you this
rabbit hole was deep)... We have a situation were strict pinning enforcement is
enabled and seems to work by not letting the extension update connection
validate when a CA cert is added to the Firefox store through its interface.
However, it's not failing due to pinning enforcement, but rather on the fact
that the certificate it's trying to use doesn't have the BuiltIn entitlement.
When we compile the same cert in directly, it is then recognized as BuiltIn
even though it doesn't have the specific entitlement libNSS should be looking
for. That's two checks it should have failed but were bypassed. So... WTF
right? Why is this happening?! Deeper we go...<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">My answer presented itself after a discussion with Erinn
Atwater (@errorinn) who is a PhD student in Computer Science at the University
of Waterloo in Ontario. I saw that she was asking pointed questions on twitter
that were in the same line as my research, so I shot her a DM. She had actually
put a bunch of students on this task last night because she had the same
concerns I did about this not being taken seriously. It turns out that Mozilla
actually doesn't use normal HPKP for certs related to their operations (like
<a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a>). Instead, they use a form of static key pinning. These
validation weaknesses are limited to these statically pinned certs and HPKP
seems to be fully functional (ie: not broken) in Firefox. That's great for
every other site but obviously bad for Mozilla and the certs they have
statically pinned.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">I don't have enough data to state this with 100% certainty,
but it appears that with statically pinned certs, the only requirement for
validation is that the certificate validates through a "built in" CA.
However, since it's not using libNSS to do the validation, it looks like
"built in" for statically pinned cert validation just means that the
CA was there at compile time. That is why the certificate didn't validate when
it was added through the Firefox interface but did when it was compiled in. In
turn, that means that any “<a href="http://addons.mozilla.org" target="_blank">addons.mozilla.org</a>” certificate that validates
through any CA that is shipped with Firefox should bypass pinning restrictions
and work. That also means that @movrcx's attack should work as advertised.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">You can actually test this without all of the compiling
custom browsers! Try going to <a href="http://pinning-test.badssl.com" target="_blank">pinning-test.badssl.com</a> in Firefox. It SHOULD
give a TLS error... but it doesn't for most versions of Firefox. Mozilla added
a static pinned cert for this test and you can find it here:
<a href="https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h#283" target="_blank">https://dxr.mozilla.org/<wbr>mozilla-central/source/<wbr>security/manager/ssl/<wbr>StaticHPKPins.h#283</a>
. The pinned cert has a static pin set to "AAAAAA=" which doesn't
match any known CA. That means it should fail validation. Yet, since the test
site has a valid cert from Google, it validates through a CA that is built in
to the browser and works. That is the core bug that makes this attack work.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">The good news is that Erinn's team was able to determine
that this bug is fixed in the September 4th Firefox nightly build.
Unfortunately, it's not clear what bug the fix was associated with or what
change in the code facilitated it. The changelog for that build can be found
here: <a href="http://forums.mozillazine.org/viewtopic.php?f=23&t=3022271" target="_blank">http://forums.mozillazine.org/<wbr>viewtopic.php?f=23&t=3022271</a> . None of
the public bugs seem to line up though. However, there are 2 restricted bugs at
the top of the list. It's possible that this issue might be one or both of
those. If you want to test the builds yourself, the issue occurs in:<span></span></p>
<p class="MsoNormal"><a href="https://ftp.mozilla.org/pub/firefox/nightly/2016/09/2016-09-03-00-40-09-mozilla-aurora/" target="_blank">https://ftp.mozilla.org/pub/<wbr>firefox/nightly/2016/09/2016-<wbr>09-03-00-40-09-mozilla-aurora/</a>
and does not occur in:
<a href="https://ftp.mozilla.org/pub/firefox/nightly/2016/09/2016-09-04-00-40-02-mozilla-aurora/" target="_blank">https://ftp.mozilla.org/pub/<wbr>firefox/nightly/2016/09/2016-<wbr>09-04-00-40-02-mozilla-aurora/</a>
. I'm not sure when this will be pushed to the production versions of Tor and
Firefox but at least a fix seems to be in the pipeline.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">So, that's it for the bug. As you have seen, the attack as
described by @movrcx should work as advertised, but it's not at all immediately
obvious why it works. If you know how pinning is supposed to work, you would
probably claim with gusto that it doesn't work, as many on twitter were very
quick to do. Yet, it does work.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">While Tor will catch the fix from the Mozilla patch, I
believe they should actually change how they handle extensions overall. It
seems ridiculous to me that they actually use Mozilla’s auto-update process for
extensions. If NoScript or HTTPS Everywhere added a new vulnerability with an
update, all Tor users would get it within an hour of using the browser. Also, with
the paranoia their organization seems to have, I would think Mozilla being
compelled to push a malicious extension to specific Tor users would be a real
concern of theirs. To me, the logical solution would be to compile NoScript and
HTTPS Everywhere themselves, sign those extensions with their own key, hardcode
their public key into the TorBrowser, and then do their own cryptographic
validation of extensions locally. Extension updates would go out with
TorBrowser updates exactly how the TorBrowser Firefox updates are delivered.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">If you made it this far, you are a trooper. I hope it was
informative. If you have any questions or input, please share. It’s always
possible I missed something major and all of this was a big mistake. ;-) If so,
please tell me! Thanks again to @movrcx for being cool and working through all
of this with me. Also, thanks to @errorinn for sharing her and her team’s
research with me. It was key in closing the last information gap I had.<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">Thanks again for reading!<span></span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal">-Ryan Duff<span></span></p>
<p class="MsoNormal"><a href="https://twitter.com/flyryan" target="_blank">https://twitter.com/flyryan</a><span></span></p>
</div>
</blockquote></div><br></div>