• Building a NAS

    Introduction

    Back in July I had a backup drive die. Fortunately I found out before I needed the backup, but regardless it got me worried. It was the only backup drive that I had for that data. It was a personal external drive, and nothing more.

    This started off with some innocent intentions where I just wanted to have some external storage for big downloads, keeping VM snapshots, and ISOs. At the time, I was on DSL internet. Downloading an ISO for an operating system was no small feat, so I kept them locally.

    This innocent intention eventually evolved in to a drive that had important backups. I wasn’t entirely stupid; real important things like family photos, documents, and source code for projects are backed up in to what we now call “The Cloud”. This drive amassed many virtual machines in specific states with specific software installed, a copy of my personal music collection, and periodic snapshots of my entire workstation. Losing them would not always be fatal, but it would mean hundreds of hours of work, lost.

    This drive has clearly gone beyond its intended use, and it is showing signs of wear. It has no redundency in itself. Most disappointingly, I know better. I’ve implemented backup strategies for data storage, HSMs, and databases. I know things fail.

    So now I need to fix this in a more permanent way. As appealing as offsite backup to a cloud provider is, it just doesn’t scale the way I need it to. I had not one but three of these fragile external drives, totally several terabytes of data. It’s also needed fairly sporadically, not just when systems fail. Pulling down a 250 GB virtual machine image over the internet isn’t fast as I’d like. Even if I have the bandwidth, the cloud provider likely throttles it. I’ll continue to put backups of real backups in the cloud, but for large local storage, I needed something different.

    I needed something like a NAS. I had thought for a while the best way to approach this. Some people recommended off-the-shelf solutions that work well for them. Synology came highly recommended. The alternative choice is go down the DIY route. DIY has come a long way with building a personal storage. There are many barebones solutions out there for building it yourself, customizing it, and throwing disks in it.

    I decided to build something myself, completely from scratch. I thought this would be a fun project for me to work on from time-to-time, and I could get as weird and crazy with the project as I wanted. I didn’t really have a budget for this, which in hindsight was probably not the best idea.

    In the end, I did have a lot of fun building it. Penny (my daughter) was fascinated by it, and she even helped! Here’s how I built my NAS, and what I learned along the way.

    read more...
  • SRI with Jekyll

    Subresource Integrity - or SRI - is something I wanted to learn, so the natural place to get started with it is right here on my blog. Though it has somewhat limited value on my site since I don’t use a CDN, it still proves useful with helping me understand these things in a relatively safe place. After all, no SLA is going to be violated if my blog’s CSS doesn’t load.

    In fact, since we don’t have any JavaScript here, the only thing I could use it on right now is my lone CSS include.

    To back up, SRI is the practice of including a hash of an external asset where that external asset is included in the current page.

    For example, using SRI on my current site looks like this:

    <link
        rel="stylesheet"
        crossorigin="anonymous"
        integrity="sha256-xk/GF3tsHVHrcjr3vColduFPXc/PrGx+WNHy+SvR8X8="
        href="/css/main.css">
    

    Where the integrity attribute includes the digest algorithm along with the digest itself. This is in the same manner and style as PKP. So main.css base64 digest is xk/GF3tsHVHrcjr3vColduFPXc/PrGx+WNHy+SvR8X8=, for now.

    The purpose of this is if I were using a CDN, or my static content were included on another server outside of my control - how do I know it hasn’t been tampered with along the way? It’s meant to stop something like this:

    <link
        rel="stylesheet"
        crossorigin="anonymous"
        integrity="sha256-xk/GF3tsHVHrcjr3vColduFPXc/PrGx+WNHy+SvR8X8="
        href="//malicious.cdn.com/style.css">
    

    Assuming that the page also wasn’t hosted on malicious.cdn.com, the digest would prevent them from changing it. The hashes would no longer match, and the browser would refuse to load the stylesheet.

    For CSS this has some advantages, but the real use here is with JavaScript that’s on a CDN.

    So SRI is pretty straight forward, in theory. When you get down to it though, many websites have an asset pipeline. These pipelines can minify JavaScript and CSS, transpile them, or any other transformation. These can either happen at build time or at run time. The digests on the assets need to be the digest of the final asset of what the browser sees and actually executes.

    As this is a static site, my CSS is actually a SCSS file that gets compiled by Jekyll. Every page includes this stylesheet. I needed to have the generated pages have the hash of my stylesheet after it gets compiled. So either I needed to do this after Jekyll compiled the site, or figure out a way to get Jekyll to put the hash in.

    Jekyll, or more specifically Liquid, support custom tags. What I wanted to be able to do was put something like this in my template:

    <link
        integrity="{% sri_scss_hash css/main.scss %}"
        href="/css/main.scss" />
    

    All I had to do here was make the sri_scss_hash tag. How hard could it be?

    Well, not entirely straight forward, considering I know little about ruby. I used Jekyll’s source code as a reference on how to implement this. The final version of this plugin is on my GitHub repository for this site.

    It uses the existing IncludeRelativeTag Jekyll tag as a base, and instead of including the content as-is, it runs it through the SCSS converter, first. The meat of it is the render function:

    def render(context)
        cache_compiled_scss(@file, context, lambda {
            site = context.registers[:site]
            converter = site.find_converter_instance(Jekyll::Converters::Scss)
            result = super(context)
            scss = result.gsub(/^---.*---/m, '')
            data = converter.convert(scss)
            "sha256-#{Digest::SHA256.base64digest data}"
        })
    end
    

    This works well, but it does one thing that feels a little hacky to me which is use a regular expression to remove the frontmatter from the stylesheet before running it through the converter. The frontmatter is required for Jekyll to trigger its compilations for the files, but the converter itself doesn’t like the frontmatter. I feel like there should be a better way to remove the frontmatter, but this works for now. If there are any Jekyll experts out there, I would love to know a better way to do this.

    All you need to do then is drop a ruby file in _plugins along with a line to register the plugin, and you can start using the plugin to generate SRI hashes for SCSS stylesheets.

    I learned a few things implementing this. Specifically, the sandbox Content-Security-Policy directive gave me some trouble. This took me a little while to understand, but the SRI check will fail if the origin the document does not have Cross-Origin access to the resource.

    All in all I think SRI is a worthwhile investment if you’ve got all of the low-hanging fruit already picked. It’s not as trivial to do as it would first seem do to the complexities of build processes.

  • Crypto in .NET Core

    The .NET Framework “Core” has been out for a while now. I spent some time getting my bearings in order and getting a handle on where some things moved around.

    This post is specifically covering the cross-platform .NET Core. If you’re still using the .NET Framework on Windows, then nothing has really changed for you.

    First I want to point out that support for cryptographic primitives is pretty good, all things considered. It’s much richer than I would have expected.

    Factory Methods

    In the Desktop .NET Framework, you had potentially a few different implementations of the same primitive function. Consider the SHA1 hash algorithm. In the .NET Framework, you had SHA1Managed, SHA1CryptoServiceProvider, and SHA1Cng. All of these do SHA1, they just do it differently. SHA1Managed was a pure, managed code implementation of SHA1. It’s useful in some situations, such as Medium Trust, but is generally slower. SHA1CryptoServiceProvider uses the now-legacy CAPI implementation in Windows, and SHA1Cng uses CAPI’s successor, CNG.

    Crucially, each primitive shares a common base. The examples above all inherit from the abstract class SHA1. These abstract classes provide a static method called Create which act as factory method. Create on SHA1 has a return type of SHA1.

    In the .NET Core, things are much different. These specific implementations, such as SHA1CryptoServiceProvider, are now gone. Instead, using SHA1.Create is the only way to create an object that does the SHA1 primitive. So your previous code that might have looked like this:

    using (var sha1 = new SHA1Managed()) {
        //SHA1...
    }
    

    Should now look like this:

    using (var sha1 = SHA1.Create()) {
        //SHA1...
    }
    

    If you were already using the factory methods, then moving to .NET Core will be even easier.

    Under the covers, much of the cryptographic primitives in .NET Core are either implemented with a combination of CAPI+CNG, and OpenSSL on *nix and macOS. The factory methods should always be used, when possible. These will always do the right thing for the right platform.

    These factory methods exist for all hash primitives, such as SHA256, and also AES and HMAC functions. This is also true for ECDSA, and RandomNumberGenerator.

    The last bears having its own example, since it tends to be one of the ones people run in to the most. If you are using RNGCryptoServiceProvider, replace it with RandomNumberGenerator.Create():

    using (var rng = RandomNumberGenerator.Create()) {
        //Use like RNGCryptoServiceProvider
    }
    

    Not everything uses a factory method. Some classes like Rfc2898DeriveBytes you should continue to use as you always have, and have been modified to work on all platforms.

    ECC Keys

    One of the things missing from .NET Core right now is an easy way to work with ECC keys. In the desktop .NET Framework, CngKey is available for loading a persisted key from a KSP (like an HSM). .NET Core expected you to work with ECC keys mostly in conjunction with an algorithm, like ECDSA. If the private key belonging to the certificate happens to be on an HSM - it will work - but there is no clean platform-agnostic API to load a key, either from an HSM or from a PKCS8 file. You can load a private key from a PKCS12 file along with a certificate.

    If you really need to work with keys directly, you can continue to use CngKey. Even though CngKey is Windows-specific, it appears in the netstandard contracts. It does not appear that there is a OpenSSL equivalent of EVP keys.

    Missing things

    Some things are missing, and many of them I would say “good riddance” to. Some are also missing but will likely appear in a later update to .NET Core.

    DSA

    The finite-field implementation of DSA (non-ECC), is gone. DSA should largely only be used for interoping with existing legacy systems, but .NET Core does not have it. This algorithm will make a come-back in a later verison of .NET Core, it would seem.

    ECDH

    EC Diffie-Hellman is missing. I would say very few people need to do a key-exchange themselves, so not many should miss it. However again I have been told this will return in a later update to .NET Core.

    Miscellaneous

    A smattering of other algorithms that shouldn’t be used are gone as well:

    • RIPEMD160 is gone.
    • RijndaelManaged is not in the contract anymore. Use Aes instead. The only time RijndaelManaged should be used is when you need to interop with data that uses a non-standard AES block size, which is very unlikely.
    • DES is gone. DES3 remains.
    • MACTripleDES is gone.
    • PBKDFv1 is gone, which came in the form PasswordDeriveBytes. PBKDFv2 remains in the form of Rfc2898DeriveBytes. PasswordDeriveBytes may make a return.
    • Some modes of block cipher encryption are gone. ECB, CBC, and CTS are all that is supported. The others were more exotic and unlikely to be used.

    New things

    Named elliptic curves are now supported, and work cross platform. This currently is limited to named curves, such as brainpool. Even more interestingly, it looks like support for other curves might be coming, such as Edwards and Montgomery curves. This would enable x25519 (once ECDH returns) and EdDSA. These are not supported yet, but clues of their arrival appear in the ECCurveType enumeration.

  • Authenticode and ECC

    While HTTPS / TLS have been making great strides in adopting new cryptographic primitives, such as CHACHA, x25519, and ECC, another place has remained relatively stagnant: binary signing.

    While many platforms deal with binary signing, I deal the most with Authenticode which is part of the Windows operating system. I thought it would be an interesting experiment to sign a few things with an ECDSA certificate and release them in to the wild.

    First I needed to find a CA willing to give me an ECDSA Authenticode certificate. DigiCert was happy to do so, and they offered it to me for no charge as part of their Microsoft MVP program. They were very helpful by making it very easy to get an ECDSA Code Signing certificate.

    ECDSA Signing Certificate

    Safe Keeping

    Normally I keep my signing certificate on a hardware token, a Yubikey 4. I’ve had some good success with the Yubikey for this purpose. I would generate a CSR from the Yubikey, it would keep the private key, and I would get a certificate issued with the CSR. The Yubikey is also able to do all of this for ECDSA / P-256 certificates. I was even able to load the ECDSA certificate that was issued on to the Yubikey, and my Mac recognized it immediately, as did OpenSC.

    Windows however was a bit different. Normally when you insert a SmartCard on Windows, it will read the public certificate from the SmartCard, automatically import it in to the Personal certificate store, and link-up the private key to the SmartCard.

    That did not work with an ECDSA certificate. The Windows service that is responsible for this, “Certificate Propagation”, doesn’t handle ECDSA certificates. Manually importing the certificate doesn’t work either, because the certificate is missing the “Key Container Name” link back to the SmartCard. It’s possible to repair this link, but it needs to be done every time the SmartCard is re-inserted.

    For purposes of this experiment, I decided to forgo the SmartCard and instead import the private key in to the Windows Key Container store, and force it to prompt me for a pin every time it’s used to sign.

    Signing Pin Prompt

    Signing

    Signing with the ECDSA certificate worked as expected. signtool had no problems signing with ECDSA, and Windows was happy with the signature.

    There was nothing different that needed to be done here. My Authenticode Lint tool was also happy with the ECDSA signatures. This was a little exciting to see an ECDSA certificate “work” in Authenticode Lint. To-date all of the ECC capabilities of it have been tested with self-signed certificates.

    Distribution

    Everything started to go wrong here. While Windows was happy with my signatures, many other things had a problem with it.

    The first were Antivirus systems. AV applications take in to account signatures to determine the trustworthiness of the application. Of the few that I was able to test, none of them recognized the binary as signed. This tripped up Windows SmartScreen, which told me my own application wasn’t signed, seemingly confused by the ECDSA signature.

    Likewise “UTM” firewalls didn’t like the binaries, either. These are firewalls that do on-the-fly virus scanning as files are downloaded and block it if it considers it unsafe. Depending on how the UTM is configured, it didn’t like the ECDSA signatures, either.

    This is easy enough to fix by mixed signing, which is a scarce-to-nonexistent practice with Authenticode. Most applications are already dual signed, once with a SHA1 file digest, and also with a SHA2 file digest. To make ECC work, you would need a third signature. The ECC one can’t take the place of the RSA+SHA2 one because then those poorly behaving applications will ignore the ECC+SHA2 one and treat it as only RSA+SHA1 signed.

    Three Signature File

    Lastly, some Antivirus vendors think even the presence of an ECDSA signature is enough to warrant flagging it, however most of these scanners seemed to be small companies. The bigger-name scanners did not have a problem with the presence of an ECDSA signature.

    Conclusions

    I understand why ECC hasn’t taken off with code signing. RSA has a lot of inertia and there is little reason to move to something else if RSA “just works”.

    If for whatever reason you want to use ECDSA to sign binaries, you will likely need to continue to sign with RSA (and RSA being the “root” signature).

    Motivations

    I mostly did this to solve my own curiosity. ECC does have a number of benefits, though. It’s generally considered stronger and more compact. Authenticode PKCS#7 also stores the whole certificate chain for the signature. If the chain were entirely ECC (alas I was not able to get an ECDSA cert that was issued by an an ECDSA intermediate) then it could shave a few kilobytes from the size of the signature.

    If you need stronger but can’t afford the complexity of ECDSA, then RSA-4096 is the way to go.

  • Password History

    A few tweets have started making the rounds about how companies must be doing password security wrong because they seemingly do magic. Let’s start with the simple one, password history. Here’s some musings on how I think some magic could be implemented without a huge loss of security.

    This is pretty straight forward to solve. Keep a history of the hashes. When the user enters a new password, take the password they just entered, and see if it matches any of the password hashes in the history.

    A password hash typically consists of a digest and a salt. I’d strongly recommend that each password in the history have their own salt, so don’t reuse salts when a user types in a new password. So a password history table might look like this:

    salt digest version
    salt1 digest1 1
    salt2 digest2 1
    salt3 digest3 2

    And the user enters a new password because they want to change it. The check process would look something like this:

    hash_alg_1(salt1 + newPassword) == digest1
    hash_alg_1(salt2 + newPassword) == digest2
    hash_alg_2(salt3 + newPassword) == digest3
    

    If any of those are “yes”, then they’ve used a password in their history.

    In real life, you are probably using something like bcrypt. Many bcrypt libraries put the salt, digest, and “version” (work factor) in to a single output separated by dollar signs. They also provide convenience APIs to make the verify process simpler. You give it a previous bcrypt output and a plaintext password, and it knows how to use the salt and the work factor to see if the hashes match.

    This approach typically works pretty well. I’d also caution how long you would want to keep password history. Too long might mean keeping around hashes that are weak. Let’s say you were using bcrypt(8) a few years ago, but moved to bcrypt(10). Keeping that bcrypt(8) password indefinitely means you’re storing passwords with less-than-ideal strength. If the password history is ever stolen and the history is weak, then you might be giving the attacker clues as to a password the user is using on another site, or their password habits.

    If you ever need to drastically change the password hashing scheme because it’s broken (straight, unsalted MD5 for example) I’d purge the history altogether. It’s too much of a liability to have lying around.

    The trickier one is fuzzy password history, but it can be done in some limited ways. The trouble with hashes is, they either match, or they don’t. There is no good way today to see if two password hashes are related to each other.

    You can however tweak the input of the plaintext when checking history.

    Let’s say the user’s old password is “I<3BillMurray11” and they change it to “I<3BillMurray12”. A site might say this password is too similiar to a previous password. You might quickly come to the conclusion they are storing passwords in plain text, or reversable encryption.

    The site also could simply try a few heuristics on the input. It’s well known that when a password needs to change, users might cop-out and just increment a number at the end. So when the user types in a new password, check the history. No matches? Well, does it end with a number? Yes? decrement it and try the history again.

    You are certainly limited to how much fiddling you can do like this. Good password hashing schemes like bcrypt are by-design not fast (to counteract offline brute force attacks). So checking hundreds of things is quite slow.

    You can however use this to try a few of the worse offenders. Incremented trailing numbers is a very common one, same with addign a letter. Try chopping off a letter from the password end and see if it matches the history. Password history for a user should also reasonably fit in to memory, so doing these checks in parallel is doable, too.

    Those are just some examples, and things that I think security engineers can reasonably implement. Unfortunately when things like this do get implemented, there is often suspicion and cries of plaintext problems.

    That’s not to say that there aren’t sites that do store passwords in plaintext or symmetric encryption. Those sites are problematic, and need to get fixed. If the password history fuzziness seems too clever, such as Levenshtein or Hamming distance, then that might indicate bigger problems.