Signing and Verifying Python Packages with PGP
When I first showed Pip, the Python package installer, to a coworker a few years ago his first reaction was that he didn’t think it was a good idea to directly run code he downloaded from the Internet as root without looking at it first. He’s got a point. Paul McMillan dedicated part of his PyCon talk to this subject.
Python package management vs. Linux package management
To illustrate the security concerns, it is good to contrast how Python modules are usually installed with how Apt or Yum do it for Linux distributions. Debian and Redhat distros usually pre-provision the PGP keys for their packages with the distribution. Provided you installed a legitimate Linux distribution, you get the right PGP keys and every package downloaded through Apt/Yum is PGP checked. This means that the package is signed using private key for that distribution and you can verify that the exact package was signed and has not been modified. The package manager checks this and warns you when it does not match.
Pip and Easy Install don’t do any of that. They download packages in plaintext (which would be fine if every package was PGP signed and checked) and they download the checksums of the package in plaintext. If you manually tell Pip to point to a PyPI repository over HTTPS (say crate.io), it does not check the certificate. If you are on an untrusted network, it would not be tough to simply intercept requests to PyPI, download the package, add malicious code to setup.py and recalculate the checksum before returning the new malicious package on to be downloaded.
I think the big users of Python like the Mozillas of the world run their own PyPI servers and only load a subset of packages into it. I’ve heard of other shops making RPMs or DEBs out of Python packages. That’s what I often do. It lets you leverage the infrastructure of your distribution and the signing and checking infrastructure is already there. However, if you don’t want to do that, you can always PGP sign and verify your packages which is what the rest of this post is about.
Verifying a package
There are relatively few packages on the cheeseshop (PyPI) that are PGP signed. For this example, I’ll use rpc4django, a package I release, and Gnu Privacy Guard (GPG), a PGP implementation. The PGP signature of the package (rpc4django-0.1.12.tar.gz.asc) can be downloaded along with the package (rpc4django-0.1.12.tar.gz). If you simply attempt to verify it, you’ll probably get a message like this:
% gpg --verify rpc4django-0.1.12.tar.gz.asc rpc4django-0.1.12.tar.gz gpg: Signature made Mon Mar 12 15:14:28 2012 PDT using RSA key ID A737AB60 gpg: Can't check signature: public key not found
This message lets you know that the signature was made using PGP at the given date, but without the public key there is no way to verify that this package has not been modified since the author (me) signed it. So the next step is to get the public key for the package:
% gpg --search-keys A737AB60 gpg: searching for "A737AB60" from hkp server keys.gnupg.net (1) David Fischer <djfische@gmail.com> 2048 bit RSA key A737AB60, created: 2011-11-20 Keys 1-1 of 1 for "0xA737AB60". Enter number(s), N)ext, or Q)uit > q
If you hit “1″, you will import the key. Re-running the verify command will now properly verify the package:
% gpg --verify rpc4django-0.1.12.tar.gz.asc rpc4django-0.1.12.tar.gz gpg: Signature made Mon Mar 12 15:14:28 2012 PDT using RSA key ID A737AB60 gpg: Good signature from "David Fischer <djfische@gmail.com>"
The fact that ten different Python modules will probably be signed by ten different PGP keys is a problem and I’m not sure there’s a way to make that easier. In addition, my key is probably not in your web of trust; nobody who you trust has signed my public key. So when you verify the signature, you will probably also see a message like this.
gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner.
This means that I need to get my key signed by more people and you need to expand your web of trust.
Signing a package
Signing a package is easy and it is done as part of the upload process to PyPI. This assumes you have PGP all setup already. I haven’t done this in about a month so I hope the command is right.
% python setup.py sdist % python setup.py upload --sign
There are additional options like the correct key to sign the package, but the signing part is easy.
However, how many people actually verify the signature? Almost nobody. The package managers (Pip/EasyInstall) don’t and you probably just use one of them.
The future of Python packaging
So what can we do? I tried to work on this at the PythonSD meetup but I didn’t get very far partially because it is a tough problem and partly because there was more chatting than coding. As a concrete proposal, I think we need to get PGP verification into Pip and solve issue #425. This probably means making Python-gnupg a prerequisite for Pip (at least for PGP verification). Step two is to add certificate verification. Python3 already supports certificate checking through OpenSSL. Python2 might have to use something like the Requests library. Step three is to get a proper certificate on PyPI.
MIT Sloan Sports Analytics Conference 2012
This will be a slight divergence from the usual programming…
I’m at the MIT Sloan Sports Analytics Conference this weekend and I’m having a blast. Jeff van Gundy is hilarious. Also, I was feeling a little snarky when I registerred (like 4 months ago) and I was also at work doing security stuff. So I put a joke in my company name. The attendees list is sorted by company and now I’m the top of the list.
However, getting down to San Diego sports, I took a look at the attendees list and nobody is here from the Chargers or Padres. I’m hoping they’re just incognito, but I’m guessing nobody came. Hopefully the home teams don’t get left behind…
Presentation at San Diego Python Users/DjangoSD
My slides are up on github.
Here’s the presentation in action.
San Diego Python Users Group / DjangoSD Meetup
I’ll be giving a talk next month on Securing Your Django Site. See you there!
Thoughts on Bitcoin
This is going to be somewhat of a departure from your regularly scheduled programming here. I had some extra time over the past week or so and I decided to learn what I could about Bitcoin. This rant is going to contain some poor semblance of economic theory and really I should just stick to tech but I couldn’t resist.
Skip this if you already understand Bitcoin
Bitcoin is a peer-to-peer currency that has gotten some press lately. If I have a Bitcoin wallet (and I do), you can easily send me some “money”. I didn’t read through the code, but my understanding is that this is handled by adding my Bitcoin address to the digital data, signing it with your private key and then broadcasting it to the world. In that way, Bitcoin participants have the entire history of every transaction and only one person possesses any given Bitcoin at any given time. As you might guess, if everybody started using Bitcoin tomorrow, scalability might be a problem. However, it works for now.
What rocks about Bitcoin
A lot of different companies are trying to “solve” payments — be they mobile, physical, digital or anything else. One reason why folks think our current (credit card based) system needs solving is that it is pricey for merchants. The generally expected fees are somewhere just shy of 3% and can also include a fixed transaction fee as well. Fixed fees mean that credit card transactions don’t scale down very well when you are buying something cheap. Ever seen a sign saying $10 minimum for credit cards? That’s why. Bitcoin payments are virtually free which sounds awesome from a merchant’s perspective. They are not generally reversible unless the receiver chooses to reverse it and there’s no central authority to decide who is allowed to pay whom and for what. This is particularly useful if you’re Wikileaks and you’ve been cut off from payment processors or you operate a gambling website and you’re having problems with US banks stopping or reversing your transactions.
For regular users, Bitcoin comes with some advantages such as some level of privacy. No more embarrassing credit card statements. Jokes aside, anonymity of payment has quite a few legitimate purposes and is obviously useful for illegitimate purposes. In addition, Bitcoin requires very little upfront cost. There’s no need to setup a bank account or have a credit limit. It functions like cash but over the internet. You get yourself some Bitcoins and buy whatever you want with them. This has great potential where credit cards have limited penetration but you still want to have transactions over the internet. It’s also quite useful as a “universal” currency because you can avoid currency exchange fees when dealing internationally.
What sucks about Bitcoin
While there are few monetary barriers for a merchant to accept Bitcoin, it requires a lot of planning and constant vigilance. In the last year, Bitcoins have gone for as little as $0.20 and as much as $20. This causes havoc for a merchant who needs to buy things from his suppliers in dollars (or Euros, Yen, etc.) but gets paid in Bitcoin. The merchant needs to constantly be exchanging the Bitcoins he receives for his local currency or absorb some level of currency risk — and Bitcoin is a fairly volatile currency. Suddenly, a merchant needs to have a feel for the Bitcoin economy on top of understanding their business. In addition, exchanging Bitcoins for local currency involves fees (~0.5% usually) and a lack of fees was one of the main reasons a merchant would want to accept Bitcoin anyway. These problems can be mitigated somewhat by a merchant whose supply chain accepts Bitcoin.
For users, Bitcoin seems like a total pain. If I want to pay for something in Bitcoins, first I need to go to one of the exchanges and get myself some Bitcoins. This involves fees! People are not used to paying fees for buying things. In our credit card system, the merchant absorbs that fee and is often prohibited from passing that fee directly on to the consumer. The same problem of currency risk also arises. Holding Bitcoin in your wallet involves risk that the exchange rate could decline. Although the reverse is also true, rational people like to be compensated for taking risk. From a consumer’s perspective, Bitcoin is also not as safe as a credit card. If there is fraud on my credit card, I can reverse the charges. With Bitcoin, there’s no such luck. There’s no recourse short of a bad review for a merchant who sells you a defective product and won’t honor a return. This is the bad side of Bitcoin functioning like cash but on the internet.
The future of digital money?
For consumers, the proposition Bitcoin offers is generally awful. There’s no reason to pay with Bitcoin unless you are forced or incentivized to do so. For merchants, it makes considerably more sense but there are a few kinks to work out. I could see a future where merchants are provided with easy ways to manage or mitigate their currency risk and then they could offer incentives to customers who use Bitcoin because they can pass the fee savings from not using a credit card on to consumers. I could also see Bitcoin becoming/remaining fairly strong where anonymity or decentralization is very strongly desired or required (funding Wikileaks, gambling, illicit materials, private minded folks) or where banking infrastructure is not setup.
That said, I intend to procure and spend some Bitcoins but mostly for novelty reasons.