Philosophy

Code longevity and rewrites

Posted on by David

A while ago I interviewed with a company planning to rewrite their existing money-making product with lots of users that consisted of CGI scripts with most of the business logic in Oracle stored procedures. The developers and management seemed sharp. All but one of the original product devs were gone and nobody really knew the system. The size of the rewrite was a concern of mine that I brought up in the interview. If stored procedures were a pain point, swapping them out piece by piece seemed wise to me, but they thought that a focusing their developers for a big switchover would be less costly. Their Python rewrite didn’t go too well. I believe they’re on Rewrite Part II with .NET and SQL Server after hiring an ex-Microsoft manager to take over the project. In the meantime, nobody wants to maintain on the old product which is on its way out.

Rewrite everything… all the time

Joel Spolsky calls rewrites the single worst strategic mistake a company can make. At the same time, only dead code is static. New features and extensions are a sign of a healthy project. Sometimes these improvements involve big and invasive changes. At his product tech talk a few months ago, Adam Wolff talked about the culture of Facebook and how they do software development. As part of their approach, he said that Facebook “rewrites everything all the time”. This strategy — at least on the surface — sounds similar to what Joel Spolsky sees as a complete disaster. Facebook can’t be making such huge mistakes constantly, though, or the competition would catch up. Adam mentioned Joel’s post in his talk and then went on to describe how Facebook developers find a bit of the stack with some code odor, carve out their niche and then rewrite it.

After thinking on this for a bit, I came to the realization that this is not a rewrite at all. At least it isn’t a rewrite in the same way Joel described Netscape rewriting Navigator. Facebook as a whole hasn’t changed. A part of it was just ripped out and replaced with a newer version with all the critical features of the last revision and more. In my mind, I would characterize that as a refactor. I don’t want to argue semantics, but Facebook’s approach — after some reflection — seems quite good. Code that doesn’t change doesn’t get looked at and when the original developers leave, a company can get left with a code base nobody knows. Jeff Atwood says that understanding means rewriting and so hopefully Facebook has a system that’s been through many iterations and a number of people understand. When my page became a timeline, a number of engineers learned a lot I bet.

Size, among other things, does matter

It is very telling that the Safari team didn’t choose to write a browser from scratch. Instead, they took a relatively small 140kloc layout and JavaScript engine and then built all the browser bits around it (having a smart and talented team along with some Netscape hindsight might have helped too). By building the pieces around it, they also built up some expertise in the initial code base as well. While Safari may not currently be the talk of the browser town, there’s little doubt that WebKit is a big success. So if rewriting a browser from scratch is crazy talk that will sink your company and doing a full rewrite on say your personal blog is OK, what’s the difference? One big difference is size. I could switch my blog from WordPress to a custom site of my own design in about a day. In his talk, Adam described a big shift in Facebook’s deployment system to use BitTorrent that a few engineers wrote in a weekend hackathon to overcome problems with the speed of deploying the latest software to their fleet. Developing a browser from scratch, by contrast, is a multi-man-year effort.

Legacy of code

Code becomes legacy code when nobody touches it for a long time, the developers who know it leave, and everything else changes around it. While code doesn’t rust or degrade with time, the know-how around it does fade away. Developers want to create something great that will be used for a long time; they are architects at heart. However, they don’t want to maintain a bloated, poorly documented system built by somebody else (who is probably gone and creating the next bloated, poorly documented system). The best way I’ve seen as a developer to keep code from becoming “legacy” is to make it my own. To make it my own doesn’t mean rewriting the whole thing from scratch, but rather making incremental improvement where I can or swapping out components where it makes sense. This builds a sense of ownership for the next generation of maintainers. The key here is to write software such that making these types of improvements or swapping things out is easy. Good APIs and separation of concerns are critical.

When big changes are on the horizon and the future of a software project looks least sure, I’ve found that breaking a project into smaller pieces always helps. Unless you end up working on something completely different, at least some of the old code will always be useful to the new project provided the API makes it easy to integrate. To give a concrete example from a project I’m working on relating to automation against an array of Android phones, a solid Python interface to ADB (the Android debug bridge) will almost definitely be worthwhile regardless of what else changes. Hopefully the legacy I leave won’t be a large unmaintainable system but a large number of small systems that are easier to understand.

Thoughts on Bitcoin

Posted on by David

This is going to be somewhat of a departure from your regularly scheduled programming here. I had some extra time over the past week or so and I decided to learn what I could about Bitcoin. This rant is going to contain some poor semblance of economic theory and really I should just stick to tech but I couldn’t resist.

Skip this if you already understand Bitcoin

Bitcoin is a peer-to-peer currency that has gotten some press lately. If I have a Bitcoin wallet (and I do), you can easily send me some “money”. I didn’t read through the code, but my understanding is that this is handled by adding my Bitcoin address to the digital data, signing it with your private key and then broadcasting it to the world. In that way, Bitcoin participants have the entire history of every transaction and only one person possesses any given Bitcoin at any given time. As you might guess, if everybody started using Bitcoin tomorrow, scalability might be a problem. However, it works for now.

What rocks about Bitcoin

A lot of different companies are trying to “solve” payments — be they mobile, physical, digital or anything else. One reason why folks think our current (credit card based) system needs solving is that it is pricey for merchants. The generally expected fees are somewhere just shy of 3% and can also include a fixed transaction fee as well. Fixed fees mean that credit card transactions don’t scale down very well when you are buying something cheap. Ever seen a sign saying $10 minimum for credit cards? That’s why. Bitcoin payments are virtually free which sounds awesome from a merchant’s perspective. They are not generally reversible unless the receiver chooses to reverse it and there’s no central authority to decide who is allowed to pay whom and for what. This is particularly useful if you’re Wikileaks and you’ve been cut off from payment processors or you operate a gambling website and you’re having problems with US banks stopping or reversing your transactions.

For regular users, Bitcoin comes with some advantages such as some level of privacy. No more embarrassing credit card statements. Jokes aside, anonymity of payment has quite a few legitimate purposes and is obviously useful for illegitimate purposes. In addition, Bitcoin requires very little upfront cost. There’s no need to setup a bank account or have a credit limit. It functions like cash but over the internet. You get yourself some Bitcoins and buy whatever you want with them. This has great potential where credit cards have limited penetration but you still want to have transactions over the internet. It’s also quite useful as a “universal” currency because you can avoid currency exchange fees when dealing internationally.

What sucks about Bitcoin

While there are few monetary barriers for a merchant to accept Bitcoin, it requires a lot of planning and constant vigilance. In the last year, Bitcoins have gone for as little as $0.20 and as much as $20. This causes havoc for a merchant who needs to buy things from his suppliers in dollars (or Euros, Yen, etc.) but gets paid in Bitcoin. The merchant needs to constantly be exchanging the Bitcoins he receives for his local currency or absorb some level of currency risk — and Bitcoin is a fairly volatile currency. Suddenly, a merchant needs to have a feel for the Bitcoin economy on top of understanding their business. In addition, exchanging Bitcoins for local currency involves fees (~0.5% usually) and a lack of fees was one of the main reasons a merchant would want to accept Bitcoin anyway. These problems can be mitigated somewhat by a merchant whose supply chain accepts Bitcoin.

For users, Bitcoin seems like a total pain. If I want to pay for something in Bitcoins, first I need to go to one of the exchanges and get myself some Bitcoins. This involves fees! People are not used to paying fees for buying things. In our credit card system, the merchant absorbs that fee and is often prohibited from passing that fee directly on to the consumer. The same problem of currency risk also arises. Holding Bitcoin in your wallet involves risk that the exchange rate could decline. Although the reverse is also true, rational people like to be compensated for taking risk. From a consumer’s perspective, Bitcoin is also not as safe as a credit card. If there is fraud on my credit card, I can reverse the charges. With Bitcoin, there’s no such luck. There’s no recourse short of a bad review for a merchant who sells you a defective product and won’t honor a return. This is the bad side of Bitcoin functioning like cash but on the internet.

The future of digital money?

For consumers, the proposition Bitcoin offers is generally awful. There’s no reason to pay with Bitcoin unless you are forced or incentivized to do so. For merchants, it makes considerably more sense but there are a few kinks to work out. I could see a future where merchants are provided with easy ways to manage or mitigate their currency risk and then they could offer incentives to customers who use Bitcoin because they can pass the fee savings from not using a credit card on to consumers. I could also see Bitcoin becoming/remaining fairly strong where anonymity or decentralization is very strongly desired or required (funding Wikileaks, gambling, illicit materials, private minded folks) or where banking infrastructure is not setup.

That said, I intend to procure and spend some Bitcoins but mostly for novelty reasons.

Code Happiness

Posted on by David

While I was reading Delivering Happiness on my vacation, I got to thinking about how it applies to me. Tony Hsieh claims that a key factor in Zappo’s success was making sure his employees, customers and suppliers were happy. So what are the best ways to keep developers, management and customers happy in a software setting?

For open source projects, there is often no management. Developers are giving away their work so you would assume that if they weren’t happy they could just move on to a different project. Users of OSS are usually happy to get something for nothing but at times I’ve seen them get picky and demanding. The situation for companies is quite a bit different. Quite often management is telling a developer what to do. Marketing or sales has promised the world and customers are unhappy at what is delivered. Senior management is breathing down the neck of middle management to get the project done faster, cheaper and better. So why are so many talented people willing to give their time — which is worth a fortune — to open source and at the same time companies struggle to keep their developers happy and engaged? Why also are a lot of users (especially power users) satisfied with open source but not satisfied with commercial software? How can management simultaneously keep developers happy and also ship products on time and on budget?

Understanding and recognition

In a commercial setting, there’s often a barrier between who worked on the code and who the users (customers) are. Developers like to be recognized for their hard work like they are in open source. It gives them a sense of pride. It certainly does for me. Now I’m not suggesting that companies should tear down the wall between developers and customers completely and always have an easy direct line of interaction between them. However, there should probably be a door on that wall. Developers may be surprised at how their creations are being used and they can come to a better agreement about new features or a change of direction. Users also may better understand why a feature was done a certain way instead of simply thinking that the software was done poorly. Open source usually has a direct line but once a project is big enough there is also a users group to field the easier questions and concerns. This satisfies developers’ desire for recognition while also allowing for good communication with customers

Ownership – the key to happiness

The idea of having developers interfacing with and understanding the wants of customers suggests that the developers, not strictly management should be at least partially in charge of what features get developed. If management can’t convince developers that their work is important, the project is doomed to failure. Nothing can lower the morale of a team of developers faster than knowing that they are working on something that they view as valueless. People want to feel that their work is useful. Problems of code ownership are for the most part unique to commercial software since there’s always a sense of ownership with open source.

On the other side of the same coin, why have Fedex days at Atlassian and 20% time at Google been so successful at motivating developers even when management is removed from control? The answer, I think, is three-fold. Firstly, I think concept of pride and recognition play into it. Secondly, developers like to have that control and not always have management dictate to them. The other reason is that when a developer works on a codebase more than a certain amount, they become uniquely qualified to make certain improvements: refactors that only they can make or features that only they could think of. In a word, they become the master of their domain and they just need the time to show it.

There’s an entire TED talk on the topics of autonomy, mastery and purpose that is worth watching. While the conclusion is that those are better motivators, I think they also lead to more happiness.

So what about management?

So if management isn’t really as important in telling developers what to do, do we still need them? While open source projects get by just fine without them, I think managers are necessary for commercial projects. However, instead of telling people what to do, I think managers (and developers) would be happier if they functioned more like idea-men. They plant the seeds of ideas into the minds of developers and see what takes root. While emersed in a project, occasionally it is difficult to sit back and take in the big picture. However, somebody needs to keep it in sight while also having an eye for improvements.

What to do about it?

One other message from Hsieh’s book is that it is never too late for a change. If your management is draconion and completely removes you (the developer) from the process, it might be time to explore other options. Likewise, if you’re a manager and you basically function as a middleman and your superiors yell at you to go yell at your subordinates, there are better opportunities out there. You might also take these things into account as a consumer of software as well. You might just end up a little happier.

Update (March 15, 2011): There’s a good article at NYT about Google’s approach to management.

Here’s a picture of some of my friends on a trike taxi in the Philippines.