Developers Rule, OK?

I warmly recommend a new book by RedMonk co-founder Stephen O’Grady called The New Kingmakers, “about how developers took over the world”. If you’re a non-tech person who wants to understand what I do in my professional life, this will help. I’m not a software developer (nor even play one on television), but a lot of my job is about helping devs communicate about their work, and helping them work as a community to reach shared goals. If you work in tech, O’Grady’s view has profound implications for how you organize and manage your company, treat your tech employees, and market your products to technical people.

O’Grady posits that: “Developers are the most important constituency in technology. They have the power to make or break businesses, whether by their preferences, their passions, or their own products.”

The company that I work for, Joyent, is both a creator and a beneficiary of this new world order. We help supply the tools that enable developers to take over the world: open source software, which we make, and hardware, which we operate so efficiently that we can afford to rent it to other developers for very, very little – aka cloud computing. As O’Grady says, “With the creation of the cloud market, developers had, for the first time in history, access to both no-cost software and infrastructure affordable for even an individual.”

This means that a skilled software engineer with an idea (and maybe a few smart friends) and very little cash can launch a business to see what the world thinks of that idea, and – who knows? – may eventually build it into a world-spanning company. Twitter, for example, in its early days was hosted on Joyent.

O’Grady gives a quick history of “How did we get here?” – helpful for those who do not live and breathe tech every day – then supports his case with The Evidence.

In a subsection titled “What would a developer’s world look like?”, one answer he gives is that: “…open source [software] would grow and proliferate. Whether it’s because they enjoy the collaboration, abhor unnecessary duplication of effort, because they’re building a resume of code, because they find it easy to obtain, or because it costs them nothing, developers prefer open source over proprietary commercial alternatives in the majority of cases.”

Perhaps for brevity’s sake, he left out some reasons that the devs I know want to open source their work, such as:

  • They believe deeply in open source principles.
  • They are artists, and open source code is their gallery show, where their peers can see, admire and use their work – this goes well beyond resumé building.
  • Recognition is also a form of compensation. Like all of us, coders bask in the admiration of their peers, especially those peers who are skilled enough to truly understand the quality of their work. Having their best work locked away behind a proprietary wall makes this impossible, obviously.
  • Very pragmatically, they want their own best tools to be available for their own later use. As Bryan Cantrill has said about his “baby”, DTrace: “it was developed out of pain”, to solve problems that he (and many others) face every day in dealing with huge, complex systems. He would not want to work in a world without DTrace.

O’Grady’s recommendations for succeeding in this new world range from “Get to them early” to “talk with developers, not at them” – all good advice, including solid recommendations on how to market to developers (hint: traditional marketing tools fail completely, beer works).

You can get a copy of the book on Kindle. 50 pages.

photo caption: To do my job well, it helps me to be around devs all the time, which isn’t hard since I report to Joyent’s CTO, and sit among the engineers at our SF office – the spot of red in the photo above is my jacket on the back of my chair. SVP of Engineering Bryan Cantrill is the one with his feet on his desk, right foreground. (This was taken in the first few days in that office – Bryan’s desk has never since been that clean.)

The title of this piece, as so often happens with my writing, is a pun that may need explaining (and therefore may not be funny to anyone but me – oh, well).

Work in Progress: This Site Has Moved

I recently moved my site from DreamHost to Joyent. Why I didn’t do this sooner? After all, I’ve been working for Joyent for over two years, and one of the company perks is some free access to Joyent services.

The reason is: I’m not a typical Joyent customer. After 12+ years of keeping my own site running in various formats (and, more recently, running sites for friends and family), I know just enough to keep things going (most of the time) and mess things up (occasionally), but I have neither the skills nor desire to run websites for a living. I’ve had my site on Dreamhost for about 7 years now, for most of which I was a very happy customer. Part of Dreamhost’s service is to automate things that I would otherwise have struggled with, such as installing and running all the software needed to run a blog (Apache, WordPress, etc.) – Dreamhost made that very easy.

I knew that I could learn how to do those things if I really needed to, but I wasn’t keen to dedicate the time and brain space to the project – my life is busy enough without taking on the technical administration of a website that barely earns enough in advertising to pay its own bills. I was happy to pay Dreamhost to make that easy for me.

I began to have trouble with Dreamhost in 2010, when beginningwithi.com fell victim to the pharma hack: malware that redirected searches to my most popular pages to “Buy [Penis Drug]!” sites. The effect was not visible to me at first because, if you came to the site from a direct link (ie, you followed a link from me on Twitter or in my newsletter), you would get the page you expected. But eventually I realized that something was wrong because my website visitors dropped off drastically; 80% of my traffic usually comes from Google searches, and when those were led astray, I started losing traffic (and advertising money, such as it was).

Dreamhost support, after one or two pokes at it, essentially threw up their hands and said it was my problem. Very fortunately, I could call on one of the best minds in tech to solve it (though I hated to ask, because I knew exactly how busy he was).

Even for Brendan, it took two months to isolate the attacks to the point that he realized they were coming from another domain I was running on the same account on Dreamhost. We were never sure how I got hacked in the first place, but it seems likely that it was the result of a malicious WordPress plug-in that I had downloaded from wordpress.org – my assumption that WordPress vets those things turns out to have been naive.

Things ran fairly smoothly after we finally got that infection cleaned out. Until a few weeks ago, when I realized that my site had become very, very slow – up to 20 seconds to load the front page. In several exchanges of email with Dreamhost support, they blamed the problem on me (Brendan thought otherwise). Their suggestions for fixing it demanded far more know-how than I had. If I was going to learn all that, I preferred to learn it on systems where my arduously-gained new skills would at least be useful in my day job – that is, on Joyent.

The move was not easy; my blog is largeish, and had some old installation quirks that made it tricky. I had to learn a lot (with much more still to learn), as well as have extensive help from Brendan and my colleagues at Joyent support – I’m still not the typical Joyent customer, and may never be. I will save the story of how we did it for another post.

In the meantime, as Google is probably still re-indexing the site and some things have temporarily moved or been deleted, you are likely to hit broken links and missing images. Please feel free to let me know about these, though I probably won’t be able to fix everything quickly. I still have a busy day job.

Also: if you were or want to be a subscriber to the very-occasionally-published email edition of this site, I have moved that as well (thanks to my friends at ListBox!), and you can sign up here.

And if you were looking for my home page, click on the big title at the top of this one, or click here.

ZFS Performance Analysis and Tools

Brendan Gregg’s talk at ZFS Day (an event I also organized and ran).

The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.

American Freedoms

For those who frame the gun control debate as a matter of your personal freedoms, let’s look at that argument from another angle:

When I was young, in the 1960s and 70s, I did not have a choice about breathing cigarette smoke. I never smoked myself, but many people around me did (including my parents), and they could do so in public spaces: restaurants, planes, offices, etc. Over the following decades, the health risks of breathing secondhand smoke came to be seen as large enough to warrant legislation to protect those who do not choose to smoke. Smoking is still legal, but the right to smoke in shared spaces is now sharply curbed, so I can easily avoid exposing myself. Most of us think this is a good thing, a pragmatic matter of public health and personal choice.

In 1989, I took my infant daughter for her first checkup at the Yale New Haven health center. The pediatrician asked me a long list of questions about health and safety factors in the environment my daughter would be growing up in: did you bring her here in a car seat? does your apartment have any old lead paint? And: do you have a gun in the home, or do you visit the home of anyone who does? This brought into sharp focus a problem I hadn’t realized I would face as a parent. I can decide not to have a gun in my own home, but I can’t know whether every other environment my daughter is ever in (say, a friend’s house) may contain guns, or whether they are secured properly against curious little children.

US law gives me the choice to protect myself from cigarette smoke, but not to protect myself from the more immediately deadly risk of gunshot wounds. And I don’t mean “protect myself” by having my own gun at the ready. Since I am not trained to it, the odds of me successfully defending myself with a gun, against a gun, are very slim. This goes for you, too. Unless you are current or former military or police, or otherwise have extensive and constantly-reinforced training – not only with guns but in crisis situations – you are also not likely to be effective in using a gun in a sudden attack.

Yes, guns are sometimes used successfully in self-defense. But does the number of those successes outweigh the number of deaths that could otherwise be avoided by having fewer guns in the homes and hands of ordinary, untrained citizens?

Having guns in your home actually increases the risk that someone in your family will get hurt. Massacres committed by mentally ill people get attention, but they account for far fewer deaths (and injuries) than the accidents, suicides, and heat-of-the-moment murders that can happen so easily when a gun is readily to hand – and these account for many of the 30,000 gun deaths that occur in America EVERY YEAR.

I would like to have a choice about whether to expose myself to the risk of injury or death from flying bullets. You can choose to own a gun, and in many states you can choose to carry it into the public spaces that I also use. I do not have any choice about whether to be in your line of fire when you lose your temper, or think you’re gonna be a hero when something goes down. And, frankly, even if you’re the good guy, in the heat of the moment I don’t trust you to hit the bad guy rather than me. Some might keep their heads sufficiently to do exactly the right thing, but most won’t.

So, gun control is a matter of protecting freedom: my freedom to choose the risks to which I expose myself and my family. Your carrying a gun infringes on my right to be safe from your bullets. Even if we start from the premise that your right to be armed is as important as my right to be safe, there are pragmatic public health reasons for my right, in this case, to be given more weight.

DTracing the Cloud

Brendan Gregg at illumos Day (an event I also organized and ran).

Cloud computing facilitates rapid deployment and scaling, often pushing high load at applications under continual development. DTrace allows immediate analysis of issues on live production systems even in these demanding environments – no need to restart or run a special debug kernel. For the illumos kernel, DTrace has been enhanced to support cloud computing, providing more observation capabilities to zones as used by Joyent SmartMachine customers. DTrace is also frequently used by the cloud operators to analyze systems and verify performance isolation of tenants. This talk covers DTrace in the illumos-based cloud, showing examples of real-world performance wins.

Deirdré Straughan on Italy, India, the Internet, the world, and now Australia