Category Archives: what I do

Developers Rule, OK?

I warmly recommend a new book by RedMonk co-founder Stephen O’Grady called The New Kingmakers, “about how developers took over the world”. If you’re a non-tech person who wants to understand what I do in my professional life, this will help. I’m not a software developer (nor even play one on television), but a lot of my job is about helping devs communicate about their work, and helping them work as a community to reach shared goals. If you work in tech, O’Grady’s view has profound implications for how you organize and manage your company, treat your tech employees, and market your products to technical people.

O’Grady posits that: “Developers are the most important constituency in technology. They have the power to make or break businesses, whether by their preferences, their passions, or their own products.”

The company that I work for, Joyent, is both a creator and a beneficiary of this new world order. We help supply the tools that enable developers to take over the world: open source software, which we make, and hardware, which we operate so efficiently that we can afford to rent it to other developers for very, very little – aka cloud computing. As O’Grady says, “With the creation of the cloud market, developers had, for the first time in history, access to both no-cost software and infrastructure affordable for even an individual.”

This means that a skilled software engineer with an idea (and maybe a few smart friends) and very little cash can launch a business to see what the world thinks of that idea, and – who knows? – may eventually build it into a world-spanning company. Twitter, for example, in its early days was hosted on Joyent.

O’Grady gives a quick history of “How did we get here?” – helpful for those who do not live and breathe tech every day – then supports his case with The Evidence.

In a subsection titled “What would a developer’s world look like?”, one answer he gives is that: “…open source [software] would grow and proliferate. Whether it’s because they enjoy the collaboration, abhor unnecessary duplication of effort, because they’re building a resume of code, because they find it easy to obtain, or because it costs them nothing, developers prefer open source over proprietary commercial alternatives in the majority of cases.”

Perhaps for brevity’s sake, he left out some reasons that the devs I know want to open source their work, such as:

  • They believe deeply in open source principles.
  • They are artists, and open source code is their gallery show, where their peers can see, admire and use their work – this goes well beyond resumé building.
  • Recognition is also a form of compensation. Like all of us, coders bask in the admiration of their peers, especially those peers who are skilled enough to truly understand the quality of their work. Having their best work locked away behind a proprietary wall makes this impossible, obviously.
  • Very pragmatically, they want their own best tools to be available for their own later use. As Bryan Cantrill has said about his “baby”, DTrace: “it was developed out of pain”, to solve problems that he (and many others) face every day in dealing with huge, complex systems. He would not want to work in a world without DTrace.

O’Grady’s recommendations for succeeding in this new world range from “Get to them early” to “talk with developers, not at them” – all good advice, including solid recommendations on how to market to developers (hint: traditional marketing tools fail completely, beer works).

You can get a copy of the book on Kindle. 50 pages.

photo caption: To do my job well, it helps me to be around devs all the time, which isn’t hard since I report to Joyent’s CTO, and sit among the engineers at our SF office – the spot of red in the photo above is my jacket on the back of my chair. SVP of Engineering Bryan Cantrill is the one with his feet on his desk, right foreground. (This was taken in the first few days in that office – Bryan’s desk has never since been that clean.)

The title of this piece, as so often happens with my writing, is a pun that may need explaining (and therefore may not be funny to anyone but me – oh, well).

ZFS Performance Analysis and Tools

Brendan Gregg’s talk at ZFS Day (an event I also organized and ran).

The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.

DTracing the Cloud

Brendan Gregg at illumos Day (an event I also organized and ran).

Cloud computing facilitates rapid deployment and scaling, often pushing high load at applications under continual development. DTrace allows immediate analysis of issues on live production systems even in these demanding environments – no need to restart or run a special debug kernel. For the illumos kernel, DTrace has been enhanced to support cloud computing, providing more observation capabilities to zones as used by Joyent SmartMachine customers. DTrace is also frequently used by the cloud operators to analyze systems and verify performance isolation of tenants. This talk covers DTrace in the illumos-based cloud, showing examples of real-world performance wins.

Making the Impossible Possible: Disposable Staging Environments At Scale

Eric Sproul of OmniTI speaks on ZFS Day.

How does one go about running intensive business-intelligence (BI) database queries on rapidly changing data? We found ourselves pondering this question while helping Etsy do periodic reporting on their massive BI database. We harnessed the power of ZFS to create lightweight, yet writable, clones of production data that could be deleted when the report was done. The side effect was that we could also stand up staging environments at any time to test upgrades, schema changes and anything else that needed to be validated against real data.

ZFS Day Panel: The State of ZFS on…

A panel at ZFS Day covering the state of ZFS on Linux, FreeBSD, illumos, and more, moderated by Deirdré Straughan wih Matt Ahrens, Justin Gibbs, Will Andrews, Don Brady, Brian Behlendorf