ZFS Performance Analysis and Tools

Brendan Gregg’s talk at ZFS Day (an event I also organized and ran).

The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.

American Freedoms

For those who frame the gun control debate as a matter of your personal freedoms, let’s look at that argument from another angle:

When I was young, in the 1960s and 70s, I did not have a choice about breathing cigarette smoke. I never smoked myself, but many people around me did (including my parents), and they could do so in public spaces: restaurants, planes, offices, etc. Over the following decades, the health risks of breathing secondhand smoke came to be seen as large enough to warrant legislation to protect those who do not choose to smoke. Smoking is still legal, but the right to smoke in shared spaces is now sharply curbed, so I can easily avoid exposing myself. Most of us think this is a good thing, a pragmatic matter of public health and personal choice.

In 1989, I took my infant daughter for her first checkup at the Yale New Haven health center. The pediatrician asked me a long list of questions about health and safety factors in the environment my daughter would be growing up in: did you bring her here in a car seat? does your apartment have any old lead paint? And: do you have a gun in the home, or do you visit the home of anyone who does? This brought into sharp focus a problem I hadn’t realized I would face as a parent. I can decide not to have a gun in my own home, but I can’t know whether every other environment my daughter is ever in (say, a friend’s house) may contain guns, or whether they are secured properly against curious little children.

US law gives me the choice to protect myself from cigarette smoke, but not to protect myself from the more immediately deadly risk of gunshot wounds. And I don’t mean “protect myself” by having my own gun at the ready. Since I am not trained to it, the odds of me successfully defending myself with a gun, against a gun, are very slim. This goes for you, too. Unless you are current or former military or police, or otherwise have extensive and constantly-reinforced training – not only with guns but in crisis situations – you are also not likely to be effective in using a gun in a sudden attack.

Yes, guns are sometimes used successfully in self-defense. But does the number of those successes outweigh the number of deaths that could otherwise be avoided by having fewer guns in the homes and hands of ordinary, untrained citizens?

Having guns in your home actually increases the risk that someone in your family will get hurt. Massacres committed by mentally ill people get attention, but they account for far fewer deaths (and injuries) than the accidents, suicides, and heat-of-the-moment murders that can happen so easily when a gun is readily to hand – and these account for many of the 30,000 gun deaths that occur in America EVERY YEAR.

I would like to have a choice about whether to expose myself to the risk of injury or death from flying bullets. You can choose to own a gun, and in many states you can choose to carry it into the public spaces that I also use. I do not have any choice about whether to be in your line of fire when you lose your temper, or think you’re gonna be a hero when something goes down. And, frankly, even if you’re the good guy, in the heat of the moment I don’t trust you to hit the bad guy rather than me. Some might keep their heads sufficiently to do exactly the right thing, but most won’t.

So, gun control is a matter of protecting freedom: my freedom to choose the risks to which I expose myself and my family. Your carrying a gun infringes on my right to be safe from your bullets. Even if we start from the premise that your right to be armed is as important as my right to be safe, there are pragmatic public health reasons for my right, in this case, to be given more weight.

DTracing the Cloud

Brendan Gregg at illumos Day (an event I also organized and ran).

Cloud computing facilitates rapid deployment and scaling, often pushing high load at applications under continual development. DTrace allows immediate analysis of issues on live production systems even in these demanding environments – no need to restart or run a special debug kernel. For the illumos kernel, DTrace has been enhanced to support cloud computing, providing more observation capabilities to zones as used by Joyent SmartMachine customers. DTrace is also frequently used by the cloud operators to analyze systems and verify performance isolation of tenants. This talk covers DTrace in the illumos-based cloud, showing examples of real-world performance wins.

To My Christian Friends (the Ones Who Are Currently Upset)

I am, of course, very happy that Barack Obama will be the US President for the next four years. I am thrilled that the tide is beginning to turn to give my gay friends the same rights I have. I am deeply relieved that the Affordable Care Act will not be rolled back, so I can have far fewer fears about my own financial future and, for the next three years, my daughter’s. There is much else to be done, and I think this President will do it better. Enough, for now, on that topic.

I am sad, however, to know that the cultural divide in this country was not created by this election cycle (though it was certainly exacerbated), and it will not likely heal anytime soon.

I was recently shocked to realize that this gulf exists even between myself and some of my fellow Woodstock alumni. One woman a year or two older than myself posted one of those pictures on Facebook, a photo of a fetus in the womb, with an anti-abortion quote that I found irritatingly facile, reducing a complex issue to seeming simplicity (as so often happens on both sides of that debate). I responded, I thought, carefully and politely. Her response was also polite, but included something about how she and I have very different worldviews, because she believes in ineluctable laws that come from God, and I do not.

In spite of what she perceives as a fundamental and important difference between us, she and I have not lived very different lives, nor do we have very different attitudes about right and wrong. I didn’t know her well in school, but I’m pretty sure she was there because her parents were missionaries, probably in India or Nepal. Most of those missionaries were doing selfless, valuable work – building and staffing hospitals and schools, etc. – which I respected, even when their religious motivation for doing it did not appeal to me.

I ended up at Woodstock School because my dad was head of Save the Children in Bangladesh, doing much the same work as the missionaries. He did it in the name of justice and humanity, not specifically in the name of God. I don’t see that that made any difference. His organization worked alongside many others, religious and otherwise. He had previously been in Vietnam as a civilian with USAID, and later worked with other organizations in Thailand, Indonesia, the Maldives…

Just like those missionary kids’, my childhood was “sacrificed” for my father’s ideals. I don’t regret that, any more than my schoolmate likely regrets growing up as a mish kid. And, like her, I naturally grew up with a sense that serving others and making the world a better place is important and worthy work. She has gone on to become a nurse (obviously, a serving profession). I’ve ended up working in technology, but in my career I have focused on making tech a better place for people – helping communication flow from customers to creators and back, helping tech communities work together, and so on.

Although I will loudly defend anybody’s right to do just about anything in their personal lives, so long as no one else gets hurt, in my own personal life (I am amused to note), I have lived closer to Christian ideals than many people who call themselves Christian. My few relationships (which have been with men) have been monogamous – not because I think sex outside a relationship is necessarily evil (see Dan Savage on being “monogamish“), it just happens that way. I make absolutely no judgements on what works for others.

I was faithfully married to one man for 20 years, have had one pregnancy and one child. No abortions, because I was usually careful with birth control – and I was lucky. I’ve never smoked, rarely drink to excess, don’t do drugs (no interest in them). I am kind, loving and courteous (most of the time). I pay my taxes, obey the speed limit, give to charity, consume responsibly, and generally try to be a force for good in the world, in my small way. Why? I guess because I was both born and raised that way.

During the heated political debates of recent weeks, another schoolmate challenged me to say where my morals come from, if not from God. There’s a great deal of research being done on precisely this, showing, for example, that other primates have a sense of fairness. My current groping towards an understanding is that we humans evolved (from and alongside other primates) to have a sense of “good” and “evil” in our dealings with one another, and this moral sense exists because it has helped us survive as a species. As to why many people have and do believe that this moral sense comes from a supernatural source, I recommend Dennett’s Breaking the Spell: Religion as a Natural Phenomenon.

I don’t actually care where you think your moral sense comes from, so long as it harmonizes with my own moral sense enough not to impinge on my life or anyone else’s. If your moral sense “requires” you to convert everyone forcibly to your religion, or cut off your daughter’s clitoris, or kill your daughter because she has “disohonored” the family, or to kill gay people, or control women, or enslave anybody – then, yes, I have a problem with your beliefs. Beyond these and other extreme examples, and some outward trappings, your religiously-driven behavior is not distinguishable from my own atheist ethics.

It makes me sad to think that half my country has somehow come to believe that the other half is morally incomprehensible and is dragging the country to its doom. (Yes, I get that this applies to both sides.) I think I have a pretty good picture of how those on the other side of this crevasse from me think in general (though I’m willing to be educated – politely), but I’m puzzled on this particular point: what makes you believe that I’m so terribly different from you?

I’ve been thinking about that a great deal. I have lifelong experience dealing with and living in cultures that are “foreign” to me. I, of all people, should be able to communicate, especially with those who share large parts of my background. So… is there maybe something I can do to help you understand me and people like me? Something that will make you feel better about where we want to take the country? Can we find some common ground? I would like to. Please talk to me.

 

 

Making the Impossible Possible: Disposable Staging Environments At Scale

Eric Sproul of OmniTI speaks on ZFS Day.

How does one go about running intensive business-intelligence (BI) database queries on rapidly changing data? We found ourselves pondering this question while helping Etsy do periodic reporting on their massive BI database. We harnessed the power of ZFS to create lightweight, yet writable, clones of production data that could be deleted when the report was done. The side effect was that we could also stand up staging environments at any time to test upgrades, schema changes and anything else that needed to be validated against real data.

Deirdré Straughan on Italy, India, the Internet, the world, and now Australia