In the last few months, I’ve spent a lot of time on the DTrace book: copy editing, managing the review process, and (ongoing) marketing – keep an eye out for video! Also provided care and feeding and a quiet place to work for one of its authors, Brendan Gregg, shown above.
Brendan and co-author Jim Mauro did an insane amount of work for this book, spending many late nights and long hours to write nearly 1000 pages (before final publishing layout) of mostly original material: 57 topics covered in over 150 new scripts and 150 new one-liners (beyond Brendan’s existing DTrace Toolkit), requiring a lot of new thought and invention. It all adds up to a comprehensive DTrace cookbook that will be useful to sysadmins, developers, students, or anyone needing to provide support and/or performance troubleshooting for systems running Solaris, MacOS X, or FreeBSD.
Here’s the Table of Contents:
1. Introduction
2. The D Language
3. System View
4. Disk I/O
5. File Systems – sample chapter available
6. Network Lower Level Protocols
7. Application Level Protocols
8. Languages
9. Applications
10. Databases
11. Security
12. Kernel
13. Tools
14. Tips and Tricks
Appendices:
A. DTrace Tuneables
B. D Language Reference
C. Provider Args Reference
D. FreeBSD Guide
E. USDT Example
F. Error messages
G. DTrace Cheatsheet
Glossary
The picture below was taken in the final throes of writing, when Brendan got down on the floor to plug in his laptop, and didn’t have time to get up again: he was chatting online with Jim, ironing out last-minute details. The manuscript is now with the publisher for final editing and layout – and we can all breathe a sigh of relief.
NB: This is a companion post to How ZFS Really Gets Done, in what might be an ongoing series about “My Life Among the Geeks” (hey, I’m allowed to use the term – I’m a geek, too).
I’m a longtime reader of the newsletters/blog of fellow Woodstock alum Jim Taylor – who has been a professional writer for far longer than I have, and is a very wise man. This week’s “Sharp Edges” column is about the disastrous floods in Pakistan, wondering Why the reluctance to help? He shares some likely correct ideas about why “international aid has been disastrously slow”, ending with a cogent thought about why this is a huge mistake. I agree with everything he says, and recommend that you read it now.
But I think there’s more to this story, and wrote Jim as follows:
One thing that has surprised me since I moved from Italy back to the US is the constant pleas to give-give-give. Everyone has their “pet” cause – often quite literally about pets – and is constantly asking for “support”. Fundraising is a business in which people are paid to stalk city streets with clipboards, waylaying passersby with earnest questions. Every corporation and celebrity hastens to associate their name with some cause, preferably a non-controversial one that won’t tarnish that name with any possible customer or fan.
The result, for me at least, is a constant low-level feeling of guilt that I’m never doing enough to save the world. No matter that I gave my childhood for my father to “develop” Vietnam, Thailand, and Bangladesh. Never mind that I have given considerable time, travel costs, and professional expertise to Woodstock and SAGE. I’m a bad person if I don’t also save dolphins, pit bulls, starving children, forests, etc.
The most bizarre request I’ve seen was in a sandwich shop in Colorado: taped to the counter at the cash registers was a flyer made up and printed by an employee, concerning the daughter of his neighbor. The child was killed by a car while playing outside her home “just minutes after this picture was taken!” and we were all now invited to help the family defray funeral costs.
It’s as if we are all expected to participate, emotionally and financially, in every tragedy in the world. I don’t think the human psyche is designed to deal with that much grief, even vicariously.
So when one more cause comes along, no matter how worthy, I just shut down. It feels like yet another imposition on my emotional energy. I don’t know whether that makes me a bad person, or simply a burnt-out charity case.
In the last three and a half years, working for Sun Microsystems and now Oracle, I have produced over 300 video assets, ranging in length from 10 minutes to 3 hours. Most of this material is software engineers talking about deeply technical topics.
By YouTube standards, our audience isn’t large: my videos have had a few hundred to a few thousand views apiece. So why bother? What’s the ROI?
This breaks down into two underlying questions:
Why share this kind of information at all?
Why do it in the form of video rather than, say, technical white papers?
Why Share Technical Information?
Although most of my videos have a limited potential audience, those who watch them are the system administrators, developers, and other techies who use our technology in jobs revolving around large, complex systems for hugely complex computing and storage tasks. They are influencers, if not direct decision-makers, in major IT purchases. They prefer to get their information from those who know it best – the engineers who create the products – and they don’t want any marketing spin on it. To these folks, great engineers are gurus, and access to our engineers’ knowledge is a selling point. My technical videos never say: “Buy this technology, it’s great!” They don’t need to, because they feature the engineers who designed it telling you why it’s great.
Why Use Video?
More Effective Learning
Although the percentages on the “Cone of Learning” are open to question (in fact, the author of the original, Edgar Dale, disavows any such numbers on his original diagram), the hierarchical concept itself is common sense. Think back to your own education. Most people find it hard to learn simply by reading. (Otherwise, why take notes from textbooks?) You absorb more by seeing and hearing an engaging teacher. Better still are small, seminar-style classes in which you actively participate. Next on the hierarchy is hands-on learning where you do something yourself (how vivid are your memories of dissecting frogs in high school science?).
Seminar-style learning and hands-on training are beyond the scope of my current job (Sun/Oracle offers classes through another department). But we can certainly do more to engage and instruct our audiences than plain old text.
More Efficient Information Transfer
Top engineers are extremely valuable people whose working hours, from their employers’ point of view, are best spent coding. Even those who have the (quite different from coding) skills and inclination to write papers or blog posts, often simply don’t have the time.
It didn’t take me long at Sun to realize that the most efficient way to get information out of engineers was to film them. This was even easier when, as often happened, they were already creating presentations for conferences or internal seminars. It cost them no extra time or effort for me to film, edit, and publish video of that same presentation. (Video also extends the reach of such presentations to people around the world who cannot attend them in person.)
Outside of formal presentations, you can still tap engineering expertise for video. Sometimes it’s just a matter of getting the right people in a room and letting them talk. Back in March, we had a rare confluence of three of our top performance engineers (Jim Mauro, Brendan Gregg, and Roch Bourbonnais) in the same city at the same time. With Dominic Kay chaperoning, we spent about four hours in a conference room, resulting in at least three hours of usable video (not all of it published yet). As Brendan later pointed out, among all possible forms of technical information transfer, this was by far the most efficient return on his time.
You may be wondering how much it cost to produce all this video. I have a very complicated spreadsheet in which I track details on every event I’ve filmed. I determined costs, calculating in my time (shooting, editing, managing) and travel, the cases where I did the shooting but paid someone else to edit, or I received tape that someone else had shot and I did the editing, etc. I then divided by the amount of video finally published from each specific event (some trips/events resulted in hours of published video, some only minutes).
With all the variables calculated for, costs ranged from $150 to $1400 per hour of published video. For comparison, I asked our marketing folks how much they were paying for “professional” video production. Their estimate was $3,000-5,000 per video (talking heads in the studio, usually), none of which was longer than 15 minutes. So figure $12,000 to $20,000 per hour.
Guerrilla video is definitely more cost-effective.
And its reach can be larger than anticipated:
I can’t claim any credit for this (though I did post a making of long after). It was filmed and posted by Bryan Cantrill in about half an hour on New Year’s Eve, 2008. Between the humorous presentation, the technical content and the discussion it raised, this thing went viral in a hurry, becoming the most-viewed video ever made about Sun Microsystems (650,000770,929 over 900k views on YouTube to date). It continues to generate conversation in the market and with current and potential customers – which is, in the end, the real point.
So the major reason to put geeks engineers in front of video cameras is this: IT WORKS.
^ top: The Perf Trio: Jim Mauro, Brendan Gregg, and Roch Bourbonnais
^ Per Jonas’ suggestion, I hung part of the Tibetan prayer flags in the bathroom, which was otherwise very stark. Then I bought some very colorful towels from Ikea to match.
^ This is the anteroom leading into my bathroom and walk-in closet, very handy because I can dress in here (there’s another door behind the camera’s POV) without having to close the curtains in the bedroom. Hence the mirror.
The maroon things hanging above the doors are torans which I used as window treatments in Colorado. There wasn’t any other place to put them here. You can see my jewelry “organization” scheme hanging on the door ahead. The scarves to the right are there for decoration, though I suppose I could also wear them. Behind that door is an alcove stacked with suitcases. There’s a Wendy Pini calendar in this picture as well.
As usual, most of these items carry memories. The prayer flags were a birthday gift from Woodstock classmate Teeran, who also had a cake baked for my birthday party in Mussoorie in November, 2007.
