How does one go about running intensive business-intelligence (BI) database queries on rapidly changing data? We found ourselves pondering this question while helping Etsy do periodic reporting on their massive BI database. We harnessed the power of ZFS to create lightweight, yet writable, clones of production data that could be deleted when the report was done. The side effect was that we could also stand up staging environments at any time to test upgrades, schema changes and anything else that needed to be validated against real data.
A panel at ZFS Day covering the state of ZFS on Linux, FreeBSD, illumos, and more, moderated by Deirdré Straughan wih Matt Ahrens, Justin Gibbs, Will Andrews, Don Brady, Brian Behlendorf
SmartOS is an Illumos-derived distro developed by Joyent as the base OS for their cloud. This presentation focuses on the OS virtualization capabilities of SmartOS and details the extensive
enhancements made to the OS, based on real-world production experience deploying SmartOS in the Joyent cloud.
For over 30 years, hard drives have designated the smallest storage location as 512 bytes. In January 2011, all major hard drive manufactures began shipping their hard drive platforms using a new standard called Advanced Format. To aid in the transition, these new hard drives provide a 512 byte emulation mode that allows the drives to advertise themselves as a 512 byte addressable devices. This can severely impact write performance resulting in the need for read-modify-write operations for any misaligned or partial writes that are issued.
The problem is not limited to just physical hardware. Other storage platforms may also provide LUNs (logical unit number) that presents themselves as a 512 byte addressable devices when, in fact, they use a 4K sector size internally. Although ZFS has built-in support for 4K sectors, it has no automatic way of dealing with the lies that the storage devices tell. This talk will focus on the methods that have been developed to work around the lies that hard drive storage platforms tell and will discuss the challenges and drawbacks that come with using 4K sectors.
