Your NAS is full to the brim with data. What are your options?
To recap: the home-built NAS holds eight 3TB Seagate drives and contains backups and user data for the household, as well as my media library. And it’s mirrored onto a backup server which contains the same data.
We elected to use ZFS as our filesystem, for reasons discussed in part 1. Available space on the main server is currently less than 3TB, with over 13TB already used. That’s 81% of the total and for ZFS that’s about as full as it should get. Above 80% ZFS starts to slow down, as it needs that space for its reliability-enhancing technology.
At this point there are quite a few concepts that need expanding, so it’s time to back up and give you the full picture.
Diving into the Detail
Part 1 introduced you to the LSI Logic SAS 9211-81 disk controller, based around the company’s SAS2008 chip. It’s widely used in enterprise and very popular among fans of home-build devices because of the number of drives it allows you to add.
The big snag from our point of view is that this disk controller has its own ideas of how RAID should be implemented. ZFS takes care of its own RAID implementation in software and those RAID features built into the SAS2008 chip are going to mess this up. So the first thing to do is to get rid of them.
Crossflashing: OpenZFS is an all-encompassing file system. To achieve its high reliability it manages the disks at as low a level as the disk will allow. So all that OpenZFS demands from a disk controller is that it should reliably transport raw data.
Hardware RAID controllers, on the other hand, can muddy the waters. Even if you switch off their RAID functionality—and not all controllers allow this—the RAID firmware is still in the loop, adding a layer of disk management, potential for data corruption and performance hits.
Crossflashing removes the RAID firmware, replacing it with code that simply handles raw data. The instructions and replacement firmware to flash it are just a web search away.
The procedure can be carried out from the BIOS level of some computers or from inside the old DOS operating system. It’s easy to do but also rather easy to get it wrong. I followed the steps listed here.
Once converted from a sophisticated RAID controller to a plain and simple Host Bus Adapter (HBA), an enterprise-grade board like the LSI 9211-8i is near-perfect for the job of managing the disks of a home/small business server. It has the performance you need and it’s cheap—you can find one on eBay for under £50.
In part 1 we saw that OpenZFS arrays formatted as RAID-Z2 devote the equivalent of two disks to parity data. This means that any two disks in the array can fail before data is lost; fortunately, I’ve never had to put this to the test.
Not just yet, anyway. But the nub of the problem—apart from the fact that while the media library and the backups of the various home devices are getting bigger but the disks aren’t—is that what they are visibly doing is getting older and showing their age.
“Visibly” thanks to SMART.
SMART (or more strictly S.M.A.R.T) stands for Self-monitoring, Analysis and Reporting Technology. It can anticipate and report physical failure, even while the drive is still faithfully delivering and storing your data. There are built-in error correction methods and SMART keeps a count of how often they need to be used. Temperature and vibration also offer clues. SMART stores all these data and draws conclusions about the state of your drive that it can pass it on to you when you ask for it.
SMART gets outsmarted by some drives—it’s thought to provide useful warnings in only two-thirds of the disk drives it’s installed in. But here it was handing me the bad news that these old drives were nearing the end of their lives.
The Origin of a Remarkable FileSystem
Sun Microsystems, generally referred to simply as SUN, dawned in the early 1980s, rose to a glorious noon towards the end of the Millennium, and sank beneath the horizon as the 21st century set in.
Its name memorialises its origin: SUN stands for Stanford University Network, and its founders, in 1982—Scott McNealy, Andy Bechtolsheim, and Vinod Khosla—were all graduates of that University. Bill Joy, another Stanford graduate, joined them that same year.
All four are revered in the 20th-Century Computing Hall of Fame and the work they did together was extraordinary, paving the way for much of the development underpinning our current technology.
The company began by selling graphics workstations running SUN’s own version of Unix called Solaris and immediately became profitable. Its share price soared, encouraging the company to invest unprecedented sums in personnel, hardware and software development. It’s no exaggeration to say that during this period SUN’s innovations quietly but inexorably revolutionised the IT industry.
SUN vanished almost as quickly as it began. Its track record in heavy investment sank it perilously when the dot-com bubble burst in 2002.
It was bought up by the enterprise database mammoth Oracle but not before SUN had released the code for ZFS to the open source community.
Two versions of ZFS are now current: the actively developed OpenZFS which we’re discussing here and Oracle’s relatively moribund proprietary ZFS. From this point on we’ll refer to the technology generically as ZFS.
Sun saw a way of playing this right down the middle. Adopt the new, cheap hardware. Don’t trust it—because by definition it’s untrustworthy. Simply ensure it behaves to enterprise standards by writing software that makes no assumptions about its trustworthiness.
Simply? Well, perhaps not. But the SUN founders understood the intricacies of the Unix operating system inside out. They knew that however complex this “building on sand” software would have to be, anyone using the system, or having to manage and maintain the system would need to be presented with an interface that was simple, logical and easy to grasp.
Out of these ideas, ZFS was born.
The ZFS conundrum
However, one of the disadvantages of ZFS is that you can’t expand an existing pool of drives (the zpool*) simply by adding another disk because the RAID array’s redundancy configuration can’t be changed—there’s more detail about why not here.
The advantage is flexibility; the zpool can then be divided up (or not) to suit.
The disadvantage that ZFS’s zpool can’t easily be expanded is something the OpenZFS team is currently working on. However, this particular part of the OpenZFS project has been ongoing for some years and doesn’t look like reaching production-readiness any time soon.
*NB: (Note added 2024-09-13)
The OpenZFS team has made substantial progress on this front. Online pool expansion was introduced with OpenZFS 2.0 in late 2020. This feature allows users to add new vdevs to an existing pool without having to recreate the pool, addressing the main limitation mentioned in the side-note above. The ability to expand zpools is now in production and widely used, allowing users to add new devices to existing pools, increasing their capacity without downtime or data migration.
- replace every disk in the system one by one, allowing the RAID to rebuild after each addition, or
- create a new zpool with new disks and migrate the data across.
The first option, which in RAID circles you’ll often hear referred to as resilvering—a fanciful derivation from the jargon of “mirroring” drives— is very much in the spirit of how RAID is meant to work. Although you can’t (currently) expand the zpool by installing additional drives*, ZFS does allow you to replace the existing drives with new, larger drives, a feature known as auto-expand. You’d have to do this piecemeal, resilvering to reinstate the data on each new, larger drive, one drive at a time, before moving on to the next one.
But when RAID was first introduced in the early ’80s, the (relatively) low cost drives it was designed to work with measured their capacities in megabytes. With much less data to handle, resilvering a single drive could finish during a longish coffee break. Rebuilding the data on a multi-terabyte drive, as Tested Technology can testify, is a glacial process measured in days.
Replacing the old disks one by one, allowing the zpool to resilver after each disk addition would certainly work. The time constraint—probably well over several days to carry out the whole replacement operation with today’s multi-terabyte drives—might not matter, as the whole idea of RAID is that the NAS remains usable (if noticeably slower) during the resilvering process.
So why not do that?
There’s one other constraint. This is a home server. It lives in the cellar. While situated away from most living spaces, it still makes its presence audibly felt, despite being configured to be as quiet as possible. The same is true of the backup server. They each also consume electricity.
Reducing the total number of disks from eight to three in each server would ameliorate both of these issues. Not only would that lower noise emissions and power consumption, it also makes future capacity expansion much easier.
So the plan I came up with was to forget about resilvering and instead start afresh. Create a new zpool with a smaller array of larger drives. Then I’d have to find some way of copying all the existing data to it.
To this end, I wondered if the storage industry might be interested in joining my adventure. They’ve been very helpful to Tested Technology in the past. So I dropped an email on the publication’s behalf to Seagate.
How this worked out is a story for the next instalment, but let’s just say it involves a 3D printer and a set of new, high-capacity disks.
Part 3 is hereManek Dubash: 24-Jul-20
Because FreeNAS will be disappearing.
In part 1 we touched on iXsystems and the two versions of the operating system it offers: FreeNAS for the Open Source community and TrueNAS, the commercial version it sells, along with its hardware, to enterprise and small and medium businesses.
Although the two versions are closely related, this has involved maintaining two separate sets of source code. Merging these has made sense for a long time, although the logistics of doing so could be intimidating, bearing in mind that this is a product enterprise customers are staking their companies on.
For years we’ve talked longingly about unifying the images, documentation, and web content due to all of the efficiency gains and benefits to be had.
iXsystems
iXsystems has finally bitten the bullet. The cautiously experimental FreeNAS and the experimentally cautious TrueNAS are now one.
The one thing that always held us back was the question of what to do with the name: after all, unification of the software meant that two distinct product names were no longer workable, and we’ve grown as attached to both names as have our Community and customers (and perhaps even more!)
iXsystems
The new, unified NAS operating system is to be called TrueNAS. The company assures the Open Source community that the code will always remain open and free. The only name distinction is that the open version will be called TrueNAS Core and the enterprise version TrueNAS Scale.
The key difference is that TrueNAS Scale will have a few firmware features designed to work exclusively with iXSystems own hardware as well as some built-in telemetry enabling this version to “phone-home” back to iXSystems. This is to help with the aspect of TrueNAS that its enterprise customers primarily value and pay for—ixSystems’ maintenance and support.