Agenda

• The journey of softraid
• Overview of RAID
• In-depth look at softraid

Why oh why?

• Why write a new RAID subsystem when there are others?
  1. I did RAID for a living
  2. Having written my own version gave me deeper insight into a RAID stack
  3. Unimpressed with the other implementations
  4. Even more unimpressed with the portability of the other implementations
  5. Trying to understand why certain RAID implementations are in the millions lines of code

And who are you?

• I work for a large computer manufacturer as a Sr. Architect for the Intelligent Data Management (IDM) group
• At the time I wrote the initial code I worked on external SAS & iSCSI RAID subsystems
• Before that I architected and wrote RAID, SCSI & Storage management code
• And before that I worked on RAID HBAs and was the SCSI-purist who hounded down and punished vendors who violated the SCSI spec
• This list goes on for a while...

How did this all get started?

• Vendors were hiding under the guise of complexity
• I had this burning itch to prove that RAID implementations were needlessly complex and that the complexity facade was the result of years of mismanagement of developer resources
• After working for years on various RAID and RAID related products I needed a tool to demonstrate that complexity was the enemy of the solution
• The proof-of-concept was a couple of hundreds of lines of code

Hardware RAID

• Most hardware RAID controllers don't speak SCSI at the driver level thus the driver is essentially a proxy between the SCSI midlayer and the HBA API
  • This puts the onus on the driver writer to make a non-generic piece of hardware play nice inside a general purpose OS
• All RAID controllers hide the real disks from the operating system
  • I/O transformations happen on the RAID controller
  • The driver makes the RAID volume look just like a SCSI disk

ami0 at pci1 dev 8 function 0 "Symbios Logic MegaRAID" rev 0x01: apic 9 int 8 (irq 10)
ami0: Dell PERC4/DC, 64b/lhc, FW 351X, BIOS v1.10, 128MB RAM
ami0: 2 channels, 0 FC loops, 1 logical drives
scsibus2 at ami0: 40 targets
sd0 at scsibus2 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 346850MB, 512 bytes/sec, 710348800 sec total
scsibus3 at ami0: 16 targets
scsibus4 at ami0: 16 targets
safte0 at scsibus4 targ 6 lun 0: <SUPER, GEM318, 0> SCSI2 3/processor fixed

Virtual HBA

• The softraid idea was modeled after a hardware RAID controller
  • Being like any other RAID controller it had to pretend to be a SCSI HBA
  • Provide sensors information via sysctl
  • Provide management capabilities via bioctl
• Why stop at RAID?
  • It is generic enough to be a scaffold for just about anything SCSI
• Work on all architectures while being as MI as possible

softraid0 at root
scsibus3 at softraid0: 1 targets
sd2 at scsibus3 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
sd2: 3773MB, 512 bytes/sec, 7727123 sec total

What about the others?

• RAIDframe
  • Doesn't work on all platforms
  • Complex research code
  • Hard to use
• ccd
  • Not user friendly
  • Error prone
• Both don't use OpenBSD's management paradigm
• Not actively maintained

Portability

• softraid uses partitions (chunks in softraid lingo) of the type RAID
  • This simplified development because the underlying layers were already in place and no additional development was required
• softraid works on all supported architectures and the code is 100% MI
  • Booting and rooting will be the only MD code

Partitions

• Even though softraid behaves like a hardware RAID controller it operates on partitions instead of full disks
• During boot softraid scans all partitions on sd and wd for the RAID type
  • When a RAID partition is found the metadata is retrieved
  • If the threshold for assembly is met the partitions are joined into a volume that is exported to the operating system
• This model has some interesting side effects.
  • Staggering RAID sets
  • External tools have access to the partitions
  • Softraid is a good kernel citizen because it plays by the rules

Metadata

• Metadata describes the entire volume and identifies self
  • Header -> volume -> chunks -> optional members
• During a hackathon a long debate erupted about something seemingly trivial
  • Where do we stuff the metadata? in the front or in the back?
• Both scenarios have their pluses and minuses

Metadata continued

• Vendor independent
  • Strategically located hooks within the metadata handling code to translate between softraid and foreign metadata formats
• Why is metadata so hard?
• Versioning metadata to prevent bad things from happening
• Variant and invariant parts

Volumes

• Volumes are the virtual disks that are exported to the Operating System
• Volumes are assembled from N number of chunks
• Volumes are managed through the same tools as hardware RAID
  • bioctl
  • sensors

Chunks

• Chunks are synonymous with partitions
• Metadata is offset 8KB from the start of the partition to prevent interactions
• Externally visible

Disciplines

• RAID 0
• RAID 1
• RAID P
• RAID 6
• Crypto
• AOE
• Each discipline owns exactly one volume

Frontend I/O

• An operating system I/O flows from the SCSI midlayer into softraid
• The I/O is redirected to the correct discipline
• The discipline I/O transformation routine is called

Work Units

• A frontend I/O is wrapped in a Work Unit (WU)
• The individual I/Os (CCBs) that make up the WU are sent into the block I/O layer
• When all individual I/Os complete the WU unit completes and that in turn completes the frontend I/O
• If I/Os collide, overlapping LBA range, then the colliding I/O is scheduled behind the frontend I/O and is restarted in the interrupt handler

CCBs

• Each I/O is wrapped in a CCB
• The first field in the CCB is a struct buf
  • This is to be able to get interrupted when the struct buf completes
• Uses vnode API

Future

• Booting + Rooting softraid
  • Metadata has the mini-discipline code as optional metadata payload
  • Will be machine dependent (MD) code
• New disciplines
  • iSCSI target/initiator?
  • Concat
  • FCoE
  • Cache
• Scrub
• Mount by ID
  • not a softraid feature but one of the main drivers

Principal Developers

• Marco Peerebom
• Joel Sing
• Ted Unangst
• Hans-Joerg Hoexer
• Damien Miller
• Jordan Hargrave

Thanks!

• Asiabsdcon for inviting me.
• Mark T. Uemura for his help getting me here!
• The friendly folks at the OpenBSD project for letting me play
• My lovely wife for putting up with all this!

Questions

Fire away!