Data Archiving Architecture: Finding a Perfect Balance of Storage Mediums.

Data archiving has always been a challenge both for the enterprises and manufacturers of data storage solutions. The basic laws of thermodynamics tend to have the effect of deteriorating data storage capacity in the long run, so it is important to understand the needs of each individual data archiving project to be able to choose the right storage system based on the appropriate technology under the proper environmental conditions. That should be also combined with relevant migration and replication practices to improve the safety and accessibility for the extended periods of time.

The most common means of data archiving today are flash memory, hard disk drives, magnetic tape and optical discs. Data storage architects usually use one of these technologies, or their combinations, when designing their systems.

Let’s try and take a closer look at these technologies.

First consider flash memory in archiving. At the 2013 Flash Memory Summit Jason Taylor from Facebook, in a keynote speech, presented the idea of using really low endurance flash memory for a cold storage archive. According to Marty Czekalski of Seagate at the MSST conference, flash writing is best done at elevated temperatures while data retention and data disturb favor storage at lower temperatures. The JEDEC JESD218A endurance specification states that if flash power-off temperature is at 25 degrees C then retention is 101 weeks—that isn’t quite 2 years. So it appears conventional flash memory may not have good media archive life and should only be used for storing transitory data.

Hard disk drives are often used in active archives because the various hard disk drive arrays can be continually connected to the storage network, allowing relatively rapid access to content. Hard disk drive active archives can also be combined with flash memory to provide better overall system performance. However hard disk drives do not last forever – they can wear out with continued use and even if the power is turned off the data in the hard disk drive will eventually decay due to thermal erasure (again we run into the enemy of data retention, thermodynamics).

In practice, hard disk drive arrays have built in redundancy and data scrubbing to help retain data for a long period. It is probably good advice to assume that HDDs in an active archive will last only 3-5 years and will need to be replaced over time.

Less active archives where data is stored for longer periods of time will be interested in storage media that can retain the information stored on them for an extended period of time. There are two common digital storage media that are used for long term cold storage applications – magnetic tape and optical discs.

Let’s look at these two storage technologies and compare them for long term cold storage applications.

Magnetic tapes used for archiving come in half-inch tape cartridges. The popular formats used to day are the LTO format supported by the Ultrium LTO Program, the T10000 series tapes from Oracle/ StorageTek and the TS series enterprise tapes from IBM. Modern magnetic tapes have a storage life under low temperature/humidity storage conditions and low usage of several decades and currently native storage capacities per cartridges as high as 8.5 TB.

When not actively being written or read, magnetic tape cartridges can sit in a library system consuming no power. Digital magnetic tape is thus a good candidate for long-term data retention and has a long history of use in many industries for this application.

Optical storage has also been used for long-term data retention and environmental stress tests indicate that the latest generation of optical media should have an expected life-time of at least several decades. FalconMedia Century Archival discs are actually able to store data for hundreds of years, thanks to special gold and platinum reflective layers, that are used in their construction. At the Open Compute Project Summit in January 2014 Facebook, presented a 1 PB optical disc storage system prototype with 10,000 discs. When Facebook started the actual exploitation of the system it actually reduced company’s storage costs by 50% and the energy consumption by 80% compared to their previous HDD-based cold storage system.

Ken Wood from Hitachi Data Systems at the MSST Conference presented research results that proved hypothesis that the migration/remastering costs for 5 PB of content over 75 years is much less for an optical system with the media replaced every 50 years rather than more frequent tape and HDD replacement.

A lot of digital data has persistent value and so long term retention of that data is very important. In an Oracle talk at MSST they estimated that storage for archiving and retention is currently a $3B market, growing to over $7B by 2017. Several storage technologies can play a role in an archive system depending upon the level of activity expected in the archive. Flash memory can provide caching of frequently used or anticipated content to speed retrieval times while HDDs are often used for data that is relatively frequently accessed.

Magnetic tape and optical disks provide low cost, long-term inactive storage with additional latency for data access vs. HDDs due to the time to mount the media in a drive. Thus depending upon the access requirements for an archive it may be most effective to combine two or even three technologies to get the right balance of performance and storage costs. As the total content that we keep increases, these considerations will become more important to drive new generations of storage technologies geared toward protecting valuable content and bringing it to the future.