TREEspan File System™ (TSFS) is a high performance embedded transactional file system with native NOR and NAND Flash support, designed for RTOS and bare-metal applications. TSFS provides developers with robust, fail-safe, data storage on virtually any MCU or SoC.
Checking All the Boxes
All-round Media Support. TSFS supports managed flash devices like SD cards and eMMCs as well as raw NOR and NAND flash devices. It does so with a single unified codebase allowing for effortless migration through the whole technology spectrum.
Built-in Flash Management. Flash management tasks like bad block handling, wear-leveling (both dynamic and static), error correction and garbage collection come built into TSFS core.
Fail-Safe. TSFS is designed to withstand any number of untimely interruptions (power failures, physical device removal and others). File system integrity is guaranteed on raw NOR, raw NAND, industrial-grade SD card and eMMC.
Transactional. TSFS is transactional which means that, upon recovering from a prior interruption, application data will always return to a consistent state, as defined by the application designer.
Low RAM Footprint. TSFS provides a high performance file system abstraction with as little as 4KiB on raw NOR flash, 12KiB on raw NAND flash and 8KiB on SD/MMC.
Fast Mount. All TSFS needs to do for mounting a file system (regardless of its size) is reading a few blocks, nothing more! No journal parsing, no clean-up procedure, partial updates are passively discarded as part of the garbage collection.
Real-time. Embedded systems must oftentimes store data at a constant time interval and without loss. TSFS was designed to meet that common requirement while minimizing the amount of buffering needed and maintaining high average throughput.
High Random Throughput. TSFS exhibits near-optimal random write throughput on raw flash memory. On managed flash devices, like SD card, TSFS log structure harnesses the high sequential throughput available at the device level to consistently produce high access speeds at the file level, independent of the access pattern.
Full C99 Source. TSFS follows the most stringent coding practices, including systematic and well-documented error handling, defensive coding style with rigorous argument checking and optional run-time consistency checks.
Pristine Documentation. We pride ourselves in doing something as complex as embedded data storage, as simple as possible, and documentation is a big part of it. Have a look!
Drivers Included. TSFS comes complete with all the drivers for your combination of hardware platform and storage device, so that you can focus on your application from the very start.
Simple Royalty-Free Licensing. The TSFS source code as well as any accompanying drivers and BSPs are delivered under a simple lifetime royalty-free licensing model. Customers can pay for optional yearly maintenance after the first year to receive continued support and updates.
Media Support
SD / MMC
TSFS leverages the fact that SD card and eMMC are really at their best when accessed in large and contiguous sequences. More specifically,
- TSFS produces sequential write accesses at the device level, independent of the write pattern at the file level;
- TSFS stores on-disk metadata in compact structures favouring large sequential read access and supporting efficient lookup operations and a high-read throughput;
- TSFS supports true “blind writes” (where no lookup is needed prior to writing) favouring uninterrupted sequences of write accesses and a higher write throughput.
NOR Flash
Embedded systems with modest data storage requirements often have both code and data on a single shared NOR flash device. This is typically the case in deeply embedded applications where RAM is rare and precious. In such a memory-constrained environment, a file system might not even be needed. Otherwise, it must certainly come with minimal RAM requirements.
With native support for raw flash memory, TSFS does not require block device emulation. No extra adaptation software is needed, avoiding buffer and data structure duplication, and keeping RAM consumption at a minimum. With 4KiB of RAM, TSFS provides a complete file system abstraction while carrying crucial flash management tasks such as garbage collection and wear-levelling.
TSFS also takes advantage of NOR flash support for arbitrarily small accesses with very low overhead. This makes a real difference for read-centric applications seeking to benefit from NOR flash jitter-free access times and near-bus speed read throughput.
SLC NAND Flash
SLC NAND is a great fit for so many embedded applications. It has it all: low cost per bit, low energy consumption, and great overall performances. In its serial, semi-managed form, with built-in ECC, you get much of SLC NAND performance potential with very minimal integration effort. At the other end of the spectrum, the bare form with parallel interface provides the highest performance through pipelining of flash array access and bus transfers, and support for multi-plane operations.
One unsung merit of SLC NAND is its potential for extremely low write latency. TSFS taps into this potential
to provide strict timing guarantees and enable real-time applications on memory-constrained platforms (see Strict Real-Time Behaviour). On parallel NAND with dual-plane operations, for instance, TSFS can produce 5MB/s of sustained random write throughput with a total footprint (TSFS + application-level buffering) of 24KiB.
Performance
High Performance Across the Board
Optimal Random Write Throughput
TSFS garbage collection optimizes write amplification, meaning that a minimal amount of data is collected at any given moment in order to free and recycle partially used data blocks (explained in our article on Flash File System Performance).
A comparison of experimental and optimal (calculated) random write throughput values confirms how close TSFS is from the upper bound (see graph). Minimal write amplification not only means higher average throughput but also lower cell wear (and thus extended lifetime) and lower energy consumption.
Optimal throughput (in green) calculated from experimental throughput values of the Mircron MT29F2G01 QSPI SLC NAND and based on the assumption of perfect garbage collection and the absence of any other form of overhead. Measurements are performed on the STM32F746ZG@200MHz.
Low RAM Usage
TSFS provides a high performance file system abstraction with minimal RAM footprint. All numbers given here include internal buffers. Some additional buffering at the application level might be needed, although TSFS offers strict timing guarantees minimizing the amount of extra buffering required (see Strict Real-Time Behaviour below).
Strict Real-Time Behaviour
Embedded systems commonly require that data be persistently recorded at a constant time interval and without loss. Minimizing write latency is key to meeting such a requirement while preserving a high average throughput and keeping RAM usage under control (see our article on Embedded File System RAM Usage).
Assuming a 1MB/s constant input rate and an SD card with a typical 200ms maximum access time, we need at least 1MB/s * 0.2s = 200KB of RAM to avoid data loss.
By contrast, a real-time flash file system like TSFS achieves high throughput with little to no buffering on raw NAND. On the Micron MT29F1G01 QSPI SLC NAND, TSFS can accommodate a 2MB/s input rate without loss with only 6KiB of buffering RAM (see graph).
Fast Mount
TSFS achieves transactionality using copy-on-write strategies (as opposed to relying on some form of journalling approach). As a result, mount cycling is read-only and extremely fast.
Everything Needed to Get You Going
Source Code
TSFS is distributed in C99 compliant source form, including drivers and support modules. All provided sources follow or exceed industry standard coding practices, including
- Systematic error handling. Error codes are either handled in-place or returned intact to the caller. Error handling always includes appropriate clean-up code (mostly for releasing resources) such that both the file system and the surrounding runtime environment can resume in the event of a timeout, IO error or other non-fatal errors.
- Defensive coding style with abundant checks and assertions including validation of function arguments, validation of external data (e.g., on-disk data structures), assertions of various expected internal states or conditions, and various bounds checking.
- MISRA compliance. With supporting documentation including a detailed compliance matrix and corresponding rationale for possible deviations.
Example Application and Validation Test Suite
TSFS comes with a fully functional and thoroughly documented example application on the platform of your choice, making for an effortless startup experience. It also comes with a full validation suite, including file system and device driver benchmarks and functional tests, to make sure that you do get the most out of TSFS.
Inline Documentation
Each API is documented in a clear and uncluttered way, complete with a detailed behaviour description arguments description, and a list of possible error codes that the caller must handle. Provided drivers and examples are documented with the same level of preciseness and clarity.
User Manual
The TSFS User Manual includes general background information and theory about file systems and flash memory technologies to help you make the best possible design choices even though flash memory might not be your main area of expertise. The TSFS API Reference is clear, precise and easy to navigate with coloured margin tags and crosslinks to related APIs. Take a look at the documentation section.
Fail-Safety and Transactions
Transactional Is More Than Fail-Safe
Fail-safe means that TSFS will gracefully recover from any form of untimely interruption (power failure, device removal, or else). This is different from transactional, which adds on top of fail-safety, guaranteeing that all data stored in the file system will be returned to its latest consistent state, as specified by the application (see How it Works below). Having transactional logic baked into the file system core takes the difficult and critical task of implementing and testing one-off recovery logic off of the application designer’s plate.
How it Works
As far as coding goes, TSFS is very much like any other file system, the most notable addition being the tsfs_commit() interface. Any modification to the file system (file write, file deletion, directory creation, etc.) is not effective as long as tsfs_commit() does not successfully complete. Modifications performed after committing are discarded upon recovery. Crucially, committing is atomic, meaning that an interruption can either occur before or after, but not “during” a commit.
This all-or-nothing behaviour means that interruption timing with regard to the ongoing transaction has no impact on the final outcome (see figure).
All that’s left to the application designer is to judiciously define what goes into a transaction such that it makes sense for the application and cuts possible recovery paths down to just a few, easy-to-handle cases.
A Simple Example
Say a new frame of samples is appended to a file at a regular interval. Upon reaching a certain size threshold, the file is archived and a separate archive index file is updated accordingly. A robust solution using TSFS is shown below. If a power failure were to occur anywhere between Commit 0 and Commit 1, all the files would be returned to whatever they were at the time of Commit 0.
Notice how the content of the transaction is purposely defined such that, upon recovery, it would be impossible to find a partial frame, for instance, or a full data file for which no archive was created, or an archive file without the corresponding entry in the archive index file. Ruling out all these partial update scenarios, there is not much left to do for the application in terms of recovery, as the application will always restart from a coherent state, independent of the failure timing.
Flash Management for raw NOR / NAND
Garbage Collection
The need for garbage collection stems from fundamental fash memory characteristics and restrictions. In a nutshell, flash memory is divided into large erase blocks, each of which in turn divided into pages. One requirement is that a whole erase block must be erased before a page can be written.
Before it can be erased, a used block must first be freed, which means that any remaining bits of valid data must be moved (copied) to some alternate location (see figure).
This process, known as garbage collection, is built into TSFS core (as opposed to a separate FTL), which significantly reduces related RAM and I/O overhead. In fact, TSFS garbage collection is optimal (see our article on flash file system performance).
Static and Dynamic Wear-Levelling
TSFS implements both dynamic and static wear-levelling. Dynamic wear-levelling is a straightforward extension of garbage collection as it operates only on free blocks. Static wear-levelling does a little bit more, actively freeing long-standing data blocks that would likely never be collected otherwise. In both cases, the goal is the same: ensuring even distribution of cell wear such as to avoid premature block failures.
Bad Block Management (NAND Flash)
Wear-levelling goes a long way towards avoiding premature block failures. Regardless, flash blocks will inevitably wear out at some point. When a block fails (that is, when a program or erase operation fails) TSFS moves its content (from previous successful page programing) to a spare block reserved for this purpose and remaps further accesses accordingly.
Error Correction (NAND Flash)
NAND flash is characterized by some bit error rate which must be compensated for by appropriate error correction algorithms. TSFS provides support for software ECC implementations, on-chip ECC such as those commonly found on serial NAND devices as well as host-side hardware ECC controllers such as NXP BCH controller and Texas Instrument Error Location Module (EML).
Further Reading
NOR vs NAND: So You Think You Know the Music?
In-depth comparison between NOR and NAND covering aspects of NOR and NAND flash technologies that, in our view, are too often ignored including the impact of the application requirements on the choice of Flash technology.
Managed vs Unmanaged: The Many Roads to Flash Storage
Let’s explore and compare two different paradigms of flash management commonly used throughout the industry: managed flash and unmanaged flash. Managed flash devices include SD cards, USB flash drives, eMMC and UFS modules — also SSDs, but those are less often seen in embedded systems. These are all NAND-based devices.
Taming the Flash Beast
This article is the first of an introduction series about flash memory with a focus on embedded systems designs using an embedded file system. A high-level introduction shall we say. Not the kind that takes you straight to the electron