Local File System Storage

Overview

The local file system storage implements a disk-based log storage system using memory mapping (mmap) to provide efficient read and write performance. Although the local file system supports append operations, we maintain the Fragment concept to ensure consistency with the object storage implementation.

Core Components

FragmentFile

FragmentFile is the basic storage unit responsible for read and write operations of individual log fragment files.

File Layout:
[Header (4K)] + [Data Area (grows forward)] + [...Free Space...] + [Index Area (grows backward)] + [Footer (fixed size)]

- Header (4K): Stores magic string and version information
- Data Area: Stores actual log entries, growing forward from 4K position
- Free Space: Unused space between data area and index area
- Index Area: Stores offset of each entry, growing backward from file end
- Footer: Stores metadata information

Data Entry Format:
[Payload Size (4 bytes)] + [CRC (4 bytes)] + [Actual Data (variable length)]

DiskLogFile

DiskLogFile manages multiple Fragment files, providing a unified read/write interface.

Directory Structure:
/basePath/
  └── log_[logID]/
      ├── fragment_[startOffset1]
      ├── fragment_[startOffset2]
      └── ...

Key Design Decisions

Use of mmap

Provides zero-copy read/write operations
Supports random access
OS automatically handles page caching
Delivers memory-level read/write performance

Fragment Necessity

Although local file system supports append, maintaining the Fragment concept provides several benefits:

Maintains same architecture as object storage implementation
Facilitates file size limitations
Supports segmented management and cleanup
Simplifies concurrency control
Improves system recovery capability

Space Management

Pre-allocates fixed-size file space
Data area and index area grow from opposite ends
Creates new Fragment when areas meet
Avoids overflow through precise space calculation

Performance Considerations

Write Performance Optimization

Zero-copy writes using mmap
Batch writes to reduce system calls
Asynchronous write support
Pre-allocated space reduces expansion overhead

Read Performance Optimization

Quick positioning through index
Supports random reads
Utilizes OS page cache
Supports range reads

Memory Usage

On-demand loading through mmap
Controls individual Fragment size
Implements memory usage limits
Supports memory mapping release

Reliability Guarantees

Data Integrity

CRC32 checksum for data validation
Magic string verification in file header
Separate index and data areas
Data recovery support

Concurrency Control

Read-write locks for shared resource protection
Supports multiple-reader single-writer mode
Atomic operations for counter guarantees
File locks prevent concurrent writes

Configuration Parameters

Fragment Configuration

Parameter	Description
File Size Limit	Maximum size of a fragment file
Maximum Entries	Maximum number of entries per fragment
Pre-allocated Space	Amount of space to pre-allocate
Memory Mapping Options	Configuration for memory mapping behavior

LogFile Configuration

Parameter	Description
Base Path	Root directory for log files
Fragment Size	Size of individual fragments
Async Write Buffer Size	Size of asynchronous write buffer
Cleanup Policy	Strategy for cleaning up old fragments

Usage Examples

// Create LogFile
logFile, err := NewDiskLogFile(1, "/path/to/logs", WithFragmentSize(4*1024*1024))

// Write data
err = logFile.Append(ctx, []byte("log entry"))

// Async write
entryID, resultCh, err := logFile.AppendAsync(ctx, 0, []byte("async log entry"))

// Read data
reader, err := logFile.NewReader(ctx, storage.ReaderOpt{
    StartSequenceNum: 0,
    EndSequenceNum:   100,
})

Important Notes

Ensure sufficient disk space
Regularly clean up old Fragment files
Monitor memory usage
Handle file system errors
Implement graceful shutdown
Consider system crash recovery