Atomic File Save Patterns: temp files, write barriers, and avoiding torn writes

filesiodurabilitycrash-consistencychecksumsczigrust

This article covers Atomic File Save Patterns: temp files, write barriers, and avoiding torn writes. Practical patterns for safely updating configuration and state files. Covers temp+rename, write-ahead logs, checksums, and length-prefix...

Many programs need to update a small state file:

  • configs
  • caches
  • “last processed offset”
  • user preferences

The naive implementation:

  • open file
  • truncate
  • write new content

…can produce torn writes or empty files after a crash.

This post collects robust patterns.

1) Pattern A: temp file + fsync + rename

This is the default for small files.

Pros:

  • simple
  • file always valid (old or new)

Cons:

  • rewrite whole file

(See previous post for the full fsync details.)

Common gotchas:

  • directory durability: after rename, you may need to fsync the directory to make the name durable
  • permissions/ownership: writing a temp file may not preserve mode bits unless you set them explicitly
  • cross-filesystem renames: rename is only atomic within the same filesystem

Related deep dive:

2) Pattern B: append-only + checkpoint (mini WAL)

Instead of rewriting, you append records:

  • each record includes length + checksum
  • on startup, scan and take last valid record

Pros:

  • efficient incremental updates
  • naturally crash-resistant

Cons:

  • file grows; need compaction

C: append a record with length + CRC32

#include <stdint.h>
#include <unistd.h>

typedef struct {
    uint32_t len;
    uint32_t crc;
} rec_hdr_t;

// placeholder crc (use a real CRC32 library in practice)
static uint32_t fake_crc32(const unsigned char *p, uint32_t n) {
    uint32_t x = 0;
    for (uint32_t i = 0; i < n; i++) x = (x * 33) ^ p[i];
    return x;
}

int append_record(int fd, const void *data, uint32_t len) {
    rec_hdr_t h;
    h.len = len;
    h.crc = fake_crc32((const unsigned char*)data, len);

    if (write(fd, &h, sizeof(h)) != (ssize_t)sizeof(h)) return -1;
    if (write(fd, data, len) != (ssize_t)len) return -1;
    return 0;
}

Reader logic:

  • read header
  • read len bytes
  • verify checksum
  • stop at first invalid record

3) Pattern C: double-buffered file (A/B slots)

Have two regions or two files:

  • state.A
  • state.B

Write to the inactive one, fsync, then update a small pointer file.

Pros:

  • bounded storage

Cons:

  • more moving parts

Practical tip: include a monotonically increasing generation number in each slot so readers can pick the newest valid slot.

4) Zig: length-prefix encoding for robust parsing

const std = @import("std");

pub fn writeRecord(w: anytype, payload: []const u8) !void {
    var len_buf: [4]u8 = undefined;
    std.mem.writeInt(u32, &len_buf, @intCast(payload.len), .little);
    try w.writeAll(&len_buf);
    try w.writeAll(payload);
}

pub fn readRecord(r: anytype, a: std.mem.Allocator) ![]u8 {
    var len_buf: [4]u8 = undefined;
    _ = try r.readAll(&len_buf);
    const len = std.mem.readInt(u32, &len_buf, .little);
    var payload = try a.alloc(u8, len);
    errdefer a.free(payload);
    _ = try r.readAll(payload);
    return payload;
}

Combine this with checksums and append-only logs for resilience.

5) Rust: checksum + last valid record

use std::io::{self, Read};

fn read_last_valid(mut bytes: &[u8]) -> io::Result<Option<Vec<u8>>> {
    let mut last: Option<Vec<u8>> = None;
    loop {
        let mut lenb = [0u8; 4];
        if bytes.is_empty() { break; }
        if bytes.len() < 4 { break; }
        lenb.copy_from_slice(&bytes[..4]);
        bytes = &bytes[4..];
        let len = u32::from_le_bytes(lenb) as usize;
        if bytes.len() < len { break; }
        let payload = bytes[..len].to_vec();
        bytes = &bytes[len..];
        last = Some(payload);
    }
    Ok(last)
}

6) Picking the right pattern

  • Small config file: temp+rename
  • High update frequency: append-only + checksum + compaction
  • Fixed storage budget: double-buffered

7) A note on atomicity vs durability

People often mix these up:

  • atomicity: after a crash, you see either the old state or the new state (not a mix)
  • durability: after a crash, the new state is actually present

rename helps with atomicity.

fsync/fdatasync is what gives you durability.

If you skip the durability step, you might get an "atomic" update that still disappears after power loss.

8) Optional: O_TMPFILE and safer temp handling (Linux)

Linux supports O_TMPFILE on some filesystems. It creates an unnamed inode that only becomes visible when you link it into the directory.

Why it helps:

  • avoids leaving stray temp files on crash
  • avoids name collisions

Tradeoffs:

  • not portable
  • not supported everywhere

If you liked these patterns, two closely related building blocks are:

References