Hello,
I’ve been reading that sudden power loss on a Raspberry Pi may corrupt SD card filesystems so I was wondering if you have any ingenious software solutions to this problem.
Scenario:
I'm running a custom video game that writes hi scores and metrics (small files) after every play, so mostly read-only unless when patching/configuring. The hi scores are also read after boot.
If I understand correctly how SD cards work, they are written internally in blocks/sectors and a bad write can take down not only the written file’s data but adjacent files too, worst case being corrupting a chunk of the filesystem table if it was writing metadata.
Possible solutions:
I've been reading the official Making a More Resilient File System document.
For minimizing writes I think the document is a bit misleading with some suggestions, for example the suggestion of using tmpfs which uses virtual memory and will write to the swap file by default. Wouldn't that put in danger other data if they internally reside adjacent to that SD card sector (thinking raw SD writes)? In this case padding the partitions may help those edge cases, possibly. But it would still keep degrading the SD card by thrashing writing cycles.
- I can use tmpfs’ “noswap” parameter, but with the suggestion to make /var/log tmpfs 16MB sized, what would happen if this fills up by running 24/7? I’m not sure.
A battery powered UPS-like solution and no-power GPIO detection could be a nice hardware solution, but that would add extra costs and I'm co-maintaining a system that has the hardware is on a whole different (low income) country so I'm trying to minimize need for extra electronics, installation and maintenance costs due to extra parts. So I'd appreciate more a minimalist software-only solution.
So…
Redundancy:
I was planning on using something like a self-healing RAID1 mirroring, even if I have to do it manually.
I’ve read Btrfs and ZFS may do it right off the bat. But that would require an extra different storage (like a USB drive). Still, using another partition of any type risks writing at the same time, and both getting corrupted on power failure. Also I read btrfs has a bad reputation of instability.
Deferring writes is the alternative. I’m not really sure how to set that up via filesystem config magic. But manually I would make a pair of partitions, writing to one, cloning to the other via rsync or per-file watching (inotifywait) deferred some seconds (sleep) and some debouncing to give more chance of avoiding corruption in one of the two partitions on failure (even if that means only an older version is rescued)
Corruption detection and healing:
Corruption detection could be done via hashing, and healing could happen on boot via hashing.
Btrfs and ZFS can scrub instead of me having to handle hashing manually (sha1sum, etc).
On the other hand if I’ll be doing the “CRC book keeping” manually, I may as well make the partitions FAT so I can easily extract/update that data from the SD with Linux, Windows or Mac, which is really convenient. Being plug and play practically doesn’t require any technical knowledge.
Although doing things manually seems like a ton of work.
Conclusion:
I have some ideas from where to start but this being a common issue I was wondering which creative solutions people came up with.
Thank you for reading.
I’ve been reading that sudden power loss on a Raspberry Pi may corrupt SD card filesystems so I was wondering if you have any ingenious software solutions to this problem.
Scenario:
I'm running a custom video game that writes hi scores and metrics (small files) after every play, so mostly read-only unless when patching/configuring. The hi scores are also read after boot.
If I understand correctly how SD cards work, they are written internally in blocks/sectors and a bad write can take down not only the written file’s data but adjacent files too, worst case being corrupting a chunk of the filesystem table if it was writing metadata.
Possible solutions:
I've been reading the official Making a More Resilient File System document.
For minimizing writes I think the document is a bit misleading with some suggestions, for example the suggestion of using tmpfs which uses virtual memory and will write to the swap file by default. Wouldn't that put in danger other data if they internally reside adjacent to that SD card sector (thinking raw SD writes)? In this case padding the partitions may help those edge cases, possibly. But it would still keep degrading the SD card by thrashing writing cycles.
- I can use tmpfs’ “noswap” parameter, but with the suggestion to make /var/log tmpfs 16MB sized, what would happen if this fills up by running 24/7? I’m not sure.
A battery powered UPS-like solution and no-power GPIO detection could be a nice hardware solution, but that would add extra costs and I'm co-maintaining a system that has the hardware is on a whole different (low income) country so I'm trying to minimize need for extra electronics, installation and maintenance costs due to extra parts. So I'd appreciate more a minimalist software-only solution.
So…
Redundancy:
I was planning on using something like a self-healing RAID1 mirroring, even if I have to do it manually.
I’ve read Btrfs and ZFS may do it right off the bat. But that would require an extra different storage (like a USB drive). Still, using another partition of any type risks writing at the same time, and both getting corrupted on power failure. Also I read btrfs has a bad reputation of instability.
Deferring writes is the alternative. I’m not really sure how to set that up via filesystem config magic. But manually I would make a pair of partitions, writing to one, cloning to the other via rsync or per-file watching (inotifywait) deferred some seconds (sleep) and some debouncing to give more chance of avoiding corruption in one of the two partitions on failure (even if that means only an older version is rescued)
Corruption detection and healing:
Corruption detection could be done via hashing, and healing could happen on boot via hashing.
Btrfs and ZFS can scrub instead of me having to handle hashing manually (sha1sum, etc).
On the other hand if I’ll be doing the “CRC book keeping” manually, I may as well make the partitions FAT so I can easily extract/update that data from the SD with Linux, Windows or Mac, which is really convenient. Being plug and play practically doesn’t require any technical knowledge.
Although doing things manually seems like a ton of work.
Conclusion:
I have some ideas from where to start but this being a common issue I was wondering which creative solutions people came up with.
Thank you for reading.
Statistics: Posted by 2bam — Thu Feb 06, 2025 10:37 am