published 04/11-26 by freddo
October last year the CephFS filesystem storing user data including homedirs, mail, and websites experienced a catastrophic failure. We were able to perform a recovery and extract data into a newly formed filesystem, with no apparent dataloss.
We've kept the old broken filesystem around since then in case we've missed something or something got corrupted during recovery. Seeing as we're starting to run low on storage, and nobody has reported any data corruption or loss, we're going to be reducing the data durability of the old filesystem from 3x to 2x replication/. This reduces it's storage footprint by 33%.
If nobody reports any data loss or corruption, we'll likely in the future delete it entirely. We recommend all users to check their files from before 2024 for damages and report it to us if any is found.