Lost Linux Data Debacle

One of my customers has several employees that use Linux on their laptops. Awesome. Now for the bad news. One of the engineers actually typed rm -rf * in the terminal.
Yup. It happened.

That command will forcefully remove everything from the directory you are working in. Unfortunately for them, it was their home directory. Ouch.
Apparently, the person was attempting to remove all content from a particular directory called gpt4all. The command the person was going for was:

rm -rf ~/gpt4all/*

But somehow, the * was typed before the directory, and not after. Add to that, the person used the -f flag and yikes, it immediately forced the deletion of all data in the home directory. If the -f had been omitted, the system would have at least started asking for confirmation (at some point) and a lot of the data could have been saved simply by pressing Ctrl + C once the user realized what was happening. But, according to the user, it all happened so fast, that before they new what was going on, the data was gone. To their credit, the person immediately (and forcefully) powered down the system by pressing and holding the power button. Good move!

Then I got the call. I think I audibly groaned when I heard what happened—and found out that there was no backup. From what I have heard and read (and tested with), in most cases, the data is not recoverable, or if it is, the data will end up being all jumbled with new filenames that are not descriptive. Normally, if the drive contained sensitive or critical data, I would have simply referred the customer to the data recovery specialist company that I use. But this laptop (and it's data contents) were not deemed worthy of that expense ($$$$). I had honestly never troubleshot this disaster or attempted to repair it before. But that was now my task—and as the manager of the techs said, it was mission impossible.

So here's what I did step-by-step:

1. Brought my trusty USB backup drive and Live USB flash drive. The person used Ubuntu, so I used a Live Ubuntu image to boot off of.
2. Connected both USB drives to the laptop, and booted to the BIOS, being very careful not to boot into the OS. I had to make double-sure what the keystroke was to enter the BIOS beforehand and must have pressed it a hundred times.
3. Accessed the BIOS and changed the BIOS boot order to flash drive first.
4. Booted to the live Ubuntu flash drive.
5. Then I navigated the system to find the drive with the OS and my backup drive. Documented what was what.
6. Started scanning the OS drive. Indeed, the user's home directory was "gone". However, the home directory was originally setup on a different partition than the OS. Good news, no chance of OS corruption or loss. I verified that this was the case, the OS looked good! Also, the main /home directory looked good as well—other than the lack of the user's account directory! But... after several commands I was able to discern that a good deal of the person's data was still there. It had not been re-written yet. The bad news was that the journal was cleared. So that makes things more difficult.
7. Mounted the backup drive. Created a directory structure for recovery.
8. Started using tools to recover data.
   1. I used ext4magic first. After several attempts I stumbled onto a command that worked. This recovered a surprisingly good amount of data: the .config directory, the user's .wine directory, and all the other hidden directories in the user's home directory - tree structure intact. I was also able to recover some data from the Documents directory. This was good, and unexpected. But... that was the end of it. There was still a lot more that ext4magic did not find including video files, images, some PDFs and business documents, and more. It even found gpt4all info, but the user wasn't using that tool anymore. (Which makes sense because they were trying to delete the directory—which caused the whole problem!)
      > Here is one of the commands I used that did the heavy lifting (in generic format). There were several others, but to be honest I did not document the whole process:
      ext4magic /dev/sdXN -M -a -d /path/to/recovery/destination
   2. So I next used PhotoRec. This tool can recover data like photos, .mp3s, .mp4s and more. However, it renames everything into generic numbered filenames. Still, it found all the images that were "lost", all of the PDFs and business documents, and a couple dozen video files. Unfortunately, some of those video files were corrupted and unusable. All in all though, a good haul of data.
      > Here is the command:
      sudo photorec /dev/sdX
      Then, it uses an interactive set of prompts that ask for the correct partition, filesystem, and so on. This took several attempts, but I was able to "recup" a bunch of stuff with that tool.
9. Then we spent some time with various Bash commands (with sed and awk options) to sift through the data and verify what we had. It was time-consuming, and brain-busting, but we got there. The user said that it looked like we had 95% of the data recovered, and that was more than acceptable. (I promptly left a fun sticky note on the manager's desk. Guess what is said?)
10. Next, I removed the user account within /etc/passwd and recreated the account fresh:
    sudo useradd -m -s /bin/bash -u <original_UID> <user>
11. Reset the password and restored standard user folders:
    sudo passwd <user>
sudo -u <user> xdg-user-dirs-update
12. Copied the recovered .config directory and other recovered hidden directories found with ext4magic into the new home. For example:
    sudo cp -a "/media/<user>/My Passport/Deep_Tree_Recovered/<user>/.config" /home/<user>/
sudo chown -R <user>:<user> /home/<user>/.config
13. Copied over the rest of the data found by ext4magic
14. Fixed ownership:
    sudo chown -R <user>:<user> /home/<user>
15. Had the user log in and verify that the configuration was right. There were some minor tweaks to do, but the bulk of the config was there.
16. Copied over all the rest of the data discovered by PhotoRec into a directory in the user's Documents and wished them good luck!

Now, we wanted to make sure that this did not happen again, and if it did, that there would be a recovery method in place. So I took the following steps:

1. Suggested using a GUI-based tool for file management such as Dolphin or Files. This provides for point-and-click surety. If the user wanted to remove the gpt4all directory then a simple click and Del would do it. And if the person was to delete something and immediately realized that it was a mistake, it would be in the Trash (plus there is confirmation required if you use Dolphin or Files). I also mentioned trash-cli and safe-rm as options for safely removing items in the Terminal.
2. Set up an rsync backup script to run automatically.
   1. First, I wrote the script (shown at the end of this article). We decided on the
      --delete option so that the backup would be an exact copy of the home directory. I also suggested using --link-dest to set up snapshot-based backup, but this was left as a future option. I actually wrote it to backup the entire /home directory and all user accounts.
   2. Then, I setup anacron on the system to run the Bash script daily. cron is great on servers because you can set it to run at 2am (I actually schedule at 2:15) and it works great because the servers are always on. However, client Linux systems are often turned off, that's where anacron comes in. It initiates backups when it sees that the system is on and logged into. (That config is also at the end of the post.)
   3. Setup a backup user account (named sa) with sudo powers that I or the user could utilize in the case of a future failure.
3. Documented the process for the user in case they wanted to replicate the rsync process or modify it. I also left other suggestions in there such as --link-dest and Clonezilla for full drive backup.

So that's it. I was able to recover the bulk of the data. Not perfectly, but not bad given the situation, and the fact that we used free, open source, and readily available Linux tools!
Now, like I mentioned, if this drive had critical data, I would have told them to send it to a data recovery specialist that I use—and pay the price.

In the end, all's well that ends well.

rsync script:

#!/bin/bash
sudo rsync -avhP --delete \
  --exclude='*/.wine/' \
  --exclude='*/comfy/' \
  --exclude='**/GPUCache/' \
  --exclude='**/ShaderCache/' \
  --exclude='**/DawnCache/' \
  --exclude='**/Cache/' \
  /home/ /media/<user>/rsync-backup-2026/

I saved this to ~/bin/backup-home.sh and made it executable:

chmod +x ~/bin/backup-home.sh

Note: Keep in mind that rsync is using the --delete option which makes a mirror, and that includes deletions. To keep all data, regardless of what happens to it at the source, omit --delete but be ready for more data!

anacron procedure

I edited anacrontab with sudo vim /etc/anacrontab and added the following line item for daily file copy:

1    5    backup-home    /home/<user>/bin/backup-home.sh >> /home/<user>/backup.log 2>&1