[Feature] Auto restore database

[marsara9]

The feature

One major issue I'm seeing with the current implementation is that should the server crash to where the database is not recoverable, rebuilding the server puts everything in an 'awkward' state.

Example: Currently, I'm running the server itself on a Raspberry Pi. This includes all of the docker containers. One exception is that my upload location is stored on a NAS, so that the important part, the photos, are on a RAID and where I already have weekly off-site backups configured. I also have daily backups of all of my Pis configured as well, but if the server crashes before the database, etc.. can be backed up I end up with several orphaned photos.

Thus would it be possible, to have a job run on boot, that checks each user's library folder and if it finds any untracked photos, that it can automatically move them back to the upload directory to be processed again?

As it stands though, unless I'm missing something, it seems best to try and backup all of the docker volumes after EVERY photo is uploaded and processed.

Maybe there is already an option in the repair section on the website, but I wasn't able to get anything there to work.

Some alternatives that I've tried but none seem ideal:

• Just delete the photo off of the NAS and re-upload it from my phone. (This only work if I still have a copy of it on my phone)
• Move each affected photo back to my desktop and then use the CLI to re-upload it. (This requires that I have an API key for every affected user)
• Manually add entries to the database. (This is rather error prone not to mention time consuming).

Platform

• Server
• Web
• Mobile

[aisbergde]

Alternatively you could use external libraries, which can be easy rescaned. There is also a scheduled task to rescan them.

There are some requests to add some of the maintainance features also to the default library.

Another savety feature is an automated database update, as described in the docs.

[marsara9]

While that would ultimately work, it's still a maintenance nightmare. Say I have 10k photos when the server crashes and only 10 got uploaded since the last backup. I'd still need to find those 10 and manually move them to another location on my NAS. Then the question becomes what happens on their phone when it goes to sync? (Mainly I haven't tested this yet) But would their phone try and reupload those photos again? Or would the server be smart enough to see that a dupe exists in an external library?

My main ask is to have a way to restore the server back to the state it was in prior to the crash with relatively little effort from the admin.

When I first had a crash, it was actually not long after setting up the server for the first time. The downside was, was that I had just transferred all of my photos from Google as well as my wife and kids photos from their accounts as well. Since I didn't have any backups of the database at all at that point I effectively had to:

1. recreate all of the users again
2. hijack each account to create an API key (technically a security breach imo)
3. run the CLI tool on my desktop to reupload those photos back to their respective accounts
4. log back into the NAS and delete the now duplicate/ orphaned library folders that were from the original upload.

This ended up taking the better part of the day just getting things back where they belong. Now though that the server has been running for several days, I at least have backups of the database and at most will lose 24hrs worth of data. But should it crash again, it would still require me to hijack each account to create API keys again to fix the untracked photos. (Unless I go with an external library but I'm unsure what if any other side effects that may surface at this time).

Since the database is where a lot of this information is stored, it has to remain in sync with the stored photos at all times. Thus, ideally in my opinion, there should be a way to rebuild the missing parts of the database from a single source of truth, essentially the actual photos.

[bo0tzz]

have a job run on boot, that checks each user's library folder and if it finds any untracked photos, that it can automatically move them back to the upload directory to be processed again

Doing this sort of thing automatically is bound to hide issues that need to be addressed by the admin. The intent is indeed for the repair page to be used for these sorts of things, that just hasn't been fully implemented yet.

[marsara9]

Fair enough. But even if it's a job that can be kicked off from the repair page, it's better than the options today (in my opinion).

My main ask is just that if the server goes down, there should be a way to effectively restore the database from the already existing photos. Granted things like face recognition names may be lost but having users re-enter that is trivial compared to the amount of effort required to restore entire photos.

[aisbergde]

I think, immich should care about photos, and backup solutions should care about backups, snapshots and so on.
That's why I have a NAS, because I want to be able to restore previous states, if needed.
I use versioning with snapshots, additional incremental backup on an external drive, synchronization with OneDrive, and I will add an addition remote for even more safety. Because I know, how important it could be, to get access to previous states.

The same with photos. New photos are on my smartphone, and they sync to my NAS. One more copy in "default quality" is on Google Photos, another on Amazon. Maximum I can lose assets for one day, if I assets upload happens only when I am in Wifi. This happened once, when I went with my smartphone into water and could not repair.

And it is always possible to save assets a second time and then to remove duplicates. Duplicate detection is well possible using different software. I don't need to manually search 10 k photos to find detect duplicates.

But if someone will implement and maintain additions backup solutions into immich...

[AngelaDMerkel]

I think you're correct, but Immich is somewhat more than a Photo utility. It is also a database and catalogue too. While some people have the technical skills and resources to have a very complete NAS with robust versioning, I think most people won't and will rely on Immich as their only backup solution. People use Google Photos this way too.

Having some ability to repair or restore the database would be useful and is present in other similar tools.

[aisbergde]

I agree, it is important to be able to backup and restore databases. But after restoring the database there will be a discrepancy with libraries.

The main issue with the default immich library are the very limited possibilities to maintain (rescan and so on). I think, it will be very hard to bring a restored database and the default library in sync. The current concept of the default library is not sufficient for backup and restore.

That's why I desided not to trust the default libraries and to use external libraries. External libraries can be rescanned, default libraries cannot. I add new assets only to external libraries, until I think, default libraries are safe enough to handle restoring a database and to sync.

The concept of the external libraries is so much better. I only miss a way to mark assets to be deleted and to edit some metadata (geo data, rotation).

[mmomjian]

Duplicate of #5441 , we won't be automatically dropping/fixing the database as this is a sign of serious issues with the system

[marsara9]

My hope was that I could create a fresh instance of immich, via docker, and point it to an already curated / sorted set of images and it could at least reassign those back to a given user. Sure I might have to rerun the machine learning or re-add people/faces, etc ... But at least the images would still be assigned to the correct users.

But since there're essentially two databases here: one with the postgresDB and another with the images themselves there's always a risk of the two becoming out of sync.

For example, if I lose power and if I didn't have a UPS, the images might get added to their correct directories but the database might end up corrupted as it might lose power during a write. Now I have a backup of the database taken every night but that means I still have to worry about the photos taken that day.

Other services like the *arr stack are able to re-scan a folder for example and rebuild much of its data without much loss. Home Assistant's data is also self contained.

One possible solution, in my opinion: what if immich didn't have an actual database but instead stored all of the metadata in the images themselves or even just in json/config files in the same directory.

But I essentially fear the day I have to do disaster recovery on my Immich instance.

[mmomjian]

When the admin repair page returns, you can easily find the few untracked files in case of a < 1 day out of sync like this. Restoring Immich using the builtin database backups is rather trivial. I have done it and I encourage you to try it.

If you are really paranoid like me, you can setup you own Postgres instance and do streaming offsite database backups using something like PG Backrest.

A large portion of the fancy features immich has (CLIP, map, facial recognition) depend on a real database. It's nothing like an *arr app, so removing the database is a non starter.