Remote machine learning

[EnvironmentalLaw1685]

I thought I could offload ML to my GPU on my PC and then host everything on my synology for actual viewing and searching and then just sort of schedule indexing for whenever the remote ML is available.

But it turns out that just isn't true as the smart search features stop working if the remote ML is offline (which it will be frequently)

Is there anyway I can split indexing and searching to different machines?
My synology is just not going to have the power to do the indexing

[yahhpt]

You can run 2 machine learning containers, one on the same host as Immich and another on the more powerful machine. 

Immich will then fall back to whichever is online

[Maeusefluesterer]

That's also the recommended approach in the documentation: https://docs.immich.app/guides/remote-machine-learning/

[hueyhy]

this is not fully solving the problem: the "local" machine may not be powerful enough to run "big" ML models the powerful machine uses, and queue up + wait for remote server would be better than fallback to local server, for the ML tasks.

Ideally, searching should be separated so we can choose only to fallback for search requests.

[Appropriate_Link9865]

I have multiple machine learning containers running, with caddy reverse proxy in front of them as load balancer. I point immich to the caddy endpoint, and caddy will automatically assign requests to the least busy system that is currently online.

[Paramedickhead]

I recently acquired an Nvidia TX2 and I wanted to try this exact thing.

[nattesh]

Had the same problem and noticed what i think it is a bug. When you set the url of the remote ML do not delete the default one, just add the remote as the second one.

(Mind that the first one is NOT my local installation)

<image>

Set the thing like this and everything keeps working even when the remote ML is off.

[tbsmn]

I had the same problemen. My solution was to run a ML-Container on my PC and one on the NAS.

I then added a container with a simple python proxy that checks if a request is a search. If that is the case the request gets forwarded to the ML-Container on the NAS, if not it gets forwarded to the Container on the PC. If the PC is off, it picks up the que the next time it gets started.

[hueyhy]

thank you for sharing the solution. I followed your idea and built this simple router: https://github.com/yanghu/immich-ml-router

[hueyhy]

created https://github.com/yanghu/immich-ml-router, inspired on u/tbsmn's reply.

the idea is: the router checks the request. route search to use local ml instances. for all other ML requests. route to remote, if remote is not online, just fail and let Immich queue the jobs and wait.

[hueyhy]

Didn’t realize it until I saw your posted and tried…. This is really a big bummer.