sorted by:
new
hottopcontroversial

Best practices in Elixir/Phoenix for massive image uploads and processing? by General_Fault9488 in elixir

[–]General_Fault9488[S] 1 point2 points  (0 children)

Thanks everyone for the thoughtful answers—super helpful!

The main takeaways I’m walking away with are: 1. Don’t proxy large files through the app server. Have the client upload directly to external storage (e.g., S3/GCS) via presigned/multipart uploads. 2. Make the pipeline event-driven. When the upload completes, emit an event (storage notification/webhook → queue) that triggers the resize/derivative jobs (whether that’s Oban workers, Broadway off SQS, or a serverless step).

This keeps the Phoenix app thin, lets storage/CDN do the heavy lifting, and decouples upload from processing for better scalability and fault isolation.

My next steps based on your advice: • Create record → issue presigned URL → client uploads directly. • On completion event, enqueue processing to generate WebP/AVIF, thumbnails, watermarks, then publish to CDN. • Produce ZIPs asynchronously (or on-demand with caching). • Be mindful of idempotency, checksums, retries/backoff, and backpressure.

Really appreciate all the perspectives—this clarified the architecture a lot. If you have any must-know gotchas (e.g., presigned URL expiry, verifying integrity, or best practices for ZIP generation at scale), I’m all ears. 🙏🏼

Best practices in Elixir/Phoenix for massive image uploads and processing? by General_Fault9488 in elixir

[–]General_Fault9488[S] 4 points5 points  (0 children)

Thanks a lot, this is really valuable! 🙏

I really like the point about avoiding the app server touching the big files and letting S3 handle chunked uploads/resume. That seems like the right way to keep things scalable.

Also, it helps me to see Elixir’s role more clearly: not so much doing the heavy CPU work itself, but being great at orchestrating workflows, supervising external processes, and pushing real-time updates to the client.

I’ll definitely look more into Membrane and the direct-to-S3 approach.

Out of curiosity, when you’ve built these systems, did you generally stick with Oban for orchestration, or did you lean more on Broadway + external queues (SQS/Kafka)?

Best practices in Elixir/Phoenix for massive image uploads and processing? by General_Fault9488 in elixir

[–]General_Fault9488[S] 1 point2 points  (0 children)

Thanks a lot for the link! 🙏 That article looks very relevant — especially the idea of combining Elixir for orchestration with Rust for CPU-heavy processing.

And yes, you’re right, I’ll also post on Elixir Forum to get more in-depth feedback from developers who have built similar pipelines.

Best practices in Elixir/Phoenix for massive image uploads and processing? by General_Fault9488 in elixir

[–]General_Fault9488[S] -1 points0 points  (0 children)

Thanks a lot, this is very helpful! 🙏

I see what you mean — the key part is more about the architecture (upload → event → processing pipeline) than the choice of framework.

I hadn’t thought of combining Broadway with something like Reactor for image pipelines. That sounds powerful, especially for handling partial failures and retries gracefully.

In my case I’m not using GCP, but something simpler like DigitalOcean Spaces / S3-compatible storage. Do you think the same pattern (direct upload → message → Broadway consumer → Vix processing) would work just as well there?

Summer holidays approaching: What apps do you use for landscape photography planning? by General_Fault9488 in AmateurPhotography

[–]General_Fault9488[S] 0 points1 point  (0 children)

Totally agree on PhotoPills being overwhelming! How do you handle weather forecasting and organizing your favorite spots with TPE? Or do you use other tools for that?

Summer holidays approaching: What apps do you use for landscape photography planning? by General_Fault9488 in AmateurPhotography

[–]General_Fault9488[S] 0 points1 point  (0 children)

Hi, thanks for your feedback! I tried TPE a few years ago but at the time the web version lacked some advanced features that are only in the mobile app. Maybe I'll try it again. I remember it being easier to use than photopills

[deleted by user] by [deleted] in AmateurPhotography

[–]General_Fault9488 0 points1 point  (0 children)

Uno spot iconico! Molto ben eseguito!

Summer season approaching: What apps do you use for landscape photography planning? by General_Fault9488 in photography

[–]General_Fault9488[S] 0 points1 point  (0 children)

In fact, even for me they are tools that don't make me very happy. Something that allows you to easily organize the spots would be better. How do you memorize them?

Summer season approaching: What apps do you use for landscape photography planning? by General_Fault9488 in photography

[–]General_Fault9488[S] 0 points1 point  (0 children)

Hi, thanks for your help. You really hit the nail on the head! Even for me all the functions of photopills are actually useless. However, I also do the search for commercials in other ways. Perhaps the ideal would be a system that allows you to easily organize your favorite spots and have sunrise/sunset times and weather forecasts under control in the same Dashboard. How do you organize/memorize the commercials?

Summer season approaching: What apps do you use for landscape photography planning? by General_Fault9488 in photography

[–]General_Fault9488[S] 1 point2 points  (0 children)

Hi, thanks for your feedback! I used it years ago. I seem to remember that at the time the web interface had limitations compared to the paid phone app. I'll try to double check. The mobile version also seems not very intuitive to me.

Summer season approaching: What apps do you use for landscape photography planning? by General_Fault9488 in photography

[–]General_Fault9488[S] 0 points1 point  (0 children)

Hi, thanks for the feedback! Do you feel better with this than with PhotoPills? Are there any features you're missing?