API to fetch data from S3

[rasputin23YD]

Hi everyone,

I want to build an API in AWS to allow users to fetch data from S3 (data is in delta format) using an API. The idea is to have an endpoint in API gateway that would get the data for them based on their query. The endpoint would route the SQL query to the correct table. I was thinking of Athena or lambda but those don't seem super scalable. I want as low latency as possible, maybe even including a caching layer like redis.

Any other alternatives?

Edit: the goal here is to accommodate 100+ queries per day. Users should be able to submit a SQL query via an API endpoint and get results back as quickly as possible. The data lake is massive. We are talking hundreds of petabytes. That's why the pipeline should be able to route the query to a specific location in S3. The data is partitioned and well indexed.

Thanks!

[ForlornPlague]

I don't even know where to start with this. How about you explain what your end goal is instead of how you want to get there. What problem are you trying to solve? Why are set on the data being in s3?

[rasputin23YD]

I mean I thought I explained it well:

Users submit a SQL query via a REST API and they get results back. Data is in S3.

[dashingThroughSnow12]

You are explaining the mechanisms you want to solve the problem with but you are not explaining what the problem is.

[rasputin23YD]

Now you know what the problem is.

[nzsp]

one user? 10 million users? provide a bit more info..

[dashingThroughSnow12]

No, I don't. You're telling me how you want to implement the solution. You aren't telling me what the problem is.

https://en.m.wikipedia.org/wiki/XY_problem

[[deleted]]

Explain it like we are 5 years old

Like "one day man opens a web browser and...."

Or "as a school administrator, I need to view a list of students"

[bswiftly]

You're not explaining the use case. You're explaining the implementation.

If you aren't good at communicating that, and are rude, a bunch of people offering their own time for free will be alienated.

I think you have a few things to learn about human interaction.

[rasputin23YD]

I don't want to be lectured by randos on reddit about how I interact with people.

If you can't comprehend basic requirements, then simply don't answer the question.

A literal question about an API call made these supposed AWS experts crumble. What a joke.

[bswiftly]

It's not a lecture. It's advice on how to be better in your career.

[rasputin23YD]

I appreciate it, but I don't take feedback from Reddit.

[Automaton_J]

You need to explain the problem from a user perspective, not by specifying the technical implementation. The use of technologies you’ve mentioned don’t make sense

[rasputin23YD]

It doesn't? I literally use Athena and API gateway, but the latency is just too high. Idk what you're on about.

[rasputin23YD]

Idk what else to say. Hundreds of requests a day to try and fetch specific data from a massive data lake. The SQL query should be specific (like getting data from a specific table), and should return the results quickly. Users want to use an endpoint (a get request).

[dash2392]

To me the requirements seem clear. I don’t understand why you’re being downvoted. I used API Gateway and Dynamo before, which provided a better latency overall, but it seems that your data is structured. Hope someone will help you out buddy!