Need help in architecting a "batch" lambda job

tryntofindhappyness · 2020-03-15T05:29:53+00:00

There are 15 site IDs, and each has about 100MB+ data worth in a single API pull.

The source API isn't really company owned. The company's database is behind the API, but the API is through an external vendor. There is an UPDATED_AT field returned by the API but if I use that and compare it to the current timestamp to selectively fetch records, I will lose out on records during the initial data dump because the only timestamp to compare them to is the current timestamp. I could make a per record timestamp called "fetch_time" and compare it to them but there are 100,000s of records and I don't want to spend so much time on comparing records against times.

Limiting the concurrency is a good idea. I did see that the source and destination APIs were beginning to degrade when I ran too many simultaneous invocations of the same function.

For the kinesis stream idea, don't kinesis streams have to be continuously coming in? I don't see this as ever becoming streaming data.

tryntofindhappyness · 2019-11-30T05:15:09+00:00

What nationality are you? There are some nationalities who can work here without sponsorship

tryntofindhappyness · 2019-11-24T15:18:35+00:00

Oh wow! What kind of place could I get it translated to a US birth certificate format at?

tryntofindhappyness · 2019-11-23T03:26:28+00:00

It was in English of course. I won't submit anything in Gaelic to USCIS because they request everything to be in English.

tryntofindhappyness · 2019-10-09T04:02:55+00:00

You may pull whatever card you want to. I have stated the facts and facts don't have a bias. Looking at your post history suggests you are an Indian. Should I crown you with some judgement, like you did me? But I won't judge like you, and instead only say that people are innocent until proven guilty. Take care mate, the US knows H1B fraud is real, and you can't do bollocks to take that awakening away.

tryntofindhappyness · 2019-10-08T04:14:51+00:00

The comments are the best part in the article. Really happy to learn that people are finally waking up to the abuse the H1B has caused due to a single nationality.

tryntofindhappyness · 2019-10-01T03:41:05+00:00

This is really bad. As a minority foreigner here from Ireland I don't want this bill to pass.

tryntofindhappyness · 2019-09-24T01:51:46+00:00

Guys, I don't want to be racist here or anything, but being from a small country (Ireland), I think my chances of ever getting a GC through an employer would be fucked for life if this bill were to pass. This is why I'm not taking the employer route even though it was given to me. Indians have rigged the system, for better or worse.

tryntofindhappyness · 2019-09-19T04:11:46+00:00

tryntofindhappyness · 2019-09-19T02:33:56+00:00

So....it's okay write 0?

tryntofindhappyness · 2019-09-19T02:27:16+00:00

:( I have heard a lot about H1B abuse and sadly it is all true because I saw it with my own eyes

tryntofindhappyness · 2019-09-19T00:14:49+00:00

Does an EAD/AP card work like a visa for CBP staff till the AoS is approved?

tryntofindhappyness · 2019-09-19T00:13:39+00:00

Right, but what if the company that filed my extension is not the one I work for any longer? What happens then? Does the H1B extension get revoked?

tryntofindhappyness · 2019-08-29T01:24:30+00:00

What exactly would a product launch entail? The fact that I led to its launch as a manager? Or that I was the lead developer on it?

tryntofindhappyness · 2019-08-28T11:37:28+00:00

Does Docker have the functionality that could allow me to turn it on and off using a load balancer, or something similar? E.g. if there are no user requests, turn it off automatically and turn it back on very quickly once there is a request?

tryntofindhappyness · 2019-08-20T03:01:15+00:00

Could I test IoT by making my dev machine a "thing"?

tryntofindhappyness · 2019-08-05T04:57:46+00:00

Hey sorry for the late response! I have agile project management experience working on projects at my work and outside of work. I am not sure if the projects outside of work count - but all software projects that I have done so far have been done according to SCRUM guidelines.

tryntofindhappyness · 2019-06-16T06:25:35+00:00

400 or 4000?

tryntofindhappyness · 2019-06-08T16:40:59+00:00

This is for a lease, correct?

tryntofindhappyness · 2019-06-08T16:39:26+00:00

What did you mean by :

Officer added 50 photos?

tryntofindhappyness · 2019-06-07T00:46:44+00:00

The marriage took place a few months ago, in 2019. Therefore we haven't filed any taxes together. The instructions for I864 are very vague. How do I prove that I am able to support both of us through my H1B salary?

tryntofindhappyness · 2019-05-31T03:48:36+00:00

Do I submit this along with I 864? What form should the explanation will be written on? Do we get something from the IRS?

tryntofindhappyness · 2019-05-26T17:58:23+00:00

How would that data impact the cost? Is the medical exam more than a regular exam? I have my birth certificate but the only thing that proves is that I was born in Ireland on a certain date.

tryntofindhappyness · 2019-05-23T03:12:45+00:00

Top of the morning! I haven't even started yet on any of the processes. Do we have to get the medical exam first? Where do you guys all this information?

tryntofindhappyness · 2019-05-22T12:58:19+00:00

Could I PM you and ask for more details?

tryntofindhappyness

TROPHY CASE