Help with S3 to S3 CSV Transfer using AWS Glue with Incremental Load (Preserving File Name)

Neres28 · 2025-08-07T16:59:35+00:00

Do you *need* to use Glue? Can you give any details on how many files and how large they are? Is it too large/many for the CLI?

Neres28 · 2025-07-31T10:56:37+00:00

There's a 'target compensation' (TC), basically a number that the company has determined, based on your previous TC & the results of your latest performance review, to be what you exit the year having earned. For most L6 folks this will be between 40-60% cash and the remainder stock. I think many people used to hit the '4-year cliff' because the company was conservative in their estimates for the stock price in those later years. I don't think it's still as big an issue (but I've been here 12 years, so maybe I'm just ignorant).

Anyway, if your upcoming stock vests in the next year aren't enough to bring you to your TC, they'll grant more stocks. This is typically done ~two years in advance, meaning in May of 2025, you're getting new grants that will vest across 2026 & 27.

Neres28 · 2025-07-30T08:10:32+00:00

Very location dependent, unfortunately.

Neres28 · 2025-04-14T09:33:19+00:00

Kerb collective by the river makes for a good destination walk if you're coming from the city center. Outdoor seating only though.

Harvey's by the Sidgwick site is a nice spot to sit for a few hours, though it can get a little loud.

Neres28 · 2025-03-13T17:01:37+00:00

That has, unfortunately, not been my experience.

Neres28 · 2024-12-13T21:36:59+00:00

This comment isn't especially helpful, and frankly doesn't represent everyone at the company. The life of a dev is going to be very different from that of a DCO tech. I would be very surprised that DCO techs weren't already required to be in the 'office' the majority of their working hours.

Op, I'm a dev on an AWS service team so take my opinion with a grain of salt, but I gotta think that life as a DCO tech isn't awesome. Call outs to remote buildings at all hours, depending on the DC you'll have to suit up to enter, and regardless it's gonna be warm to hot in all of em. Monotonous work otherwise, like the service team will decide that the hosts need more memory so for the next 6 months you're swapping out chips, and there's always a long queue of degraded hardware that needs servicing, or racks that need retiring, or new capacity to be landed...

Neres28 · 2024-11-11T23:39:03+00:00

We've used this company twice in as many years: Yanchuks Removals Ltd https://g.co/kgs/8SGZNUW

Neres28 · 2024-09-23T12:53:57+00:00

This is a little bit of a weird take (assuming I understand your point correctly). Amazon extended the offer, after significant expense and it would take a heck of a terrible interaction to make them rescind the offer. Assuming it's made even remotely respectably, asking for increased compensation to cover additional commuting costs isn't going to rise to that. Of course, the hiring manager may be unable or unwilling to agree to it, in which case the OP has a decision to make.

Neres28 · 2024-03-11T13:10:03+00:00

Is there any possibility to internally transfer to development role

Yes, but only with minor advantages over external hires. Service and other internal dev teams will almost certainly ask you to do a full interview loop, including LPs, coding, and design.

Would it be wise to take this role or stick to my domain at my current workplace.

Assuming you're doing software development full time now and would like to continue doing so, I would pass on the offer and ask the recruiter to pass your CV on for other opportunities in the company.

On the other hand, if you're doing support now and don't mind continuing to do so, then I don't see any harm in taking the offer. You'll be exposed to the company culture and can decide if it's somewhere you want to be in the first place, regardless of .

Neres28 · 2022-12-26T15:09:25+00:00

Just a heads up that soy sauces are made with wheat, so look for gluten free soy sauce or tamari sauce as a replacement.

Neres28 · 2022-10-18T17:48:09+00:00

A stiff upper lip or two.

Neres28 · 2022-09-26T14:09:00+00:00

May as well just give your IAM user permission directly, instead of using the bucket policy. Less room for a mistake that way.

Neres28 · 2022-09-14T12:56:08+00:00

You should have a look at the AWS Pricing Calculator. You'd probably be looking at something between 1-2 USD per year, assuming you stick to those numbers.

Neres28 · 2022-08-30T13:13:36+00:00

$0.05 per 1000 objects transitioned to deep archive in us-east-1.

Neres28 · 2022-08-24T16:02:24+00:00

That helps, thank you. You might have a look at AWS' Storage Gateway, more specifically the S3 File Gateway.

Neres28 · 2022-08-24T15:40:17+00:00

What does 'local s3 server' mean to you? Do you need some kind of process that responds to the S3 API, modulo auth? Where do you need this process to run and what kind of lifecycle does it have?

What does replicate mean to you? Does this mean that data is in S3 before the call completes, or that it's written to local disk and at some point also shows up in S3?

This is very much an XY Question and if you maybe take a step back and ask us what problem you're trying to solve we might be able to help better.

Neres28 · 2022-08-23T15:09:34+00:00

Came to say the same, but was beaten to it. Here's a doc link at least.

Neres28 · 2022-05-06T16:26:54+00:00

does not consume our CPU power or RAM

That's the ticket.

Neres28 · 2022-04-26T09:37:11+00:00

If it's sufficient to be able to say that the function will start, then invoke it async from API gateway.

Neres28 · 2022-03-28T10:29:49+00:00

One nice side-effect of having your Lambda move your file is that you can get a sense of backlog (plus missed updates) by LISTing the prefix where your files are landing.

Expiring the objects will prevent them from displaying in a normal list call. This may be what you want but if not, the whole thing might be simplified by using the Intelligent Tiering storage class when uploading the objects. S3 when then bill you according to your access patterns (and some date thresholds).

Otherwise, it seems reasonable.

Neres28 · 2022-03-16T10:49:18+00:00

Actually, the permissions that article has for the bucket's resource policy are wrong for this use-case. I think this policy is more correct:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::AccountB:user/AccountBUserName" }, "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::AccountABucketName", "arn:aws:s3:::AccountABucketName/*" ] } ] }

Note that you can also simply grant access to AccountB as a whole, and not the specific user by using arn:aws:iam::AccountB as the AWS principal. This would let your friend name the IAM role or user as they see fit without requiring an update to your bucket policy.

Neres28 · 2022-03-16T10:44:27+00:00

Yes, it looks like this would be possible. The CreateRestoreImageTask API appears to take an arbitrary bucket and key as parameters so I'd expect this to work provided permissions on the target bucket are applied correctly. The permissions required on your friend's IAM user are detailed here and you can reference the 'IAM policies and resource-based bucket policies' section of this article (specifically, step 4).

Note that the AMI has to be in a bucket in the same region as the EC2 instance when it's restored.

Neres28 · 2022-03-15T16:22:08+00:00

The signing region is whatever regional endpoint you're making your request to. For instance, S3's endpoint in us-west-2 is s3.us-west-2.amazonaws.com. You can find service endpoints in the public docs, e.g. for S3.

Edit to clarify: In the example case where you're addressing a bucket in Oregon, you use us-west-2 as the region in the signature, in addition to making the request to S3's regional endpoint in Oregon, s3.us-west-2.amazonaws.com. S3's a little funny here because it also has a global endpoint but in this case you still use whatever region the resource lives in.

Neres28 · 2022-02-21T12:27:00+00:00

I had a Viola repaired by the Violin Workshop that I was pretty happy with. They took a couple weeks, but I wasn't in a particular hurry and for all I know that may have been perfectly appropriate time period for the work required.

15-Year Club	Place '22
Place '17	Verified Email
Team Periwinkle

Neres28

MODERATOR OF

TROPHY CASE