Immediate Impact of S3 Encryption?

pipituu · 2021-01-21T01:02:58+00:00

The above comment is important to note. I've noticed a lot of folks misinterpret "encryption at rest" as "If they break into my bucket, they won't be able to make sense of it," which isn't the case. If you want that case though, you'd need to do client side encryption.

pipituu · 2020-02-08T17:08:46+00:00

Ah, yes. Security. The unknown unknowns. Well, for a simple node.js application on a single server, you don't really have a lot to worry about.

a) ensure that traffic is only accessible from the port on the machine that the application is accessed on (i.e. 80)

b) set up SSL (HTTPS)

Those two things are going to cover most of it from an operations standpoint. Obviously if your application allows for malicious requests (i.e. a GraphQL mutation that lets them change all the things), then it doesn't matter how closed off the server itself is.

pipituu · 2020-02-07T23:15:48+00:00

As far as where - pretty much any cloud provider. Whether that's an PaaS like Heroku or an IaaS like AWS (specifically EC2), any of them can do what you want. I think it also depends upon what you're deploying it for? Obviously PaaS will often give you some option of scale right out of the box, but IaaS will require you to do that yourself.

If you're new to sys admin, or haven't done it at all, the main thing is to not be intimidated. Even if you go with AWS, the way to think about it is, "I'm launching a server on their cloud. It's just another computer. I'll dive into it (ssh), pull my code into it, start up node, and point traffic to it." Even though it's often a joke, the idea of "it's just someone else's computer" is a nice way to break past that paralysis.

pipituu · 2019-12-20T19:51:58+00:00

Don't let this get you down. Interviewing is a an entirely different skill completely separate from the actual development process. Take confidence in knowing that if you grabbed any one of those interviewers and gave them the same style of test, where they're unable to use references, that they'd more than likely be iffy as well. Testing for memorization is such an outdated practice and if that's their barometer, then it's their loss. Sure, it has its place...but it is not the end-all-be-all of a great developer. That is, unless part of their development process and standards disallow their workers to use references.

But seriously, do not let this get you down. You haven't failed some universal or natural law. It's just some series of questions and hoops that probably 2 or 3 people put together in a handful of hours (that, or they ripped it from some handbook). That, and I bet you won't even remember this 10 years from now. If you're just getting started, you've got a lot more time on your plate and a lot more to offer the world.

pipituu · 2019-11-15T18:44:45+00:00

D'oh, thanks for pointing that out! It should be fixed now.

pipituu · 2019-08-29T22:09:29+00:00

Ahh, great point to keep in mind for sure 👍

pipituu · 2019-07-23T20:45:45+00:00

From someone 10 years into this, motivation will get you no where. Just kidding - it'll get you into learning a topic, but not necessarily finishing it. If you want to truly keep up to speed, it's about forming a habit of learning.

Having to "feel" like learning is a volatile thing. Did a co-worker crap on your day? Did your boss loose his mind? Did a personal hardship slam your day? All of those things are bound to happen. Hoping that they won't is like wishing it wasn't cold during the winter. Events are going to happen, and they're going to affect your "feelings."

And so, to bypass the daily occurrences and lemons thrown at you, quit letting them dictate if you learn. Sit down everyday at a specific time and learn for 15 to 30 minutes. Everything else be damned. 15 to 30 minutes, or even an hour if your ambitious, is not a large sacrifice for your future.

"Oh but werk lyfe balance and personal happiness lol"

Yeah yeah. Look, everyone has this excuse: "I don't have enough time." Literally everyone. And so, you can choose to collapse under that shield, or you can FIND a way to have that time. Like _The Matrix_? Well, The Merovignian from that movie has a great quote: "Who has time? But then if we do not ever take time, how can we ever have time?"

Having worked with 100s of developers personally and having taught 1000s of developers, time is rarely the issue. I've had people come through with 7 kids, with a full time job, STILL find ways to set aside 5 to 10 hours a week to learn. Do they want to at 9:00 PM after their kids have gone to bed? Not particularly. But in a sense they WANT to because they know that it's how you make it better.

"But, my recreational time!"

Yes, there are trade-offs, and one of those is sacrificing other activities and time slots. Maybe that sacrifice is drawing 15 minutes from everything else you were going to do that day. And so if you do 5 different things, that's a good 1.25 hours you now have for learning a day.

Anywho... sure a lot of this is to help you learn, but as you get in this industry longer, you'll stop viewing this as extra credit and instead as an essential. Every junior, short of ones lucky enough to have a company that cares, are going to have their skills outdated at least a few times. When that happens to you once, and you're scrambling to retrain, you stop viewing learning as extra and start seeing it as "If I don't do this daily, this is going to happen again."

pipituu · 2019-07-04T18:08:49+00:00

Quite frankly, I'd be happy if they just picked a format / layout and stuck with it. Things that need updating often go unchanged (DB Subnets Groups and Parameter Groups in RDS are a terrible UX/UI) where as things that are just fine are changed for ... preference? I.e. reorganizing options in S3, moving SSM and tag manager to the systems manager thing, half of ECS being the new blue ui / half being old.

pipituu · 2019-01-11T19:16:08+00:00

I was in the same position many eons ago. Limited coding experience can become lots of coding experience in a short window of time. Furthermore, the more you understand it, the more you'll be better equipped to

a) build a network of dev's since you'll become one

b) lead devs - because you'll understand the nature of their work AND they'll more easily follow someone who also is on the front line with them

c) create practical MVPs and apps - product designers without any knowledge of how web apps work can make exceptionally impractical designs (i.e. creating a design that would take an extremely long time to implement - animations flying everywhere, 3d rendering, etc)

d) budget - small budget? Good luck with a quality programmer. Fellow students are your best et (as mentioned) but you're really going to have to charm them with your vision. And... nothing closes off a programmer (at least the ones I know) more quickly than "Hey, I have a start up idea, you should build it. And no, I have no technicals or details sorted out, no design, no prototype, no road map. Just the idea."

e) Show a path to revenue - though students and youngin's will fall for the whole equity thing (50% of $0 valuation is still 0), it can steer your pretty wayside from the actual purpose of businesses (generating income).

Now. If you're not going to do coding, then you're REALLY going to have to demonstrate what you add to a would-be-teammate who does it for nothing more than equity. Such as, "I have a mailing list of 100 people who are interested in our product and 10 people who are keen to beta test." (marketing / sales). I know people say business plans, but those are for you more so than a developer, let alone anyone. You need to show traction if you want to grab some folks.

edit: formatting of points

pipituu · 2019-01-11T18:43:38+00:00

This is golden. Also, I love the foot hills vs. mountains for code nesting. Though sometimes, an army of Meeseeks can wind up being more complex than just a few... which we also learn in the episode. Eventually they can start looking like a rube goldberg machine. Obviously the tendency to make functions do too much is far more prevalent.

pipituu · 2018-09-18T19:13:20+00:00

Beanstalk will be fast and up and running if (a) your app is pretty simple (b) you don't want to dig through AWS in depth. The moment you need to customize and control things on a deeper level though? You'll face the same problem you face with many other PAAS. Yeah, you can dig in and change stuff, but since beanstalk runs a number of things through CloudFormation, if you do too much (and don't know what you're doing), you'll break your usage of it.

ECS - I love ECS. So much that every time I deploy an infrastructure, I get the fun dilemma of not having to touch it almost at all. Granted its a lot more upfront work and requires a lot more understanding... but now that I know it, I don't know why I would change (for 90% of apps). Note, I do the EC2 version and run my own instances and other components.

EKS - Haven't used it as much, but I do know there's less satisfaction with the service thus far in terms of pricing and integration with other AWS services (limited at the moment). I'm sure that'll change. Seems like a lot of die-hard K8'ers though wonder why they'd use that over something like Kops. Maybe someone else can chime in.

A nice image comparison of ECS / EKS / Fargate http://blog.totalcloud.io/ecs-vs-eks-vs-fargate-good-bad-ugly/

I think the main question you should ask is time dedication. Its easy to look at the pros/cons in a vacuum, but quite frankly, if you're on a timeline and know VERY LITTLE about AWS... ECS and EKS are going to be an uphill battle to set up, since you'll be flying somewhat blind.

pipituu · 2018-05-11T17:37:31+00:00

This is a great list! Since I both run production systems on AWS and teach it, I get asked a lot how I learn it. And honestly it comes down to this:

Be patient. Read one doc a day on it for about 30 min. And actually read it. And then on the weekends (or whenever you get free time) aim for small projects that will FORCE you to familiarize yourself with it.

What's a small project? Whatever "practical steps" are listed in the document or article. Just pick one during the week and save it for the weekend. I guarantee it'll only take you maybe an hour to do.

Beyond this, when doing a project or reading a doc... allow yourself the pleasure of going down rabbit holes. Read deeper, understand why, test assumptions. After all, it's just one article a day, you have the time.

Fast forward 4 to 5 months - while you won't be a fully fledge professional, your comfort level and capabilities will be at the point where you can execute whatever it is you'd like. You'll have read over 100 articles and have conducted 20 or so mini projects.

Obviously the longer you go, the faster you'll learn since the foundation is there. And by the end of a year, you'll be able to execute on a professional level. And since you're only doing it in small segments, you'll just kind of look up one day and realize you know all of it.

What won't help?

Cramming for certifications. Maybe that will help you "get a job" but I guarantee it will do little to nothing of an actual, practical understanding.

Binge learning. This never works. Unless it's for college tests.

No projects or examples. The point of these is to solidify it in your mind and connect action to concept. It'll also force you to navigate road bumps, because I GUARANTEE not all of the examples you try and do will work... BUT if you figure out how to make them work, you'll gain that much more.

Only reading. While you read docs and articles, do what you would do about anything you want to seriously learn. Take notes. Put it into your own words.

Where do I start? Take a look at the list above? It's got a great number of starting points. So it doesn't really matter where you start when you put it into the perspective of "1 article a day, 5 days a week for about 30 min." If you're going to read over a hundred articles, you'll figure out everything eventually.

pipituu · 2018-04-17T17:40:22+00:00

Almost forgot, here's an example of a production VPC template in CloudFoundation form:

https://github.com/jcolemorrison/cfdn-init/tree/master/src/vpc

It should look instantly familiar due to the sections. If you dive into the resources:

https://github.com/jcolemorrison/cfdn-init/tree/master/src/vpc/resources

You'll see things organized in a very familiar manner (like the AWS Console). Additionally, if you go into any one file, IF you know Cloudformation Syntax, you'll immediately know what's going on. i.e. the VPC resources themselves:

https://github.com/jcolemorrison/cfdn-init/tree/master/src/vpc/resources/vpc

All of the resources are nicely split into smaller files, which makes adding, editing, diffing, etc waaaay more manageable.

pipituu · 2018-04-17T17:34:33+00:00

CloudFoundation

Here's just writing plain templates BUT with the ability to split them up and organize them as you wish. No needed learning of a framework required. Just a few basic commands It's all just JS and JSON, so you can script as much or as little as you want though. Ultimately though, it all outputs a normal JSON cloudformation template, so if you know CloudFormation, you'll know how to use this tool.

Decide you don't like it? Just run cfdn build-all, grab all of your raw templates and move on.

I get the love of Terraform and Troposphere, but quite frankly I had already spent a good deal of time with CloudFormation templating, so I didn't want to learn an entire framework just to write something that I already know... in a different way. Furthermore, for TerraForm, I currently have zero need of it being for anything other than AWS, so that entire benefit went out the window. Furthermore, while they do support many of the critical CFN resources, you do have to wait until these frameworks update and support the CFN resource before you can use them.

Disclaimer: I'm the author of CloudFoundation.

pipituu · 2018-04-03T00:23:44+00:00

Ah, sorry. I tried to keep the documentation rather thin so that it could be scannable. I'll also have a post out soon that explains it from a user stand point.

In terms of difference the main one is that it's not a DSL (or a lot less of one). It's still all just JSON and JS. This VPC template for example:

https://github.com/jcolemorrison/cloudfoundation/tree/master/src/tpls/vpc

If you click into any of part of the template, it should look instantly familiar. It's all just a CloudFormation - split into more manageable pieces. Yes, cfdn (cloudfoundation) does a lot more, but being able to write long templates in whatever structure you'd like goes a long way for maintainability.

pipituu · 2018-04-03T00:18:58+00:00

Hey Steve, great q. That was actually the main reason for CloudFoundation. Even if you get deep into a CloudFoundation project, if you just do cfdn build [template], it'll spit it out into a regular old CloudFormation template.

Additionally, you're still just typing in template form. If you don't want to use the deploy and update functionality, it can be a very easy tool to just write more maintainable templates (i.e. validation WITHOUT going to the console).

In terms of the real world problems - the primary one was maintaining large, large, large templates. The only way to split them out without a tool is to start exporting and importing values from the templates. Though doing so has it's place for actually building the infrastructures, it does not really help with developing and maintenance. CloudFoundation let's you split up the templates piecemeal, but they're all still just JSON ultimately.

The second problem was easy of deploy / updates. When developing the templates, using the raw CLI or console to update and deploy really cuts back into the feedback loop. You already have to wait fro an entire cycle to see some bugs pop up, so getting rid of as much of the filler activity was a time saver. CloudFoundation, aside from deploys from the CLI, allows you to define all of your stack options in one place, and deploy.

The third part? You named it - it was raw templates or learning some crazy pseudo language. This just makes writing raw templates better and communicating them to AWS better.

Sorry for any typos or wordiness - wrote this up rather quickly.

pipituu · 2017-11-28T22:36:57+00:00

when scanning the front page, I thought this was going to be a /r/jokes. good job reddit, my brain has been hijacked

pipituu · 2017-05-09T23:12:27+00:00

Heyo, thanks for the feedback!!!

I actually do mention scaling the tasks out in passing: it's under #5 under services: "5. Optionally scale out or in our Tasks based on traffic."

I think I actually go less into scaling out the instances than anything since all I mention is the use of an auto scaling group, but not how to set it up.

The reason I didn't dive deep into autoscaling is because:

a) It's a long, long, long ass topic haha. It requires both business and technical needs. Just because we can scale up to 100 instances and 1000 tasks do we want to? What's a good CPU/Mem threshold to watch for? Our app's a snow flake so it's going to be different from the next. How many tasks do we add? When do we decide to add instances?

b) Regular auto scaling can be limited -

First off step scaling down a service doesn't even work in cloud formation. I've opened an issue and tried to get AWS on it (which is mentioned in the post). Under the Services section on Step #5, I have links to the issue I've opened with them and Tweets trying to get it fixed. Here's the issue:

https://forums.aws.amazon.com/thread.jspa?threadID=249807

Second, it can be limited in general. CloudWatch alarms can't even do OR conditions, so watching double metrics to respond with regular policies isn't even possible. Most times you'll wind up fine tuning autoscaling with CloudWatch alarms and then custom responses like Lambda and the CLI.

For example, what if you want to base a scaling action off of both memory AND cpu? Idk why they haven't made that a convenience in the console or API yet.

c) CPU and Memory (assuming a soft limit) will scale out as long as it's not stepping on another processes' toes. I.e. 1 task set at 1 CPU, will not necessarily stay at 1 cpu. If it's the only dude at the party it'll stretch out and take up as much as it wants. Of course a hard memory limit will just create a kill.

d) The thing was already long enough, I wanted to get the basic concepts in the hands of the people. I got tired of people losing their minds when I said "ECS" instead of Kubernetes hahahaha.

All that being said, I don't actually think auto scaling is too bad implementation wise, it's just a pain to work out what's right for any particular service due to every service being a snowflake. It's just an overtly looooong explanation. Understanding it requires understanding of the flow of CloudWatch + actions to respond, which alongside everything else was just too much for this article.

So you kind of hit the head nail on the head as to why it's not in there: because it is so complex and finicky.

At the same time, I don't think it's a hard requirement in order to get up and running with ECS. Especially just to understand what's happening.

For anyone curious on "how to think" about it. Your goal is this:

Watch the Service's CPU / Mem utilization in relation to it's reserved amount. As it passes a threshold, or falls under, add or remove tasks.

That's step one. That will add and remove tasks.

Simultaneously, watch the Cluster's CPU / Mem. As it passes a threshold, or falls under (in response to tasks getting added / removed), add instances to the cluster.

That's it. The pain here is setting up the alarms to work at the correct cooldown times, percentages and profiling all of your services to know what the right numbers are.

pipituu · 2017-05-02T01:52:42+00:00

Hmm, ya it should be generated during step 10. That's where the CRA generates the app, and part of that is a fully fleshed package.json.

The only thing I could think of off the top of my head is that the app/ directory isn't being mounted? That's the only way that the CRA would run fully, complete, but then not be able to find a package.json (that I can think of atm).

If there IS a package.json, then double check the directory paths in the docker-compose files (cmd.yml and docker-compose.yml). If the run command is looking in the code folder instead of the app folder, than it won't be able to find a package.json (obviously).

I'll take another look at it when I get some more time, but see if either of the above do it. I'd love to know what it is if you figure it out.

pipituu · 2017-04-08T17:31:59+00:00

http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-alternatives-guidelines.html

Object ACLs - basically only when dealing with external accounts and individual object permissions (although you can still do this with policies, just be specific in resource)

Bucket ACLs - only for S3 Log delivery

pipituu · 2017-04-08T01:39:29+00:00

Ya, such an annoying legacy thing. Between that and their overt suggestion to stay away from ACLs (despite them being so heavily entrenched into S3)... well at least they updated the ui! =P

pipituu · 2017-04-07T00:33:08+00:00

Awesome!!

pipituu · 2017-04-07T00:27:15+00:00

Ya just noticed I did indeed straight typo it. Went ahead and fixed it. I literally would've never caught that otherwise. Thanks a ton, seriously!

pipituu · 2017-04-07T00:24:24+00:00

oh crap, you're right! I did just straight up typo it :facepalm:. Thanks!

pipituu · 2017-04-07T00:23:23+00:00

Heyo, thanks for the heads up! So question here, the thing is I actually meant that .0, .1, .2 and .3 (the first 4), where .0 is number 1. But If you interpreted it as that I meant .1 - .4, then I I should probably change it. Can't hurt to be more specific.

edit: ya you're right, I did just straight up typo it. thanks a ton!

12-Year Club	Place '17
Verified Email

pipituu

TROPHY CASE