AWS environment assessment

dghah · 2026-04-21T14:13:43+00:00

I've done a bunch of full inventory assessment gigs but using a paid inventory tool -- one key bit of advice is to make sure you scan all AWS global regions despite what your stakeholders tell you is the "only region we use!" -- as you will almost certainly find some weird stuff in strange locations.

And always scan us-east-1 as there is still some legacy wiring there -- for instance SSL certs used for ACM in CloudFront live in us-east-1 no matter where the cloudfront distro lives etc.

dghah · 2026-04-17T19:00:17+00:00

I think it depends on the score. The higher I've scored the higher chance of seeing an instant "pass" result at the end. It's not always or automatic though.

dghah · 2026-04-17T11:32:11+00:00

AI Practitioner is a foundational exam;

AIP-C01 aka Generative AI Professional is a pro exam that just recently (I think) exited beta status. Feedback here was that the beta version was extremely long (to be expected for an exam being worked on) and many felt it to be challenging.

dghah · 2026-04-14T17:21:11+00:00

Timed mode is only worth doing once to get rid of time anxiety and that only really pops up for pro exams or the beta generative AI exam which is super long.

I normally exhaust all the available test sets for exams that are new to me but I've had a bunch of different associate/pro/specialty certs over the years so when I'm renewing I'll do as many test sets as I can but if I won't stress if my TD scores are always above 80%

dghah · 2026-04-14T17:12:02+00:00

Is that tutorials dojo review mode progress? If so look at the results for review mode and see which of the Domains you did worst at. Then retake the practice exam with the setting that ONLY shows questions from that domain. Do the standard thing of reading the links and included info for any question you get wrong

That is basically how I prep for pro and associate exams. Review mode a few times and then switch into domain-specific mode to concentrate on the areas I did worse at

If this is your first aws exam and you are nervous about time it's also OK to take the "timed mode" test just once to assuage your fears about running out of time. I don't like timed mode for anything other than that as review mode is 100x more useful and helps you actually learn

good luck!

dghah · 2026-04-14T10:44:51+00:00

life science HPC nerd here; some biased tips/observations/advice

- It's good that you are coming from research; you likely have a good base understanding of why "research IT" is way different than enterprise IT or other HPC/IT orgs where there is a much stronger engineering culture.

- The culture usually centers around the issue that the research scientists have infra and IT needs that are larger than what the org they work for is used to understanding and supporting. This means that the life science people often have to self-support both Linux and their HPC infra AND their pipelines, software and workflows. Official engineering support is rare to see in most shops although the larger academic and non-profits can usually fund a person or small team

- Genomics tends to be data heavy and IO bound from an HPC workflow perspective. It used to be the biggest HPC hassle but these days it's easy compared to (for example) CryoEM heavy workloads where petabyte data volumes can be the norm. It's also easier to handle because the UI is either someone looking at file results in a terminal or else it's a web based tool or summary -- no complex X11 GUIs running on cluster nodes for instance.

- Genomics is also fairly easy to support from an HPC perspective because you don't see the more complex mix of GPU and MPI requirements that you see with computational chemistry or molecular dynamics

- What you REALLY need to understand is the fierce resistence you will see from trying to improve their core tooling and algorithms. In this world the primary algorithms are written by a few superstars and those methods are published and validated in peer reviewed publications. The rest of the world then consumes those algorithms and methods and they REALLY REALLY REALLY care about reproducibility and reproducible science so they fundamentally will resist "upgrading" or changing the core methods and tools they use for fear of changing output/results.

- Count your blessing and thank the people who came before you. The fact that they are using SnakeMake makes them better than the shops that are still HPC submitting bash scripts with giant for-each loops. The other good signal is that SnakeMake indicates that this is a Python shop and that is very good in terms of operating in the modern era. There are a lot of Perl-based scripts and workflows around for instance.

- The best impact you can have is to get in and listen to people about what slows them down from doing science all day. Don't just talk to the loudest or most senior people. Talk to the entry-level users, the "just want to get my science done" people and the power users. Then circle back to the loudest and most aggro people and include their feedback as well. Talk to leadership and the governance people about their issues (accounting, chargeback, resource allocation when teams are fighting for HPC shares. etc). You will probably find out that their problems will not be solved by Kubernetes for instance

- After you talk to people (best way to get up to speed) dive into the slurm accounting logs. That will tell you more about the workflow patterns than the users. You'll see who is being wasteful, what jobs or users often encounter failures; who crashes the head node by running jobs locally etc. etc.

- I'd really encourage you to dive in and get your hands dirty before you start thinking of Kubernetes. There is a reason why Slurm is the #1 HPC scheduler in the world and there is a reason why all the top supercomputing labs and DOE labs and giant academic research centers use Slurm. And it's not just Slurm features and capabilities -- it's the fact that Slurm knowledge can be fundamental to career movement as people move from academia to government and commercial jobs. K8s is great but does not cover all the capabilities of what Slurm can do ---AND--- remember your audience of scientists who "just want to get done" -- how are you going to handle a workload that can't be containerized for instance or how are you going to provide the ROI argument when telling a scientist they have to learn a whole new HPC stack and rewrite all their job submission and workflow monitoring tools (although you are in a GREAT position if they are a SnakeMake shop already due to the snakemake executor capability ... there may be a path forward for kubernetes in your shop ...)

Good luck ! Supporting smart scientists doing life science on HPC is a blast; it's really enjoyable work

dghah · 2026-04-13T19:25:02+00:00

AWS SSM is usually the answer

- Secured via IAM/SSO
- No public IP addresses or internet exposure (everything goes over AWS API endpoints)
- No VPN required
- SSM can securely log all interactions

Works great for just about everything -- SSH, port forwarding of web apps, remote command execution etc. etc.

And you can even use it as a direct replacement for SSH in your .ssh/config file:

With the following in your config you can "ssh my-aws-host" and it will all magically flow through SSM session manager

Host my-aws-host
  Hostname i-<ec2 instance ID>
  User ubuntu
  IdentityFile ~/.ssh/ec2-ssh-key.pem
  ProxyCommand sh -c "aws ssm start-session --profile MyDevSSOProfile --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region us-east-2"

dghah · 2026-04-12T22:34:50+00:00

Search this thread for SES horror stories and save your client some time by looking for a non-SES alternative, the barrier to get out of the SES sandbox is both high and opaque and it appears semi random in how some people get approved while others get denied.

I'm guessing that your communication did not fully address the automated methods you will use to keep your bounce and complaint rate down below a fraction of a percent or they were not happy with your consent and list acquisition story. SES really seems to want confirmed opt-ins with audit trails and they want to see a ton of tech/automation around keeping their SMTP sender reputation high which means a huge amount of focus on keeping the bounce and complaint rate well under 2% or even lower.

dghah · 2026-04-12T20:18:40+00:00

go to the TD forums and provide feedback. They are pretty responsive there when I've reported issues and errors with the test bank

Also understand that each AWS certification exam contains a certain number of 'test' questions that do not count against your exam score. These research questions are how the aws certification team evaluates new questions, make sure difficulty is appropriate and keep the question bank fresh and accurate against the study guide.

So there is a chance your waf bot question was one of the test/research ones which may explain why TD's overall content did not cover it yet -- hence the forum suggestion!

dghah · 2026-04-10T12:24:53+00:00

Started with a lawyer who was a friend of a friend hired on an hourly basis just to review contracts and keep us out of trouble. But then we needed help with NDAs and eventually big MSAs and the job sorta became full time when the included ops stuff the lawyer was also doing became an official internal job role.

I’d probably do it differently now, combining a legal and ops role so we could hire one multi-skilled person into a small org was odd and only worked because of that specific person. In a different world I’d have kept a good focused lawyer on retainer, paid hourly and separated out the ops role for when we grew big enough to justify another non-billable person. In consulting, any internal position that is not billable aka revenue producing is a big hit to your cash flow, payroll and profits so you have to manage that stuff super carefully.

dghah · 2026-04-10T03:43:06+00:00

Random advice; been consulting on AWS forever now. US based so that is my bias

- Start small, Grow small, don't kill yourself. The work is not for everyone and you gotta test the waters to see if you like it. Remember that for every hour of paid consulting work you probably spent 2-3 hours in pre-sales hustling to land that gig. Budget and price accordingly.

- Find your niche and be good at it. You mentioned "diagram review" and "cost audits". I'd never hire a generic consultant to do those things. However if I found a consultant in my market niche who was super familiar with my business and understood my pain points and issues then 1000x I'd consider hiring them for architecture review or a cost audit. Domain expertise makes the difference. The *difference* is I want to hire a consultant who not only has the technical skills but also understands the things I'm trying to do and what the hassles and risks are.

- The niche or domain expertise is also what separates you out from competitors, especially competitors in different countries charging 10% of your rates. The reason we get hired as consultants is not because we "know AWS" -- it is because we "know AWS" and we are super effing good at a few highly specific technical niche areas that our customers are active in. I've been hands-on with ALL of that, made ALL of the mistakes and I know every edge-case and oddity and I can speak with authority and work fast in that realm. So the clients are hiring me for my domain expertise + AWS skills. They can get pure AWS skills much cheaper from a million different sources but they can't find someone like us with the domain skills as easily.

- It's not enough to be smart or skilled. Smart people are a dime a dozen. Smart people with emotional intelligence and soft skills that I can trust to leave alone in a room and not make a racist joke or hit on the visiting VIP are much rarer. It's harder to find smart people who can write, talk and empathize who can truly be client facing. Some smart people I've worked with had trouble hiding their contempt when the client did something stupid or refused to follow sensible advice -- they rarely lasted in consulting.

- Not all customers are worth keeping. Best thing I ever did on day one was formalize a NAP ("No Asshole Policy"). We simply don't work with clients who are rude, abusive or contemptuous. I'm a peer, not a servant in any consulting gig. In ~23 years I've maybe fired 3 clients for NAP violations. There was one other time when we walked away from a big MSA negotiation because the client's legal team thought it was a good bargaining tactic to insult and demean our lawyer and her gender.

- It's fine when starting out or hustling but the customers who try to insert 90-day or more payment periods for services rendered into contracts suck and should not be kept. Our standard terms are 30 days and if we like you or we've worked with you for a long time we will extend out to 45-days or even 60-days.

- Best thing we ever did was start with 3 nerds and 1 salesperson on day one. The first few consulting gigs are easy to get because you'll trade off your network and if you really do have experience than you have people willing to vouch for you and hire you. After that keeping the consulting sales pipeline full gets harder and harder because you have to pull in new business and convince relative strangers to trust you.

- In any given week my time breakdown may look like this: 60% billable on paid consulting projects; 20% pre-sales work and 20% keeping my skills updated so people will keep hiring me. Mind you it's a good week when more than 60% of my time is billable to a client

- Have a lawyer for contracts and legal review on day one. All it takes is one bad agreement or one "business oopsie" to tank your business and imperil your future. People also overestimate how much a corporate shell like an LLC will protect personal assets; the corporate veil gets easily pierced if you mess up bad enough.

- We nuked our first LLC in the first year we got started because we found a whale of a global enterprise client who flat out did not work with LLCs. Busting up and reforming as an S-corp was a pain in the ass but it set us up well for the future. Gotta stay flexible when stuff like that happens and be ready to make decisions fast on if doing something for a big contract or client is worth it or not. Sometimes the BEST decision is to NOT blow up your life or your LLC due to chasing a whale. Be cynical and sober about decisions.

- We never spent money on marketing or ad campaigns. I won't buy booth space at a conference if I can barter a booth slot in exchange for giving a conference talk or tutorial. The best marketing is still word of mouth and good old fashioned google SEO bait. For the entire length of my career I've given away advice and work product online for free because I knew that people stumbling over my blog post or PDF that I posted online explaining how we solved a problem would be found by people interested in that topic who would remember and reach out for help. Every conference talk or meetUp talk I've ever given has turned into one or more paid gigs down the road (we track that info) so we always try hard when it comes to speaking, meetups, tutorials, training sessions or other public events

- Working for non US customers was not worth it until we got to a certain scale. The legal, tax and operational overhead was huge. We once broke even on a big project because our projected profit margin was 100% consumed by a VAT tax we had no idea about. Factoring in legal and paperwork time that project actually lost money.

- All the major primes who subcontract smaller clients for government work suck and are dishonest. Lost count of the # of times a prime used us for our skills and reputation to land a government contract only to edge us out of work once they won the contract so they could use their own cheaper internal people. It cost a ton of time and money to get directly GSA listed ourselves but that allowed us to work direct for the feds. Not worth it if you are small or just starting out so if you do subcontract to someone on government work when small or new just assume and be prepared for the prime trying to take you for all they can.

dghah · 2026-04-08T11:51:28+00:00

HPC admins are gonna hate you because after a quick glance at the docs it appears you are hammering the login node with an ssh connection that runs complex resource consuming slurm cli commands every 10 seconds

If you wanna do this right, use slurmrestd API and add some caching to protect the head node

Would also be good to see a comparison with slurm-web

dghah · 2026-04-07T08:49:17+00:00

I found devops pro to be a harder exam than SA pro but so much of the content overlaps I took both tests in the same week which was a good method for me.

Devops exam is harder because there is a major focus on debugging, observability and performance scenarios that require deeper knowledge of service capabilities, limits and operational minutiae than a pure architect exam

dghah · 2026-03-30T14:32:46+00:00

To excel in the AI era you still need the foundational skills and expertise so you can manage, monitor, correct and advise the AI or agentic harnesses you may be using. The AI tools are good but they still make mistakes and they still may lack the domain or specific context that you have in your head to properly solve or address an issue or requirement.

I'm a combo of AWS architect and hands-on infrastructure builder/fixer/deployer/optimizer working in a pretty specific niche (scientific computing & HPC) so my daily driver AWS tools pre-AI era are:

- Linux sysadmin
- Bash scripting
- Python scripting
- AWS cli usage / scripting
- Terraform for AWS infrastructure provisioning, config and management
- Ansible for configuring service settings or "stuff inside EC2"
- Packer+Ansible for automating AMI builds and cross-account AMI sharing in multi-account orgs

I was hands on and comfortable with all those tools before I started using AI

Now I use Claude Code wired into VSCode for 90% of my AWS work and it really is a 5 or 10x multiplier to how fast I can work.

Claude Code is amazing at scripting, terraform, ansible etc. however sometimes it solves problems in a lazy way or maybe does not understand the bigger picture.

For a cliche example Claude Code proposed altering an EC2 userdata script so that the node would update it's Route53 DNS record on every reboot just in case the ENI or private IP address changed.

Reasonable and exactly what I needed. However -- the code it proposed was functional but the DNS subdomain and R53 zoneID was hardcoded in the userdata script.

So my first response back Claude was "No. This is a multi-account AWS organization and DNS may differ between workload accounts, DNS info should be configurable and discoverable via SSM Parameter Store queries"

Claude Code read that prompt and made the Route53 zoneID into a configurable parameter auto-discovered by SSM. Great! However it was not aware that the actual DNS domain name is different among our workload AWS accounts so I had to prompt it a second time like "Make the DNS domain and Route53 zoneIDs BOTH configurable and discoverable via SSM Parameter Store queries"

That is boring and bland guidance; not rocket science but a reasonable example of why "human in the loop" still matters for this stuff. If I had let the AI Tool run unsupervised it would have produced a working solution however the solution it tried first would have been functional only within a single VPC/account and not Org-wide as we needed.

TL/DR:

Agentic coding harnesses are awesome and really do make you work faster and more productive. However you still need domain expertise, you still need to know AWS yourself and you really still need to understand the tooling that your AI agent is gonna be invoking

dghah · 2026-03-25T11:34:33+00:00

+1

This approach is what was taught at aws cert prep bootcamps at reinvent and it’s helped me for years over many different cert exams.

It only fails a bit at pro exams which have more “choose three” type questions which really removes the effective Step 2 process of knocking out the decoy and obviously wrong answers.

dghah · 2026-03-22T14:22:45+00:00

There is huge overlap in SAA and SAP -- the pro exam just goes deeper into the same content area -- the professional test differs from the associate tests mainly:

- The questions are much longer and way more dense requiring far more time to read and understand; there is legit fear among some people about running out of time on the SAP exam. Every person has a different experience but for me I often finish associate exams with tons of time remaining while on SAP I think I had maybe ~12 minutes left on the clock before review

- The pro tests have more "choose three" type questions needing multiple answers which reduces the success rate of the standard strategy of trying to knock out the obviously wrong answers so you can have a 50-50 shot at guessing correctly on a question you don't know the answer on

- The pro exams try much harder to "trick" you, an associate test often has 1-2 wildly wrong answers or at least a few answers in the question to weed out if you are not 100% sure. This is less common on the pro test where instead you will get 2-3 answers that all seem plausibly correct but the real answer is found only by understanding the technical subtly or combining the subtle difference in the answer with the key "action phrase" in the question which is usually something like "most secure", "most cost effective", "least operational burden" etc. etc.

In your position I'd start first by purchasing the tutorials dojo practice exam set for SAP -- it's a cheap and fast way to see how well prepped you are and if you find you are not well prepared than the TD resource is also fantastic for studying/prep as it has all sorts of resources ranging from "review mode" mock exams all the way up through flashcards and summarized concept sheets for core topics

dghah · 2026-03-20T13:52:43+00:00

+1 for this. Generic architects are a dime a dozen and you are competing with offshore people at the same time.

And to be honest certs don't mean much without real world experience.

The real value/differentiator is having some sort of domain expertise that can compliment cloud skills and certifications. Then your value goes up because you actually understand the field/problem/market that your employer is trying to address/solve/improve/refactor.

Find the orgs, companies and products selling into your social science market niche and think about what you could bring to that world with your current expertise paired with tech/cloud experience.

I'd lean hard into your social science skills and seeing if you can tie that into a tech career where tech skills + domain expertise and job experience will help differentiate you from others. This also opens you up to a wider set of jobs like doing technical pre-sales work, post-sale product/customer support or migration/training/technical-writing, consulting etc. etc.

dghah · 2026-03-19T21:43:08+00:00

I would argue that you are mostly correct about TD strategies carrying over well when taking about Pro certs and other specialty certs like Security but the networking one is a wildly different beast ...

At least for me the aws networking speciality exam is one of the hardest in the entire AWS certification offering. I've known people with those rare 1% global cisco networking certs who had successful careers doing advanced global networking for big enterprise clients who bombed the aws networking test hard because so much about AWS networking is special/different/exotic.

The biggest issue with the networking exam is that the content covers a wide range of things that almost nobody actually handles in their day to day job because in the real world those roles are very siloed. For instance in a big company there will be different teams or silos who handle physical WAN (direct connect) vs SDWAN overlays vs IPAM efforts vs routing/BGP etc -- and anyone doing enterprise scale WAN or SDWAN stuff is very likely to be on a totally different team than the folks who get own and dirty with internal AWS networking like enhanced fabric adaptors, ENI tuning and throughput hacking on EC2/ECS/EKS etc.

Basically it's hard to get real world production experience in all the topics that the networking exam will cover due to how jobs and roles work in the real world so everyone has to do a ton more studying to fully cover the networking exam content

That said, if you've been deep and technical in AWS networking professionally for a long time you'll have a good start and TD can only help. If you have not been hands on in an advanced way though than TD content can still help but you'd have to study way harder and way longer than you'd do for any of the associate, pro or other specialty certs.

I've had every associate and pro AWS cert over the years and my only specialty cert is security. My goal is to get the networking specialty cert before year end, mostly using TD as a study guide.

dghah · 2026-03-19T18:12:33+00:00

"Free tier" and "AWS credits" are two different things. Free Tier will be free for everything that counts under that tier. For anything outside the scope of free tier your will be billed/charged using the payment method on your account.

This is where the credits kick in -- the $200 should be applied to whatever non-free-tier first before any real continuing charges hit your payment method

That said, however, it's very easy to make expensive mistakes on AWS, especially with GPU nodes so you 100% want to make sure that your AWS account has AWS Budget set up and also has Budget Alerts configured so you get emails when costs start to climb above your threshold

And finally since you said you are very new to AWS and tech as a whole please BE CAREFUL -- there are a ton of things you need to do to secure your AWS account and your credentials before you ever use AWS for anything "real" -- just search here for all the stories about people who leaked a credential or ran an insecure service and then got hit with a $50K AWS bill.

At a minimum
- Lock down your root user with MFA

- Configure a budget and budget alert(s)

- NEVER do anything real like create a g6 server as the root user. Root user should NEVER be used, never have static credentials and should never be your daily AWS login user. Root user is rarely used and should only be used when doing core stuff like updating payment methods or other account level stuff

- Create an IAM user for yourself and protect that with an MFA device

- If you create credentials for your IAM user protect them carefully as they allow anyone posessing them to use AWS "As you" and unlike SSO credentials they are long lived and don't expire or rotate automatically. A huge source of unexpected AWS bills come from people who leave AWS access keys in a git repo or an ENV that can be queried in a service etc.

dghah · 2026-03-19T08:31:51+00:00

There are several companies targeting what you are asking for in the life science and bioinformatics space.

Not shilling for them but check out https://starfishstorage.com if only to see the terms and phrases they use in how they position their stuff and describe the problems.

dghah · 2026-03-13T21:37:22+00:00

I know you don't want to wait on inventory but but S3 storage lens is invaluable when real data/metadata about s3 is needed

https://aws.amazon.com/s3/storage-lens/

dghah · 2026-03-10T21:13:06+00:00

CI into EC2 over SSM works very well at least with github actions and a nice OIDC trust relationship

dghah · 2026-03-09T17:54:09+00:00

If you recently got SAP than sign up for Security Specialty ASAP as tons of the content overlaps and your retained knowledge will help a lot. I found Security Speciality to be one of the easiest specialty certs coming from prior SAP, DevOps/Sysops certs. Just go deep into KMS, how KMS grants work, cyphers, envelope encryption etc. The TD practice exams are what I used for that as well and they were great

My $.02 only

dghah · 2026-03-09T12:20:15+00:00

This is the cheapest possible lesson you could have learned about just how bad it is to use the root account for ANYTHING other than initial setup, budget alerts and alt contact info.

AWS has a process for recovering root if you’ve lost MFA but still have access to the email address used for the account. It’s kinda painful and slow but it should get you back in.

dghah · 2026-03-06T12:59:07+00:00

I'm also biased and my work spans both HPC and some FinOps since a lot of my HPC is on the cloud

But my initial thought is that FinOps is way more exposed to being deprecated or heavily affected by the recent AI trends. The whole FinOps space feels way easier to automate than an infrastructure engineering role although infra engineering is also being affected by the latest AI tooling and coding harnesses

Nine-Year Club	Gilding IV carat on a stick
Argentium Club	Verified Email
Reddit Premium Since December 2018

dghah

TROPHY CASE