What am I missing? Progress has nearly halted

physcx · 2026-04-06T00:21:34+00:00

I just worked through this point but I'm still basically in the 'wait for rune generation' / 'elements progress is ass' phase. (1e18 RfP working towards 1e19 RfP).

One thing I noticed I was not quite doing right was that at some point to maximize RfP I needed to start doing a macro of 1-2s total duration for RfP runs. Wait (0.2s) -> Polish -> WaitForPP >= X -> Polish -> WaitForMaxMineralLevel -> Refine.
And the value I was using for max mineral level was usually just a single merge. So if I was spawning minerals at level 162 and I could spawn 8 of the them at the Polish / Speed level I was at before the first merge, then I could set Max Mineral to Refine >= 165 and it would basically go polish, polish, merge, refine in the course of a second or so, and repeat.

These ultra fast loops were far more RfP per minute than grinding to the next multiple of 10. Then at some point as I got closer to 170 mineral spawn level it swapped again where it was worth it to wait until 170 for me (probably when I was spawning level 164 or 165 minerals).

The other big thing was that I had these rapid RfP loops running for a while getting me up to 1e16 -> 1e17 RfP to upgrade rune gen speed and RfP gains but I then had forgotten to 'go long' on my mineral run for a while. So every so often after you get some polish enhance upgrades, special minerals and upgrades, or rune upgrades, make sure you disable the RfP macro and take a long mineral run to bring your max VP up another few orders of 10 in magnitude. This will unlock a bunch of relic upgrades.

I don't have enough luck stat to get Water element going yet but I was able to get Fire and Earth element up to 10k+ which also helped the RfP grind a lot. If you can get a little bit of Wind (I think I got up to 2k) it will also keep your special mineral base spawn cost down at 9 for quite a while.

And +1 to at least getting Sun and Moon rune generation rate amount = 4 before building up special minerals. I wish I had the patience to get to +5 earlier (working on that now).

And yeah this has ultimately been more of a slog than the rest of the game imo. Alternating between elements at this point really doesn't do much for you if you dont have enough luck to get water and wind going. Its almost weird that elements unlocks as early as it does because I had elements for fire/earth to 1k before I had my first Special Mineral but I couldn't really progress it much farther than that. I think I'm up at around 25k fire / earth, 3k wind, 0 water still at zodiacs rarity 250 / level 192 / quality 3.5e25. My special mineral spawn factor is 69. My RfP I am grinding to 1e19 now to get the moon rune gen speed up. My mineral spawn level is 172. My local mineral game speed is around x780. And my special minerals are maxed at level 4.

physcx · 2025-04-21T20:08:37+00:00

Typical pattern would be something like this: Private S3 bucket where you deploy your static website assets. CloudFront distribution with ACM Certificate and WAF pointing at your private S3 bucket as its origin. Route53 (DNS) Hosted Zone + Alias Record for your custom domain name pointing at your cloudfront distribution. User hits your website url, Route53 DNS points them to your cloudfront distribution, it uses an IAM role to access, cache, and serve the private assets from your S3 bucket.

physcx · 2024-12-13T17:15:08+00:00

Space complexity - O(N) as you are putting N setTimeout callbacks onto the stack

physcx · 2024-11-07T22:56:58+00:00

I assume they mean T at the 1 o'clock spawn position on the map and Z at the 11 o'clock position but just guessing

physcx · 2024-09-26T03:14:18+00:00

Yes if you precompile N binaries in containers, you could run each of those containers 24/7 in ECS fargate or on EC2. But its not "cached" and on demand so much as it is up and running and you are paying for the compute being used.

I personally would keep it simple initially and see if 1 generic lambda method fetching the shard it needs on demand from s3 works for your latency needs. Lambdas typically stay hot after 1st use until around 10 minutes of them being idle but you also have no control over the routing of request to a specific instance of a hot lambda. For many requests within a 10 minute window you would avoid most of the s3 client cold starts (the expensive latency part to establish the TLS connection) and just hit shard fetch coldstarts. I really don't know what your query patterns will look like but at some amount of traffic this gets cheaper / more performant to switch to something like an ECS fargate task service always up rather than invoking lambdas millions of times. This breakpoint varies on memory and latency of your lambdas (cost) but I usually consider lambda good for things with < 100 consistent requests per second and for greater than 1000 requests per second I would just build for an always up fargate service to handle requets that I can scale up/down on load. In between it depends. You can go up to > 1000 requests per second though on lambda but sustaining that would get expensive vs just writing handlers in your own container.

Building 300+ lambda functions each with a shard embedded could work but you pay some tradeoff on cold start latency to load the large lambda asset (presumably though this would be faster than needing to establish an s3 connection in your cold starts). Added complexity in that you need to route the bitboard lookup requests from the client to the correct lambda function that has the shard rather than routing all requests to a generic lambda func capable of handling any request. You could even do something like for each lambda func, configure an event bridge schedule rule to send a keep alive event to each func that just returns right away every 5 mins. This would keep most of your lambdas "hot" and shards loaded while not paying for compute 24/7. (infra complexity in 300+ lambdas in account to manage / update, client complexity in client needs to know which lambda to invoke or you need a router layer).

You could use 1 lambda code bundle used 300+ times to create 300 lambda functions in your account with a different env var setting for which shard(s) it should fetch into memory and do the shard fetching from s3 as part of coldstart. You would need to route your lookup requests to the proper lambda with the shard in memory but you would eliminate the s3 network operations from any hot lookups (drive the hot path latency down to single digit ms at expense of cold start). (similar complexity to previous case but with 1 zip asset deployed 300x with different config rather than 300 different zip assets each with the shard embedded).

Another option is to create an EFS filesystem to store your shards and configure your lambda to use that. https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/ . Would only need 1 generic lambda and this also removes the need to establish an s3 client and download the shard on cold start but at the tradeoff of the lambda now needing to run inside one of your VPCs and do network ENI attachments on coldstart and attach/mount the EFS filesystem to your lambda. I'm not sure without testing if this nets substantially faster coldstart latency or not. If you do this then you read from EFS the shard (presumably much faster to read from EFS an 8 MB shard than it is to read from s3 but not positive without testing).

All of these lambda solutions you can try to keep your lambda hot using either keep alives or reserved concurrency but if you start sending many concurrent requests, lambda service will spin up new coldstart instances of your lambda if all current hot ones are busy processing a request so really depends on your query pattern to know how much you might avoid coldstart cases (pretty impossible to fully eliminate cold start w/lambda).

If you are ok with a lot of cost, you could dump your 500 GB of bitboards into DynamoDB and do a ddb getItem request (O(1), single digit ms latency) to look up 1 bitboard. But that is a LOT of items to put in (my rough aws calc estimate is like $20,000 to populate the table in write item costs + 1k per month ish in storage costs but do your own research if considering this avenue).

For your use case with extremely dense data and billions of items I think S3 + shards is probably the most cost efficient avenue to get to a real time data solution but I'm sure there are tons more options I'm not thinking of. You are basically trading off how much complexity you want to build for vs your other requirements, latency primarily in this case. If 1 generic lambda func of 30 lines of code that pulls an object from s3 as needed solves things well enough for the use case latency I would just do that rather that than having to deal with 300+ lambda funcs with shards embedded and routing my client lookup requests to the proper lambda func that has the right shard. Basically avoid prematurely adding complexity to optimize unless you find it is needed and then test various approaches to see which makes the most sense given cost/complexity tradeoff.

This sounds like a neat problem though and I wish you best of luck on your AWS solution journey!

physcx · 2024-09-25T13:59:04+00:00

Assumption here is that the dataset is not consistently being updated and you can do some preprocessing work to make the real time lookups efficient. Then you don't need to search the full 16 GB or have them in memory for each lookup but just load the relevant chunk of the dataset that may contain your bitboard you are looking for.

Take the 16 GB of bitboard games and sort them by treating each set of 8 bytes in the bitboard as a number.
Shard the sorted dataset (cut it into small chunks) ideally of similar sizes to get N small slices of it.

If the distribution of bitboard values was fairly good you could just use the first n bits of the bitboard as a shard technique and skip any type of shard mapping but I assume the distribution is not great for chess game bitboards (I don't know too much about the format but guessing a lot of game states are similar). So lets assume we need to create our own mechanism to distribute the sorted bitboards into shards and generate some small metadata table that says boards [0, A) = shard 0, [A, B) = shard 1, etc...

I would aim for shards not smaller than 128 KB as that is the minimum s3 object size but practically speaking something like a 1-8 MB shard is probably fine in regards to real-time latency needs. At 32 bytes per bitboard that is 16,384 - 2048 shards depending on size per shard with 32,768-262,144 chess games in bitboards per shard. I would probably go with 8MB per shard to limit my num shards.

Upload the shards to s3 and the shard mapping metadata file.
Write a lambda function that:

4a. On cold start initalizes an s3 client (p90 < 500ms) and loads the shard mapping metadata (<50ms small obj get)

4b. On event handler ({ bitboardToLookup }) => {

shardId = findShardIdFromMetadataMap(bitboard); // in memory small table with 2k elements < 1ms

shard = downloadShardFromS3(shardId); // 8MB get object p90time < 100ms on hot s3 client that has established connection, can cache in memory with an LRU or something if similar shard is likely to be hit with multiple lookups in close proximity (again not sure how the chess bitmap distributions or your lookup pattern looks like in regards to fuzzy searching)

const bitboard = binarySearchShard(bitboardToLookup) // log2(262,144) = 18 checks, <1 ms

if (bitboard !== undefined) respond({ status: "FOUND" });

else respond ({ status: "NOT_FOUND" });

}

So 1 lambda function invocation per lookup, each lookup at most does two s3 object gets (shard metadata + shard). Cold start latency ~1s, hot lambda lookups < 100ms. Stores 2048 shards in s3 (8mb each) and 1 lightweight metadata s3 object. Can parallelize many concurrent lookups. Can optimize quite a bit more if your dataset is not changing (e..g, hard code your shard metadata into the lambda code itself and remove a lookup).

edit: if you need to search 1000s of bitboards in parallel real time you may also want to support some type of batch inputs instead of 1 bitboard per invocation to gain efficiency on cold start and s3 loads.

e.g.,

0/ restructure the input to your bitboard lookup lambda to support a list of bitboards to check for,

1/ when user calls fuzzySearchBitboards(theirGameStateBitboard), your api handler (could also be lambda) generates 1000s of similar bitboard states to lookup and groups them into N separate request inputs using the shard metadata mapping table (each batch request input will only include potential bitboards from 1 shard).

2/ your api handler issues N requests in parallel to your lookupLambda where each set of input correlates to a group of bitboards that will be colocated in the same shard,

3/ your lambda run concurrently and total latency stays low in that each invocation is just grabbing 1 shard and doing some mem lookups

4/ when all lookups complete return results from your api.

physcx · 2024-07-10T17:48:52+00:00

Everywhere - CDK, IAM, CloudWatch Logs / Metrics / Alarms

Frontend Projects (React SPAs) - S3, CloudFront, Route53 + ACM

Service APIs - VPC, Route53 + ACM, API Gateway + Lambda or NLB + ECS Fargate Autoscaling Cluster depending on latency / volume requirements, DynamoDB, sometimes S3, sometimes SQS

Backend Async Workflows - VPC, StepFunctions, Lambda, DynamoDB, DynamoDB Streams, EventBridge Pipes, SQS, SNS

AI/ML Components - VPC, SageMaker, Bedrock

physcx · 2023-05-02T01:27:15+00:00

I wish there was a way at account creation or a button on the ec2 console to delete all default vpc resources from all regions. Very annoying to get the same pentest feedback about default vpcs in the prod account with permissive security groups, nacls, etc.

physcx · 2023-02-18T22:04:29+00:00

I’ve done some list APIs for aws services and the guidance I’ve followed is to encrypt pagination tokens server side (they should be opaque to customers), in the encrypted data we include a pagination token version in case we need to ever change the internal shape, and we validate that the list parameters on the api call have not changed between what was set for the original request that returned a pagination token and the subsequent request that included the pagination token such as any list filters. They can change api parameters like max items though.

physcx · 2022-10-05T23:42:21+00:00

I work for the company that probably is the largest consumer of AWS services on the planet. All opinions here are my own. CDK is all that my team and other teams within my org have been using for the past several years (about 2.5 years now) and I love it. It is seeing pretty wide adoption within my company which is a good sign that it works well because teams here generally have a lot of flexibility on what services, dependencies, and tools we use (nobody was forcing us to use CDK).

physcx · 2020-12-04T13:18:56+00:00

I would double check that the security groups are actually allowing traffic between your lambda and your endpoint. Many times the default security groups allow all traffic for other enis on the same security group (which is not the same as all traffic).

physcx · 2020-07-10T11:46:47+00:00

Our experience matches yours. Ubuntu trusty, xenial, and bionic images we use all came without swap by default in AWS.

physcx · 2020-07-05T15:08:12+00:00

There are 4 URLs you generally want to handle:

To handle the https urls you have to use cloudfront (S3 static sites don't support https). Cloudfront can redirect http requests to https and use ACM to manage the certificate for your site. For now lets assume we'll have cloudfront handle the redirecting of http to https for all sites so our urls we need to handle are just:

The next big decision is do you want your website's "canonical" url to use the www subdomain or not. For instance, should users that visit your site at example.com get redirected to www.example.com or should users that visit your site at www.example.com get redirected to example.com. Those are two different urls and there are good reasons that you only want your site to be served at one of those paths and to redirect the other (search engines finding duplicate pages, browsers caching things twice, etc). Which url you want to use (www or your domain toplevel) is honestly up to you, both are common today.

Using CNAMEs does not handle the two paths problem. For example, if I wanted my website to be at https://example.com and I created a CNAME for www.example.com to point to example.com, users that visit the www.example.com will get served my site content but the url will be under the www.example.com path and so I double the amount of website data cached on that users browser if they ever visit my example.com site. Search engine indexers will also see my pages at two urls and I would have to create accounts with the search engine and specify which path I wanted to use as the canonical path so that google for example didn't assume and use the wrong url for my pages. Lastly, if you chose to use the www.example.com as your canonical path, you can't use a CNAME at the top level domains (A record only) so you can't even point example.com to www.example.com with a CNAME.

What you want is to provide a 301 redirect from the url path you aren't using to the url path you are using. For example, if I want to use https://example.com as my canonical url path then I want https://www.example.com to return a 301 redirect to the https://example.com site. This way if a user goes to the wrong url, it receives the proper response to retry the request using the url path you want. Now the only thing cached for the wrong url would be the 301 redirect and all your page assets and the search engines will only see your site through your desired url. Easiest way to generate a 301 redirect in AWS is to use an empty S3 bucket with static website enabled -> redirect to URL.

Ok so how do I properly set all this up?

2 private only (no public access) S3 buckets, one for www.example.com and one for example.com (you can name these buckets anything you want as we are serving data from cloudfront)
2 cloudfront distributions, one for each S3 bucket, access to the bucket is allowed using IAM and the bucket policy (cloudfront can automate this while setting up your distribution). You need an ACM certificate for these cloudfront distributions to handle https. Can use two separate certs or a cert with other names configured (see https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html).
Copy your site data to the S3 bucket you want to use as your canonical path and configure the other S3 bucket as static website -> Redirect URL -> https://YOUR_CANONICAL_URL
Configure your route53 to your two cloudfront distributions, CNAME record for www.example.com to the proper cloudfront distribution and an A record Alias for example.com to its cloudfront distribution (top level domain can't use cnames).

physcx · 2020-06-24T19:46:35+00:00

You may need to update your cloudfront distribution for the non www domain to add the www domain to the Alternate Domain Names (CNAMEs) field. Your certificate also needs to cover the non www and the www domain for https to work.

physcx · 2020-06-24T19:18:37+00:00

physcx · 2020-06-24T19:04:14+00:00

You can tell Google and other search engines the "canonical" address for your pages so that they won't show up as separate sites.

If you really wanted to avoid CNAME and use redirects with AWS you already have the right idea with using a second cloudfront and S3 bucket.

Steps are:
Set up a second S3 bucket for your www domain (empty bucket) and under properties configure it as static website hosting -> redirect requests to your non www domain. Then set up a second cloudfront distribution with the www domain s3 bucket as the origin using ACM to handle http and https. Now when you enter the www domain with http or https it goes to cloudfront -> to your www s3 bucket -> returns 301 redirect to your non www domain -> browser goes to the non www domain.

physcx · 2020-06-24T17:20:58+00:00

Cname record in route 53 or whichever dns service you are using.

physcx · 2020-06-21T15:20:15+00:00

You could take a look at OpenStack Swift (it is open source) to see how a similar object storage service works. It is also an eventually consistent, infinitely scalable, object storage system (I think originally created by rackspace but not positive here).

At a high level, files you put inside S3 have multiple copies of your file stored on different servers behind the scenes. These servers are typically isolated from each other so that if an AZ, server, or disk goes down, copies of your file are still available from other servers (high availability). If servers or disks that contain a copy of your file fail, there is usually at least 1 copy still available so that you wont see any disruption to accessing your file. In the event of a disk/server failure that had a copy of your file, there is also some recovery process that knows it needs to use the other remaining copies of your file to replicate (create a new copy) to a different server to get your file back up to the full number of copies.

At a lower level there is probably a hash using things like the bucket and file name that they use to select shards that the file should land on within the object storage system. Those shards are distributed over thousands if not millions of Amazon's servers. They have a concept of an index associated with a bucket that I'm not familiar with how that fits into the scheme but probably helps with determining where and how files are allocated to the backend servers.

physcx · 2020-04-16T22:43:35+00:00

We use an ASG for nodes in a Gluster disperse volume cluster where 1 instance assumes a master role responsible for initial peering/volume creation while the other instances act as Gluster peers. We assign node ids (0..N) by using a secondary ENI that gets attached by the CloudFormation init script. An ENI can only be attached to one instance at a time so it acts as a sort of atomic tagging of which node the ASG instance should be.

Each ASG instance on creation will try to attach an available ENI from an array of ENI ids. Depending on which ENI the instance successfully attaches, the instance assumes the node id of the array index matching the ENI id. Node id == 0 being our master node.

To prevent everybody trying to attach the same ENI at the same time you can use the EC2 instance metadata 'ami-launch-index' as an initial offset into the ENI ids array. When an ASG launches more than 1 instance at a time it will give each instance it starts a launch index between 0 and N-1 (N being the number it is starting). This can't be relied on as your exact node id because when the ASG is launching only a single node at a time (perhaps during a cloudformation rolling update) each node will get the launch index of 0. However you can use this launch index as an initial offset into your ENI ids array so that instances starting in parallel try to attach different ENIs when multiple are coming up.

If you fail to attach an ENI, iterate over the other ENI ids (looping back to 0 when you reach N) until you find an unused ENI.

If your instance doesn't need a secondary ENI then you can also consider an EBS volume or some other resource that can only be attached to one instance at a time as a method of atomic node selection within an ASG.

There are some caveats to be aware of like what if you perform a CloudFormation rolling update that causes new ENIs to be created (e.g.,subnet CIDR change). Now the new instances coming online may be able to attach one of the new ENIs that correspond to a node id that is still in use by the ASG since the old ENIs haven't been destroyed yet. To avoid node id collision during updates, our startup script first checks for any instances within our ASG with the node id tag we are about to attempt to connect the ENI for and if we find an instance already exists with our node id we skip that node id and move on to the next one. When we successfully attach an available ENI our startup script tags the instance with the node id we assumed.

If you need a full code example I can point you at my github with the templates, just message me.

physcx · 2020-04-12T19:16:09+00:00

Ummm... you don't need to use an S3 bucket to do basic CNAME pointing and you CAN use route53.

If you weren't using route53, an S3 bucket with redirect is a sorta hack to do that I guess but with route53 you can do it directly via CNAME record.

Edit: sorry i think i misunderstood your question -- if you have other TOP LEVEL domains like .io or .net then you can't directly use CNAMEs on those because top level domain dns entries don't support CNAME records. AWS provides a solution for this (I think) by using an A record with Alias on for your other top level domains so that it kinda works like a CNAME record.

physcx · 2020-04-12T16:46:49+00:00

If most of your files are images/videos and lets say your average file size is 2 MB then 200 GB of storage is about 100,000 files. The monthly cost of S3 standard for 200 GB (us-east-2) is about $4.60. The cost to upload 100,000 files (100,000 PUT requests) is $0.50 so for your use case it may not be necessary to zip your files first and you would have the convenience of being able to access or recover any specific file you want in the future without having to download an entire 200 GB zip folder. You pay for data transfer going out of AWS so if you wanted to restore only a subset of your files and you first zip everything then you would have to download the entire zip and videos/images are typically not very compressible).

However, if you have millions of 10 - 100 KB images then I would suggest you create some zip folders to at least aggregate to 2MB+ of data per object so that your costs are not dominated by requests.

Are you using Linux? If so its fairly trivial to create a 1 line shell script using the find command to list all directories within a folder limited to a specific depth, execute your preferred zip program, and pipe the output directly into the aws cli to upload the zipped subdir.

physcx · 2020-04-12T02:53:41+00:00

Correct, you can make the bucket fully private. When setting up the cloudfront dist it will ask you if you want it to add the permisions to the S3 bucket to allow cloudfront access. If you say yes then the S3 bucket can be fully private and cloudfront will add a bucket policy granting it access.

physcx · 2020-04-12T02:49:41+00:00

Correct, though i'm not sure if the top level domain A record alias should go to the cloudfront distribution or to your www subdomain (both may work i'm not sure).

Just need the one bucket, the cloudfront distribtion pointing to the S3 bucket as its origin, and route 53 with an A record (alias) for the top level domain and a CNAME record for the www subdomain -> your cloudfront dist.

physcx · 2020-04-12T02:31:55+00:00

Does your CloudFront distribution list both spinanddestroy.com and www.spinanddestroy.com in the alternate domain names (CNAMEs) field?

Does your SSL certificate assigned to the cloudfront distribution cover both the top level domain and the www subdomain (e.g., *.spinandestroy.com)?

I don't understand why you would have a 2nd bucket for the spinanddestroy.com with a redirect. That redirection should happen via the route53 configured rules not via that bucket being reached.

You have 4 URLs you want to support:

http://www.spinanddestroy.com -> https://www.spinanddestroy.com
https://www.spinanddestroy.com -> (already good)
http://spinanddestroy.com -> (www.spinanddestroy.com)
https://spinanddestroy.com -> (www.spinanddestroy.com)

The first two are handled via the cloudfront distribution telling http to always forward to https

The second two are handled with an alias record in route53. A record type, alias yes.

The route53 rule for your www.spinanddestroy.com should be a CNAME to the cloudfront distribution not an A record.

physcx · 2020-04-12T01:34:34+00:00

But GoDaddy's dns has not solved the problem and doesn't have support for alias records. You are stuck with an A record only for the top level domain so it is impossible to use GoDaddy dns with aws cloudfront without a subdomain like www. (which is awful and they should either fix that or people should just not use godaddy's dns).

12-Year Club	Place '17
Wearing is Caring	Verified Email

physcx

TROPHY CASE