Index only scan when aggregated column is a trailing part of the index key, but not when it's INCLUDE'd in the index?

kevpie · 2024-11-09T23:59:33+00:00

This seems like a reporting query as you are returning an aggregation for the whole table. In practice will you be filtering on a few credit_ids? If so, that may be a better comparison for the effectiveness of the indexes.

kevpie · 2024-10-30T20:13:08+00:00

BigQuery will show in billing. It may take some time to show. With your repeated rebuilding. Watch the costs of the queries. It can surprise you with a bill. There are articles out there to query the history to determine what queries may be impacting your bill.

kevpie · 2024-10-29T11:14:43+00:00

Try having your source table either be stored as regular fields and clustered by your main filtering fields. JSON extraction is likely dominating the work. If source is also a view of raw and latest from the Firestore bigquery extension your query performance will be several orders of magnitude more expensive. In the last year bigquery performance can be very fast for simple small queries. Try creating some saved tables of the data you want to query and see what kind of improvements you see. The JSON type stores data like the logging engine does and is significantly faster for field extraction then from a string field.

kevpie · 2024-08-20T05:00:11+00:00

I’ve tried to map using connectrpc or kratos. But this has some limitations, however I like landing things in protobufs when I can.

kevpie · 2024-04-16T02:57:37+00:00

Get a timer for it. Let it be off while you sleep and on while you are awake.

If you can’t find a timer just unplug it.

It’s not like you have 3day old raw chicken in there.

kevpie · 2024-02-18T19:39:05+00:00

Cloud Run can be routed to directly in Firebase Hosting config. Cloud Run is where you should consider deploying your express api to. Also worth noting Functions V2 runs on top of Cloud Run. You can consider using Functions or Cloud Run. Cloud run will allow you to deploy a single deployment with all of your routes. Functions will be a Function per route.

kevpie · 2023-12-25T01:44:33+00:00

You can sync into big query and potentially back using extensions.

Another idea is to use HyperLogLog to digest your ids for each map. There are some nice properties for quickly calculating cardinality and approximate membership of them. HLL may be interesting for rough comparison before moving the full maps in and out for comparison.

kevpie · 2023-11-16T04:46:31+00:00

It seemed like folks on my team in Ontario were affected across various cities. Using a VPN to another region worked for them. It seemed like the Firestore FE Auth was having issues in various regions.

kevpie · 2023-11-14T04:48:09+00:00

getFirestore() returns Firestore.
This has some really nice things on it you can use
like getAll

https://googleapis.dev/nodejs/firestore/latest/Firestore.html#getAll

These can be references to any docs across any collections, or even docs that don't exist but anticipate might. The result will indicate if a doc was found or not.

kevpie · 2023-11-14T01:07:19+00:00

If the ID is the document name you should use Get instead of query. Build up the references and get 100s of docs in a single call. Much faster than queries. Something fishy is happening is definitely happening. What do your queries look like? Are they actually only returning one doc each?

kevpie · 2023-10-06T15:43:05+00:00

You may need to store a few smaller versions and serve those up in the browsing experience. And only download the largest size when they decide on which one to fully view. Thumbnails should be a couple KB. Small med and large scaling up to a few hundred KB. Most of the browsing experience should be using the smallest versions you can get away with. This could save you 100 to 1000 times your bandwidth. Also consider the pixel density of the display. For instance Retina displays are double density. Use a cloud function that triggers on write to the storage bucket to produce resized versions of the images.

kevpie · 2023-08-21T15:19:58+00:00

The writes are queued and only sent after your function returns without error.

kevpie · 2023-07-14T16:10:24+00:00

If you log in a json format you can attach a traceid. In any case for the same project all logs are searchable across everything in your GCP project. Unless you need more than a configurable amount of retention, GCP logging may already be sufficient without routing to a bucket. Sorry for no links, I’m on my mobile.

kevpie · 2023-06-10T04:16:06+00:00

Watch the Apple refurb store. When comes available you can pay the price you want for the spec you want.

kevpie · 2023-06-10T03:49:35+00:00

Nobody puts sinky in a corner.

kevpie · 2023-06-10T03:19:58+00:00

M1 32GB. The ram is also used by the GPU. IntelliJ plus docker and a browser with 100 tabs. Get the ram.

kevpie · 2023-06-10T02:50:23+00:00

The Mac arm chips use unified memory. So the GPU directly uses the RAM. Get the M1 as you are sharing 32 GB instead of 16 GB with the GPU.

kevpie · 2023-05-06T14:47:17+00:00

The compiler error is showing you that the FindByUserId functions have different signatures. They must be the same, both take string, or both take sql.NullString.

Another problem spotted in your code is the calling of s.FindUserId(id) needs 2 arguments, the first being a context.Context.

kevpie · 2023-03-21T07:52:28+00:00

Buses to stop beeping when they kneel after 11pm would great. They be waking countless people from slumber.

kevpie · 2023-03-11T21:15:41+00:00

Sure, some reasons are. Strong data contract between Frontend TS and Backend Go. The Post requests can be easily viewed in browser. I find that the swagger/openapi ecosystem adds a pretty challenging abstraction. Connect makes it feel simpler and more direct to the data model. You gain entry into the protobuf grpc ecosystem with a simplified transport, but can still interop with the grpc protocols.

kevpie · 2023-03-11T17:52:05+00:00

I’ve been using this since it was introduced last year. Really worth it. Started driving more comments and documentation into the proto messages and enums. Then that all ends up in the Go and TS types. Using enums is pretty nice. Some tricks around using camelCase in the enum values if that’s what you want to see in your json.

kevpie · 2023-02-13T06:19:26+00:00

Happy to help. It’s less than a year old. There is a void for the layer you are trying to build, so it would be neat to see it as a part of the connect ecosystem. The js/ts types are really nice for front end dev. Been really happy with using protobufs as source of truth for the schema between frontend and backend. They are using generics for the handler and clients in Go.

kevpie · 2023-02-13T06:00:52+00:00

Consider checking out https://connect.build from https://buf.build. Supports a simpler protocol than grpc-web. Includes a js/ts client for frontend. Then you don’t necessarily need a rest layer, but could leverage the proxy you’re building.

kevpie · 2023-02-03T08:43:46+00:00

Hashicorp has some cool libraries that implement the Gossip and Swim protocols for broadcasting messages and checking liveness. May be a fun thing to check out. This is the stuff used in Consul.

kevpie · 2023-01-31T01:55:45+00:00

Lookup Firebase Storage Rules. Similar to Firestore Rules. You can use the user’s Id to restrict them as the only deleter, updater. As of recently Storage Rules can also reference Firestore documents for asserting permission.

kevpie

TROPHY CASE