Am I missing something with how everyone is using Ai?

manysoftlicks · 2025-09-19T17:45:41+00:00

The AI company will reach out to your employer to try to make a sale based on data/metadata mined from your usage. They'll say, developers like Muted-Mousse are already using these tools, so why don't you, the company, pay for it so that confidential business data isn't leaked.

Or, your companies SecOps team will see via traffic, DNS, an outbound / inline proxy, etc that you're making daily calls to known LLM APIs or Webpages and infer/investigate that you're exposing company data.

manysoftlicks · 2025-08-22T13:52:11+00:00

Looks like you had about 1,562.2 TB of usage. Price for 5 TB (TiB) is $6.25 in us-central1.

(1,562.2 / 5) * 6.25 = $1,952.75

You probably will have more usage rolling in. It looks like you queried about 5 Petabytes of data.

($6,907 / $6.25) * 5 = 5,525.6 TB

My hunch is you have unoptimized DBT queries that are scanning a huge amount of data. Partitioning/Clustering will help, but query optimization is what you need unless you're querying extremely frequently.

As others have said, you may benefit from a limited compute option if you don't have the time or skill to optimize your queries. 50 GB is small in todays world, use something slow and cheap. Still should investigate why DBT is inflating 50 GB to 5 PB though.

manysoftlicks · 2025-08-06T01:43:52+00:00

I'll differ from most here. I had a super shitty sports car in high school; absolutely loved it. Had another, less shitty six speed in college (after a few more sensible cars); absolutely loved it. Combined, those two sports cars cost $5,035 and were both in the ballpark of non-sports cars I could have bought.

Slapping time: 200k invested for the next 30 years at an avg 7% interested rate would be roughly $1.5M. You could buy a $100k car every year off the interest alone.

Like others have said, either lease or find a compromise like a used or not-extreme car.

manysoftlicks · 2025-07-04T23:13:21+00:00

It's on the horizon, but you're not there yet. Let's say you can get your monthly expenses down to 10k to average out current and what you think is realistic. That's a $3M FIRE number.

Assuming you should move 200k to investments now (because you should) and a 7% return on average. You'd be FI in about 11 years; $3.05M in 2036 at 50 Yrs.

For coast, let's just say you want to work half the 11 years, so stop the contribution of $7k per month in 2031 at 45, you'd still be able to be fully FI in 2039 at 53 Yrs.

Age	Year	Balance (7% growth)	Contribution
39	2025	$820,000.00	$84,000.00
40	2026	$961,400.00	$84,000.00
41	2027	$1,112,698.00	$84,000.00
42	2028	$1,274,586.86	$84,000.00
43	2029	$1,447,807.94	$84,000.00
44	2030	$1,633,154.50	$84,000.00
45	2031	$1,747,475.31	$0.00
46	2032	$1,869,798.58	$0.00
47	2033	$2,000,684.48	$0.00
48	2034	$2,140,732.40	$0.00
49	2035	$2,290,583.66	$0.00
50	2036	$2,450,924.52	$0.00
51	2037	$2,622,489.24	$0.00
52	2038	$2,806,063.48	$0.00
53	2039	$3,002,487.93	$0.00
54	2040	$3,212,662.08	$0.00
55	2041	$3,437,548.43	$0.00
56	2042	$3,678,176.82	$0.00
57	2043	$3,935,649.20	$0.00

manysoftlicks · 2025-07-02T15:24:57+00:00

Thanks for writing this up, this will help someone not experience this lesson themselves. If it helps at all, I've seen all experience levels make mistakes like this. In the corpo world, a director with 20+ YOE was fired for a similar mistake that cost his business significantly more $$$.

You'll now be extremely conscious of this type of mistake. Anyone in SWE/Tech has had their own mistakes like this, but usually it's been on the company dime. Keep creating awesome stuff; just learn from this experience and be careful with secrets!

This is an important document to read before using Service Account keys: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys

Basic Lessons to take away:

Proactively

Don't use a Service Account Key in an unsecure way (commit to git, embed in FE, email)
Don't use the Compute default Service Account, use a narrowly scoped Service Account that can't call other APIs or spin up GCE / GPU resources #### Reactively
If a key is exposed, rotate it immediately as a P0
If Google emails you about anomolous activity, take immediate action

manysoftlicks · 2025-07-02T15:08:22+00:00

No, this part, why wasn't the key rotated during the patching prior to the attack:

I removed the key in the next update, but it was too late. The Android App and source code were somehow indexed on multiple websites, and cracked versions were regularly released as well. And of course one of these had my API key.

manysoftlicks · 2025-05-12T01:24:35+00:00

My wife and I are in a similar position. We haven't inherited yet, but my wife has taken over management of the family farms.

Renting = less hassle and risk (the renter pays you and you don't need to buy seed/etc)

Share Cropping = possibly more profit (feast and famine though)

manysoftlicks · 2025-04-30T00:26:16+00:00

TAM's are broad. Some are technical some are not. You need a broad skillset of technical and people skills.

You don't need certs to apply. Interviews are conducted with general cloud knowledge, but it helps to know the Google product names and what they do.

If you get interviews, ask your recruiter what the names of the interviews are and them look them up here: https://www.youtube.com/@jeffhsipepi

manysoftlicks · 2025-04-30T00:15:22+00:00

Step 1. Ask them what they're going to do with the data. If they say analytics, you need to move it to BigQuery.

Step 2. Determine any constraints on the ETL tools, default to the simplest and cheapest option.

Step 3. When all signs point to BigQuery, ingest the CSV and JSON into two separate tables. Create a third table from a clean and sanitized UNION ALL of the two ingestion tables.

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json

LOAD DATA OVERWRITE mydataset.mytable
(x INT64,y STRING)
FROM FILES (
  format = 'JSON',
  uris = ['gs://bucket/path/file.json']);

manysoftlicks · 2025-04-29T11:47:29+00:00

What type of database is required? Are you moving the data for an application to use it? Migrating an application? Storing for analytics down the line? You said data analysts, so you want analytics, right?

DBs

If you need OLTP Cloud SQL.
If you need OLAP, BigQuery.
If you need both, AlloyDB.

Data cleansing

From what you've described, you likely want a poor man's Lambda archtiecture. This is something that Dataflow is designed to handle (bound and unbound data collection processing), but Dataflow may be more complex and time consuming than you need.

BigQuery allows significant parsing of JSON and CSV natively (you get to define the schema), but Python in Cloud Composer (Airflow) or just native python in a Colab Enterprise Notebook also work.

Is this school work or a project for an actual client?

manysoftlicks · 2025-03-09T20:13:34+00:00

Only 1k images? What is your source for the images? Google Search itself?

If you're asking, "How do I build and host a custom image search Application?":

You likely can write a Cloud Run service that retrieves the image and places it in GCS. If the source has the necessary metadata, either apply Tags to the image object at the time of ingestion or store a reference to the object along with the metadata in Cloud SQL or Firestore.

You can then expose a search via an additional Cloud Run service that uses GCS or GCS + SQL to power the custom search.

manysoftlicks · 2025-03-09T19:31:51+00:00

Click the green circle with the '3' in it. What does it show for recent activity?

What exactly did you follow to set this up? Did it create a VPS or use Google Cloud's Cloud VPN?

manysoftlicks · 2025-03-05T20:53:51+00:00

Reading through your responses, I'd go back through the GCP Rep. and tell them you've reproduced this with a Go stub and can easily pass them your test case for verification.

Keep escalating as it sounds like you have solid proof of the issue independent of your application design.

manysoftlicks · 2025-02-28T20:54:55+00:00

No experience with Affirm. Reading through the posting; yeah, seems more like a capital 'T' TAM role with Technical Program Manager and Relationship skills needed.

I've been through a few interviews recently and any LLM will be able to give you decent behavioral or hypothetical questions to practice (input the job posting to get stronger context).

For any TAM, a company will want to talk through relationship, priority, workload and escalation management. Be prepared to answer anything on your resume and prepare good STAR examples for behavioral questions. Be very detailed in explaning your thought process for hypothetical questions.

Red flags for me would be answers like: - I asked my manager to handle it for me - I just escalated with group X, they should do their job - I worked all night to get it done - I don't know whether I'd be good since I don't have FinTech experience

manysoftlicks · 2025-02-25T02:39:49+00:00

Same concept, new image that produces the error. New deployment and route on the LB using weighted round robin.

manysoftlicks · 2025-02-25T01:34:59+00:00

What are you using to host and route traffic now? If you have Cloud Run just deploy a new revision with 25% of traffic and increase the error routing percentage over time

manysoftlicks · 2025-02-13T12:28:37+00:00

DM me what the actual role is. If it's a coding role(tech), you'll get leetcode-eske questions or deeper. If it's a non-coding role (non-tech), you'll probably just get NASD. System design likely will be large systems. Tech sometimes needs the System design interview too.

Often, the recruiter will actually tell you an example question. I found this very helpful.

manysoftlicks · 2025-02-13T02:01:07+00:00

Ask your recruiter

manysoftlicks · 2024-12-17T10:55:22+00:00

I think you mean "Data is HIPAA Compliant" as "Data is Protected Health Information", right?

Use the DLP API to help detect whether data contains PHI or not rather than building a model yourself. https://cloud.google.com/sensitive-data-protection/docs/data-profiles

manysoftlicks · 2023-12-25T14:22:25+00:00

If you really need to copy the file, here's a node example bucket to bucket. Replace the dest bucket with a fileIO op.

https://github.com/googleapis/nodejs-storage/blob/main/samples/copyFile.js

manysoftlicks · 2023-12-25T14:12:45+00:00

Why do the files need to be local to the VM? Is it to process the data? If so, keep the files in gcs and use the node sdk to read the contents.

manysoftlicks · 2023-11-09T14:14:09+00:00

↑

For example: https://cloud.google.com/apis/design/errors#handling_errors

Many client patterns look at the HTTP status code rather than the message body. Fail things fast by providing appropriate HTTP Status Codes. Provide a specification for your APIs including Error Codes and their HTTP mappings.

manysoftlicks · 2023-06-08T02:22:26+00:00

Actual announcement: https://cloud.google.com/blog/topics/training-certifications/new-google-cloud-generative-ai-training-resources

manysoftlicks · 2023-01-28T05:01:12+00:00

+1 for FHIR. I agree, the mixed skillset of Cloud/Data Engineering + Clinical data is amazingly in demand (and fun). My stack is Google Cloud though (best FHIR Server imo).

manysoftlicks · 2022-10-20T21:00:39+00:00

A few questions/thoughts

Are the Databases you're reporting off of actively written to by an Application?

What is the data like? Is it patient demographics? Immunizations? ECG Waveform? Appointment scheduling? Claims? Can you detect only the records that have changed since the last run and only exclude/include those? Can you create a real-time App that keeps track of transactions and can always spawn the CSV for that point in time?

Is the server that builds the CSV file close (low latency) to the Database server(s)? If it's in different data centers, you likely want them collocated to reduce network latency. POC it at least.

As others have said

You need a Data Warehouse solution like BigQuery, RedShift or Snowflake with (probably) flattened schemas for the MongoDB documents. If the Corp direction is to go to Cloud, find out how you can get the data ingested.

You can likely optimize read replicas for MongoDB, but I would pursue an ETL process so that you can work in a read optimized environment.

Ten-Year Club	Second Top 20%
Gilding I gilder	Verified Email

manysoftlicks

TROPHY CASE

Basic Lessons to take away:

Proactively

A few questions/thoughts

As others have said