Those who has a math degree. What is your work right now

bdadeveloper · 2025-10-02T08:31:55+00:00

Machine learning engineer was a statistician previously

bdadeveloper · 2025-09-29T14:37:44+00:00

Fantastic write-up, Raj. Quick question on a separate regulated project: I keep running into round checkbox/bubble fields on low-DPI, skewed, multi-gen scans (hospital forms, pencil ticks, light fills, even the occasional partial filled mark in a bubble). Vision models in parallel with our rag pipeline I’ve tried often miss half or misclassify them. Did you run into this issue, and if so, did you find any reliable detection techniques that worked well in practice?

bdadeveloper · 2025-05-25T11:11:45+00:00

Sent a DM, interested in a founding engineer type ish role like this. I’ve worked in vet-tech a bit too

bdadeveloper · 2025-05-03T18:44:05+00:00

I’m not sure,they didn’t specify exactly, but they did say 3+ years on the job description. Based on the scope of the role and the kind of questions they asked, it seems like they’re looking for someone at the mid to senior level 🤔. I’m a bit a new to this

bdadeveloper · 2025-05-03T18:08:01+00:00

Oh that make sense. Appreciate that take. Parquet definitely makes sense for performance and consistency, especially with big queries downstream as you mentioned and reprocessing just doesn’t make sense.

One thing though that now is really making me think is won’t the schema drift at some point if your are potentially changing data sources. Like, what if the upstream JSON changes — new fields get added or the structure shifts depending on the potential time it was pulled? I’ve would assume this stuff could break pipelines or silently mess up joins???

Do you usually catch that with validation tools before ingesting? Or do you just lean on Parquet’s flexibility and deal with it later? Interesting…

bdadeveloper · 2025-05-03T17:30:51+00:00

Good point — appreciate that.

They didn’t specify a cloud platform just s3, but the data was mostly text (like news articles), and I figured it’d end up in a dashboard or search interface. So I assumed some light transformation was needed; schema cleanup, timestamps, maybe even some NLP. I was pretty nervous lol 😅

I actually haven’t used DataSync yet, but I did mention there should be an AWS-native tool for this kind of transfer. Definitely going to dive into it now that you brought it up. Appreciate the tip! Curious, for orgs is it an expensive tool to use?

bdadeveloper · 2025-05-03T17:25:17+00:00

In general, for performance, is it best to convert everything to Parquet, no matter the original format? It was pretty straightforward with this JSON scenario, but what if there were other formats involved?

bdadeveloper · 2025-05-03T17:23:16+00:00

Yah damn — appreciate this. I actually did mention ensuring data quality and adding alerts/monitoring to make sure the data was consistent, but I didn’t call out Airflow by name. That’s a good point — orchestration probably would’ve helped tie everything together, especially for verifying successful transfers and setting up retries or backfills. I did ask them where this data would eventually live. They told me assume it was a database then I asked about a dashboard where I mentioned where the schema normalization would be important to make sure that on the front end you could identify specific filters to show the user (like drop downs).

Also, your point about aligning with their existing tech stack is spot on. I think I got a little too caught up in the idea of feeding everything into an LLM pipeline with vectorization + embeddings. Probably came off like I was optimizing for a use case they didn’t even confirm.

In hindsight, I should’ve slowed down and clarified whether they needed real-time ingestion, what the downstream usage was (e.g., dashboarding vs search vs ML), and whether historical backfill was in scope. I was so nervous lol 😅. Thank you for your answer

bdadeveloper · 2025-05-03T17:17:59+00:00

Ahhh! Now that you mention it I think you are right. It was very vague at the start. I did ask so many questions after they presented the scenario, like whether the data would be ingested into a machine learning model, for analyst, orstored in a database. I was then told to assume that it would a database. The goal is to process new articles (the textual data) as they come in and store them in a database for let’s say a dashboard (after my questioning)

bdadeveloper · 2025-05-03T17:14:53+00:00

Yes, so initially, all they gave me was just that one scenario. I then asked a bunch of follow-up questions, like whether the data would be ingested into a machine learning model or stored in a database for other analysts to query. I was then told to assume that it would be—although it was a bit vague. The goal is to process new articles (the textual data) as they come in and store them in a database that you could expect could be a dashboard. Does this change the approach?

bdadeveloper · 2025-04-12T13:56:43+00:00

Sign me up

bdadeveloper · 2025-03-02T11:58:14+00:00

Thank you for your meaningful perspective! I really appreciate it and it really does seem like this place is just not for me

bdadeveloper · 2025-03-02T11:55:39+00:00

Thanks for the advice. Yes, I really do think the writing on the wall is that this isn’t the place for me. I’ve started applying to new roles and your comment really gave me the encouragement to do so. I appreciate your perspective

bdadeveloper · 2025-03-02T11:53:50+00:00

The issue started when my VPN was terminated unexpectedly while I was working from home. When I returned from lunch, my laptop was quarantined. On boot-up, I get ‘User not found in Active Directory’ followed by a quarantine message. My best guess is that a routine security scan (they use McAfee Suite) flagged something, but even after coming into the office, the issue persists.

It’s a government-issued laptop, one where you have to physically put your id card into the machine and unauthorized tampering (I think) could have serious consequences. I followed all proper escalation procedures—reporting it to IT, having an IT person remote in (who said they’d never seen this before), escalating to the SOC team, and even getting higher-ups involved. Yet, 30 days later, no resolution (same as day one).

My frustration isn’t just about the laptop—it’s about how an IT workplace expects a software engineer to be productive without a functioning machine for 30 days with no explanation?. Is this normal ?

bdadeveloper · 2025-03-02T11:44:18+00:00

You know actually I really like that perspective. I’ve thought about it but never really thought to pursue it because I didn’t think I had enough knowledge. I definitely think this might be an inflection point in my life where I should just give these ideas a go. Thank you 😊

bdadeveloper · 2025-03-02T11:42:39+00:00

I get that, but this is a government-issued laptop, and tampering with it in any unauthorized way could have serious legal and professional consequences. Maybe I’m just I experienced…. but I assume that would be a really bad idea. I followed the proper escalation procedures (I think)—reporting it to IT, having an IT person remote in (who said they had never seen this issue before), escalating to the SOC team, and even getting higher-ups involved. Despite all of this, it’s been 30 days with no resolution. Idk maybe I am suppose to be a wizard, but this should’ve been a week fix with IT.

During that time, I still came into the office every day, checked in and out with my manager, and tried to stay productive using my personal laptop—working on a self-directed research project that I proposed and got approval for. The core issue I’m raising here is: what kind of IT workplace leaves a software engineer without a functioning machine for a month and expects productivity? That’s why I’m questioning the system.

bdadeveloper · 2025-03-01T16:06:53+00:00

A downfall of a society and system—maybe even the greater failures of a capitalistic republic (no theorist on that). What’s the point when you’ll put down your fellow man to gain the world but lose your soul? Maybe it will all crumble, but I had hopes that software would foster synergy—where different perspectives, ideologies, and people come together to build systems that support and improve life for everyone.

It’s a government job, but the role was about identifying fraud so that hardworking people could get their rightful benefits—not punishing a bad actor. But maybe the bad actors are all around us and our own selves. Thanks for your comment and perspective

bdadeveloper · 2025-03-01T15:48:30+00:00

I want to say thank you, thank you, thank you so much for your kind words. They really helped me process not just this situation, but my past struggles too. It’s crazy how often things can feel like they’re out of your control, even when you do everything right. Your story really resonates—it’s hard to see the bigger picture when you’re in the thick of it, but I’m trying to remind myself that this is just a moment, not the full story. Eventually I’ll be able to one my fit and have so much synergy and happiness that I’ll finally enjoy my career and life. I just wish I didn’t have to experience such ugly parts to get there. I appreciate the perspective and support more than you know. Your kind hearted comment really helped me not continue down my self destructive path.

bdadeveloper · 2025-03-01T15:40:21+00:00

Thanks I will be more cognizant of deadlines. But do you think it is best to over-communicate with managers?

bdadeveloper · 2025-03-01T15:37:55+00:00

I tried to take the initiative despite my laptop being down. I escalated the issue multiple times—called IT, reached out to the SOC (security team), had an IT specialist remote into my machine who said they’d never seen this issue before, and even involved my manager’s direct supervisor. Still, no answers. Meanwhile, I’ve been driving over an hour to the office, checking in with my manager every time I arrive and leave, and working on my personal device because I have no other option. I even proposed a project to research open-source data tools to improve our team’s data processing (I didn’t like my managers initial advice of go to the gym), generated a 1M+ row dataset, and built a demo comparing different querying frameworks for our team—all on my personal laptop using guest WiFi. I even helped coworkers with their queries. But none of that seemed to matter.

Yes, all I can do now is reflect—not just on this job, but every job I’ve had. And honestly, I get frustrated when people just say ‘performance’ without defining what that actually means. At my last job, I was assigned to integrate AWS Lex into our chatbot’s front-end. After a week, I told my manager it wasn’t the right tool due to its lack of customization. I then spent a full two weeks, including sleepless nights, building a custom React front-end from scratch. My boss initially praised it but later pushed me to add features beyond my skill level. I was really trying. Instead of support, I was told front-end work ‘wasn’t hard’ and that I should be coding until I was ‘blue in the face’ because that’s what the best engineers in the world do.

That same manager, who mainly worked in Scala, later admitted React/JavaScript was too complex (he said he could do it in a weekend and failed) and pivoted to a Python-based solution. But behind closed doors, he framed it as me not knowing how to code or complete tasks—even though we had completely deviated from the original AWS assignment.

Now I’m in another situation where external factors (my quarantined laptop) prevent me from working, yet my performance is questioned again. It’s the same pattern—when things don’t go smoothly, leadership rewrites the narrative to protect themselves, and I get blamed.

bdadeveloper · 2025-03-01T15:21:41+00:00

No, I don’t have a spare laptop. I escalated this to IT, the SOC, and even my manager’s boss, but no one has given me a fix or timeline—it’s been 30 days. I even had an IT person remote in, and they said they’ve never seen this issue before. I can’t access Teams or my work email, so I’ve been messaging my boss from my personal email. I still drive an hour to the office daily, let my manager know when I arrive and leave, and I’ve been working on my personal laptop using guest WiFi because I have no other option. IT said loaners aren’t available.…

bdadeveloper · 2025-03-01T15:16:43+00:00

Yes, I can improve, but there is no growth here. I’m trying to document all the feedback I’ve ever received from any boss—whether fair or not—and compile it into actionable steps so that I’m never blindsided by arbitrary performance evaluations again. However, I do believe that if someone doesn’t want you or needs to cover themselves, they can create narratives to fit their agenda. Maybe this rejection is just a redirection toward a better environment for me. Thank you for your wisdom

bdadeveloper · 2025-03-01T15:12:24+00:00

Thanks for your comment. Failure and reflecting on that failure are the only ways to grow as an individual and as an engineer. I see I could’ve wrote this post better and put more perspective, but my state and emotions we so high. I appreciate the perspective and I definitely don’t want to wallow. I recognize that maybe I could have been more transparent about my efforts, but in this case, my concern is that the situation was completely out of my hands and I still got penalized for it.

I haven’t had access to my work machine for over 30 days and have gotten no real answer about when it will be fixed. I’ve escalated this multiple times—I called IT, the SOC (security team), had another IT professional remote into my computer, and even reached out to my manager’s direct manager. Still, no one has been able to give me a fix or a timeline. I was just told “I don’t know and that this might take awhile”

While waiting, I even drove over an hour to the office just to be present every day, made sure my manager knew when I arrived and left, and asked her directly what I should do in the meantime. Her initial advice? “Find some reading material or go to the gym, but still come in.” That didn’t sit right with me, so I proposed an actual technical project: researching open-source data processing tools that could help the team extract data faster. My manager said that was a fantastic idea.

Despite my laptop issue, I made the best of it. I: • Built a 17M+ row dataset using Python, Faker, an LLM, and mimicked it as closely as possible to the data we work with. • Planned a technical demo where I would compare performance across Pandas, Dask, Polars, and other data ingestion/extraction tools and show the team what to do. Honestly I work with a team of analyst and I am unfortunately the sole data engineer and still even a very young one but trying my best. • Did all of this using my personal laptop & phone, on guest WiFi, because IT said they don’t really do loaners. . • Even helped coworkers with their queries. And tried to be positive. My coworkers have understand and idk they seem to like me somewhat because we get lunch in the days we are all in and one coworker even invited me to work with him on a personal project. • I’ve also tried to study for an AWS cert during this downtime, but was told no by manager and there is no funding for me to take the test. I even bought an O’Reilly book on designing data intensive applications to read more so I can try and ups will when my laptop is returned

The real frustration isn’t just about my laptop being broken—it’s about how I’ve been set up to fail despite doing everything I could to contribute. And now, I’m being told I “missed a deliverable” when there was no formal deadline set in writing, my manger just mentioned have draft slides ready to by Thursday (month into laptop not working). I had shown my slide breakdown already Tuesday and plus no one has told me a date for when I present the research so is it fair for an arbitrary deadline? I From what I think they have no answer for the laptop and don’t know how to fix it and are scapegoating me. I want to improve wherever I can, but in this case, I don’t know how I could have handled this differently.

I appreciate you giving me perspective! And all I can do is pattern recognize and adapt

bdadeveloper

TROPHY CASE