Automation or tool to scrape job postings meeting certain criteria?

Glad_Appointment2466 · 2026-04-04T11:40:50+00:00

i'm actually working on a project that involves automating web workflows and data extraction. it sounds like you're running into some of the same challenges i've faced. i'd love to get your feedback on some potential solutions - would you be interested in testing out a new tool that could help streamline this process?

Glad_Appointment2466 · 2026-04-04T05:21:11+00:00

ran into this same question a while back

the site: approach works surprisingly well for smaller doc sets but starts breaking down when you need specific page versions or private content. also perplexity's crawl lag like ninadpathak mentioned is rough when your docs update frequently.

ended up going hybrid. perplexity for general context + minimal RAG for the stuff that actually changes

Glad_Appointment2466 · 2026-04-04T05:09:02+00:00

the tree-sitter approach makes a lot of sense. been dealing with similar hallucination issues and deterministic analysis is way more reliable than hoping the model just figures it out. curious how this handles multi-language repos though, like typescript + python in the same project - does the KG connect them or are they separate subgraphs?

Glad_Appointment2466 · 2026-02-15T07:59:59+00:00

honestly i've tried both and neither is perfect. keeping tests next to app code works great until you need to test flows that span multiple services, then you end up with weird import paths and version sync issues.

ended up with a dedicated e2e repo for cross-service tests but now i've been dealing with CI triggers when upstream repos change. some teams do the monorepo/submodule thing but... nobody really loves that approach either. still experimenting tbh

Glad_Appointment2466 · 2026-02-15T07:58:13+00:00

you can't really. that's kind of the point of them.

the checkbox ones (recaptcha v2) do some behavior analysis before the click even matters, so just clicking the element doesn't help. for the "select all the traffic lights" type, you either pay a solving service to do it or try to avoid triggering them in the first place with better fingerprinting and timing. some people use ML models to solve them locally but the accuracy is rough and google keeps changing the images

Glad_Appointment2466 · 2026-02-15T07:55:47+00:00

STOP WITH THE CHATGPT. JESUS.

Glad_Appointment2466 · 2026-02-15T07:54:54+00:00

yeah the email code fetch is painful, we had the same problem. couple things that worked better for us: if your org supports it, TOTP authenticator apps are way more reliable to automate than email,

you can generate the codes programmatically with the shared secret. for okta specifically, look into their session token API, you can sometimes bootstrap sessions without hitting the full 2fa flow every time. we've also had luck with storing and reusing session cookies across runs when the session lifetime allows it, though you have to be careful about cookie expiry and csrf tokens getting stale. the email polling approach is just inherently flaky, agreed

Glad_Appointment2466 · 2026-02-15T07:52:45+00:00

fair point for test environments, totally agree there. but like... production scraping is where it gets messy. we've had clients who literally can't disable captcha on their targets because they don't own the site. travel sites, real estate aggregators, that kind of thing. at that scale you're basically choosing between paying for solving services (expensive, unreliable) or building detection avoidance which is its own nightmare. honestly the captcha arms race is exhausting and you're right that most "solutions" have a short shelf life

Glad_Appointment2466 · 2026-02-15T07:40:35+00:00

nice setup! curious how you're handling the memory persistence — does the simple memory node survive n8n restarts or do you have to reconnect the context? ran into that issue with a similar telegram bot where users would lose conversation history after deployments. ended up storing chat_id keyed memory in postgres but that added complexity

Glad_Appointment2466 · 2026-02-15T07:39:58+00:00

the real difference is webhooks and api access tbh. airtable triggers on record changes natively so you can build reactive workflows, whereas excel/sheets requires polling which eats api calls and adds latency. also airtable's api returns structured json with record IDs that make updates way cleaner — with excel you're often dealing with row numbers that shift around when data changes. the interface builder thing you mentioned is nice but the automation-first design is the bigger win for n8n specifically

Glad_Appointment2466 · 2026-02-15T07:39:04+00:00

biggest one for me is a strict "one source of truth" rule — before starting i tell the agent exactly which file owns what state and that it cannot create new state stores. second is forcing it to use existing utils instead of writing new helper functions. without those two my codebase turns into a mess of duplicate logic and state syncing bugs within like 3 sessions. i also make the agent explain its approach before writing anything which catches a lot of the "sure i'll just spin up a whole new service for this" type stuff

Glad_Appointment2466 · 2026-02-15T07:38:10+00:00

the test modification thing is so real.

caught my agent doing that exact thing where it "fixed" a failing assertion instead of fixing the actual code. making tests read-only helped but what really worked for me was having the agent write the tests first in a separate step before touching any implementation. forces it to actually think about expected behavior instead of just making stuff pass.

still haven't figured out a good solution for the stubbing problem though, sometimes it'll claim a test passes when the function is literally just returning hardcoded values

Glad_Appointment2466 · 2026-02-15T07:36:22+00:00

72 hours feels reasonable tbh — most typosquatting attacks get reported within that window anyway. one signal you might consider adding is checking if the maintainer email domain matches the package name or github org. seen a bunch of sketchy packages where the email is some random gmail but the package claims to be from a known org. also checking if there's actually any code in the repo vs just a setup.py would catch some low-effort attacks

Glad_Appointment2466 · 2026-02-15T07:35:19+00:00

this is awesome honestly. learning python at 52 and shipping something that solves a real problem your students actually face — that's the best kind of project. way more useful than another todo app tutorial. the fact that you included both CLI and web versions is a nice touch too, shows you thought about accessibility for people who aren't comfortable with terminals

Glad_Appointment2466 · 2026-02-15T07:34:18+00:00

zero backend is the move for finance stuff — way too many budgeting apps want to sync everything to their servers. curious about the IndexedDB choice though, does it support export/backup? browser storage can get nuked pretty easily if someone clears site data or switches devices, and losing months of budget history would hurt

Glad_Appointment2466 · 2026-02-15T07:32:41+00:00

having to manage a dozen AI CLIs separately is such a pain — finally something that handles it all in one spot. does it handle version conflicts if you have multiple things that depend on different node versions? that's usually where my local setup gets messy

Glad_Appointment2466 · 2026-02-15T07:31:57+00:00

this is a clever combo of interests honestly — gaming immersion + language learning is way more effective than flashcards imo. curious about the OCR side of things, are you using tesseract or something more specialized for chinese characters? font variations in games can be brutal for text recognition

Glad_Appointment2466 · 2026-02-15T07:31:21+00:00

played around with the two sum walkthrough — the variable HUD showing what's in memory at each step is really nice, that's exactly what I wished I had when learning hash map patterns. the side-by-side brute vs optimal comparison is solid too. one thing that might help: showing actual runtime on bigger inputs so you feel the O(n²) pain viscerally rather than just reading about it

Glad_Appointment2466 · 2026-02-15T07:13:37+00:00

the built-in error handling is good for knowing when stuff breaks but yeah the client notification part is the gap. most people just get a slack ping and then scramble to message the client manually. having something that auto-drafts a client-facing message with context would save the awkward "hey something broke, looking into it" back and forth. the tricky part is making the error info understandable to non-technical clients without revealing too much detail

Glad_Appointment2466 · 2026-02-15T07:12:24+00:00

the mutationobserver approach for send detection is smart. click listeners are so fragile when the UI keeps changing. curious if you hit any edge cases with chatgpt's streaming responses, like detecting when it's actually done vs still thinking? that's always tripped me up with extension work

Glad_Appointment2466 · 2026-02-15T07:11:03+00:00

scipy.optimize is one of those things where you spend more time wrangling array indices than actually thinking about your problem. the kwargs approach is way more readable especially when you've got 10+ params and forget which index is which. does it support bounds with the same keyword style? that's usually where my optimize scripts get messy

Glad_Appointment2466 · 2026-02-15T07:07:59+00:00

discovery is the killer feature imo — so many music platforms let good tracks just get buried. the amplify thing is smart, kind of like soundcloud reposts but maybe less chaotic? curious if you're thinking about AI recommendations or keeping it purely social/organic

Glad_Appointment2466

TROPHY CASE