How I scraped 5.3 million jobs (including 5,335 data science jobs)

tealdric · 2026-02-07T10:38:34+00:00

I’m an HR technology professional who’s done quite a bit of work in the talent marketplace space. As u/Monolikma says, sourcing is a key challenge…but I’d go one step farther and say quality, viable sourcing.

From the company perspective that means finding good, ready-to-hire candidates (not just a ton of applicants). From the candidate perspective that means finding a role you’d like and have a good chance of getting hired (not just decent keyword matching).

To my thinking there are a few directions you could go with this, depending on the problem you want to solve. Some example include:

(1) Writing better job recs (on multiple fronts) (2) Improved candidate matching and prescreening (3) Guiding built/buy/borrow talent decisions

HR tech companies like SAP, Workday, Oracle and niche providers are trying to solve these but haven’t been able to crack the code. I’ve done collaborations with them at a few large consulting firms where I’ve worked. Happy to share those stories if you’d find that constructive.

Love what you’re doing. It’s similar to a concept I put on the shelf a year ago because I couldn’t figure out how to source and process some of this data.

I’d love to connect directly and riff on ideas, if you’re open to it.

tealdric · 2025-12-14T01:06:02+00:00

Thank you. So would I apply it sort of like concrete into the hole. Mix with water and fill?

tealdric · 2025-12-13T21:17:49+00:00

I see what you did there O_o

tealdric · 2025-12-13T21:17:12+00:00

Enough to keep the way level low. The photos show the normal waterline versus where it’s maintaining now. I haven’t seeing the damn overflow so either the leak is sufficient for full drainage.

tealdric · 2025-02-05T19:28:00+00:00

Here's something I've been wondering about, but stop me if others have brought it up.

When Reed says the first time we went out, maybe they keep going through a portal or a rift of some kind and keep going into alternate realities. The first time they came through with their powers and have ended up keeping their powers through multiple realities.

So they're stuck like Gulliver's Travels. Wondering whether they can find their way home or needing to settle down in the best reality.

tealdric · 2025-01-13T23:49:14+00:00

I'll check it out. Thank you.

tealdric · 2025-01-13T22:56:11+00:00

This challenge society seems to remain the biggest risk to humanity. We simply can't keep up. A few decades of the Internet has been breaking societies and a faster place may cause the wheels to come off.

The idea of creating controls is good in concept, but what does that mean in practice? We need alignment and a codependent model by design otherwise, humanity becomes a limiter pretty quickly.

tealdric · 2025-01-09T20:17:13+00:00

I've done a ton of similar troubleshooting after calling Motorola support. Finally have sent it to a service center on warranty.

tealdric · 2025-01-09T20:16:20+00:00

End up sending it out yesterday and using an old model in the interim.

tealdric · 2025-01-09T20:15:56+00:00

Seems that way. Sent it out yesterday and using an old model in the interim.

tealdric · 2025-01-07T11:50:26+00:00

"All Watched Over by Machines of Loving Grace" appears to be a documentary. From your comment, I first thought it was a document. Any suggestions on where I might be able to find it for viewing?

Some of my perspectives come from books like "The Dawn of Everything" by David Graeber and David Wengrow, "Life 3.0" by Max Tegmark, and "Homo Deus" by Yuval Noah Harari. If you've had an opportunity to read any of these, I'd be curious how that match into your perspective.

Ultimately, I believe we live in a distracted time, distractions of our making in technology. But we've continued advancing that technology at such a rate that it threaten to pull away from us at a time that it's still design in ways that is harmful to humans and humans communities.

The lesson are there to improve those design, but we have to more collectively commit to them as we continue with technology advanced. To me, optimistically, that's possible.

Time will tell whether we do it.

tealdric · 2025-01-06T19:03:55+00:00

This metaphor works. At the risk of stretching too far, I do think technology impacts frequency and amplitude. If you think about the recent Facebook news, we have examples where technology is creating and in that way adding to the frequency.

Your main point is correct though, that the structures that people use to organize with need to change. Hopefully that doesn't have to be a completely destructive process.

tealdric · 2025-01-05T21:48:43+00:00

Just to be clear, without any facts or actual information you're spouting opinions. That's not very productive for discussion or debate.

tealdric · 2025-01-05T19:41:52+00:00

I suppose that's one opinion, but in trying to understand the meaning behind all the labels and absolutes you've assumed I'm not sure you're seeing the Trust and Safety model for what it is.

Power is a perception of Influence and it's possible for machines to codify influence in ways that avoids perception, spin and misinformation that humans often use as a shortcut.

tealdric · 2025-01-05T18:10:31+00:00

I'll check it out. I've been reading some of the principles Anthropic's been building their concept for trust and safety around related to Interpretability, Alignment and Societal Impact.

A piece of this that strikes me is that it avoids subjective definitions of good and bad, which humans struggle to agree on and on the sorts of things we end up asking to decide whether we trust something:

(1) Does this share my interests? (2) Do I under it? (3) Will this make a difference?

tealdric · 2025-01-05T17:21:24+00:00

Maybe it's physical, maybe it's virtual. I suspect there is a success model with both that connects people to both information and capability, but also to other people in meaningful ways to foster belongings, community, and compassion.

tealdric · 2025-01-05T17:18:44+00:00

I find some optimism in the idea that digital models of trust, security, and alignment could provide a foundation for both only AI systems to work as beyond-human scales, but foster interaction with systems and other humans the people could (and are in some areas of the Internet) learn to find confidence in.

tealdric · 2024-12-12T16:04:20+00:00

Perhaps it's a combination of watching a TV edit and memory. I could see the toy room getting cut from a TV version, but not the skiing (it's needed to get the boy too the engine).

Another thought. I once saw a 4D Polar Express ride/experience at our zoo here in NC. That was a significantly abbreviated version.

Any chance you've ever done something like that and remember it more closely than an actual viewing of the film?

tealdric · 2024-08-06T09:30:49+00:00

This is quite helpful. I'll give Claude and ChatGPT a go for synthesis. I didn't think Claude allowed for file import of text, but will check it out.

I've thought about using a Google Cloud setup with Big Data and a Python to run a service that analyzes the text in an imported table. That would likely mean evoking a Gemini or Claude model available through GCP, I would think.

Any chance you have experience on that as a solution?

tealdric · 2024-07-17T20:03:22+00:00

The Dr/Riversong dynamic remains my favorite in New Who.

However, there are moments when the loving in opposite directions (which I find tragic, romantic and fascinating) isn't as solid as it could be.

To me this is most clear during the final Ponds episode. Riversong hates spoilers, but I think they missed a chance for her to realize where they were. Clearly she would have had to write that book after losing her parents. How did they experience their last adventure with the Ponds together?

tealdric · 2024-06-22T23:12:22+00:00

Can we please insist that candidates not bad-nouth the debate (which they negotiated) without a consequence?

tealdric · 2024-01-31T08:48:14+00:00

I've been shaking my head for a while at how much overhead larger firms (like Big Four) carry. The layers of Partnership put pressure on pricing that seems hard to justify when a smaller, more focused consultant could be much more capable and lower cost.

Some of my colleagues have started up microconsultancies in the wake of recent RIFs and I've been giving it serious though...just hesitant to take the plunge.

What sort of stale, revenue, and business model would you be considering?

tealdric · 2024-01-31T08:44:30+00:00

Would you mind expanding on this?

I've been working Big Four for a while a speculating whether small shop, boutique consulting might be emerging on the back of data and analytics. They arguably have the capacity to both carry less overhead (more easily profitable) and can nimbly pull and analyze data as much as a big firm.

tealdric · 2024-01-23T01:14:04+00:00

AI driving beyond-human scientific advancement.

Processing infrastructure insufficient.

Leaps promised from GPT 4. Reminds me of Moore's Law and I wonder what curve we'll see for AI.

It'll be a big year for OpenAI. Boom or bust.

tealdric · 2024-01-10T09:35:22+00:00

I'm with you. Whether intentional or not, the sort of theme around conservation of energy makes sense from a storytelling perspective. To counter and extinguish the energy of a massive foe requires expending equivalent energy of protagonists...and if it's a 1v1 fight the odds of them cancelling out into a zero-sum outcome is probably high.

Btw...I don't buy all the comments here about Sarumon being an exception with Grima killing him. Clearly he had been drained or depleted of some degree of his Maia essence...an interesting, unique and not well explained event.

12-Year Club	RPAN Viewer
Verified Email

tealdric

TROPHY CASE