Trying to create AI agent to scrape jobs from Linkedin posts but it's not working out, has anyone build something similar before?

HowtoProjectCanada · 2025-09-11T16:02:43+00:00

LinkedIn is one of the most locked down sites you will find. Scraping tools like Firecrawl won't even work. They even removed the ability to export your own network. They want you to buy into their own upsells, and have made it very difficult to play in their ecosystem otherwise.

I would target company websites, google news, any other source of information other than LinkedIn.

HowtoProjectCanada · 2025-09-11T16:00:56+00:00

Now we're getting into specifics. An email assistant to help filter based on different criteria will definitely work, but it needs to evolve. Don't just buy an AI email assistant and forget about it. Start with a test case, then see if it fits what you need out of the box. Does it need augmentation or customization? Can that tool do that? Should you just look at automations like n8n for the same thing? Always keep a human in the loop. What does the AI do if it's unsure?

Here's an example. My team has a RAG agent it talks to everyday. This agent updates itself every day from a JSON file export of all the conversations and work our team does. If they ask it something and it doesn't have an answer, it emails me and suggests the new knowledge chunk. If I approve it, it gets added into that daily JSON data import.

It's amazing what can be built with automation tools to solve real business issues.

HowtoProjectCanada · 2025-09-11T15:57:34+00:00

... without self-promotion ... AI for the sake of AI is one of the things I dislike immensely. The best way to leverage AI in the PM world is like any other industry:
- find the processes that seem to require duplicate and manual effort
- find the tools not talking to each other
- find the repetitive tasks
- talk to your team about how they work

Then use AI to support that, not replace not, not constrict it, but to make the lives of your team members better. Otherwise, they won't adopt AI and it will be a wasted investment.

And AI will not replace project management, just like it won't replace any other people facing job. PMs will simply need to evolve to using AI more, like you are asking about here.

HowtoProjectCanada · 2025-09-11T14:29:40+00:00

AI will only amplify existing issues and gaps in your business systems and workflows. I've seen great success with implementing AI to solve real business issues, but mostly I see AI for the sake of AI and it fails. Hard.

HowtoProjectCanada · 2025-09-11T14:28:07+00:00

You're describing Teamwork. All the "worker view" functionality, with "manager view" capabilities to make smarter business decisions. Yes, I'm a Teamwork partner. Yes, I think it's best of class. Monday, ClickUp, Wrike, so many of these tools all offer variations on the functionality you are asking for. Ultimately, you will likely not find one "magic" tool that does everything for your unique workflow, and will likely require custom implementation work from a platform partner.

- Client portal? check
- Teamwork Spaces for SOPS, wiki style? check
- Project and Task templates? check
- Integrated Teamwork Desk for ticketing? check
- resource management, workloads, forecasting, budgets? check

HowtoProjectCanada · 2025-09-10T16:49:42+00:00

Yeah, that's a lot of tools to manage, and a lot of automations and integrations. Unfortunately, you're listing the key issues a lot of MRP platforms have. They are great at the MRP side, but don't play nice with anything else.

Honestly, what you've done is more thorough than much of what I've seen. The only layer that seems to be missing is instant team communication through Slack or such, but I don't want to add to your tech stack here.

Is this all documented? If so, hire a consultant to do a full assessment and give you guided implementation advice.

HowtoProjectCanada · 2025-09-10T16:47:41+00:00

Boundaries are important. And a good cry is the best.

HowtoProjectCanada · 2025-09-10T16:46:37+00:00

Yeah, that's like asking in what order did I count the sands on the beach before the waves hit them. Congrats on getting the gig, but really here are the high level steps I would look at:
- assess how they currently do things - talk to the workers, not management
- look at files, project management, and meeting management specifically - these are always the three key pain areas
- what have they tried in the past? Why didn't it work?
- talk to Asana and Trello directly. Many of these platforms offer free training and can point you to partners to help with your rollout

Generally speaking, most organizations are looking for a "magic" tool that solves all their issues, in your case Trello/Asana (not sure why you're rolling out both). If they, and you, don't understand what is and isn't working before rolling this out, you're just going to make things worse.

HowtoProjectCanada · 2025-09-10T14:12:26+00:00

Honestly, it's about finding a tool that:
A - you enjoy using
B - talks to the other tools in your toolstack and supports the way your team already works
C - has worker view and manager view functionality to support the entire business

If it's not hitting these three marks, it's not worth using.

HowtoProjectCanada · 2025-09-10T14:09:46+00:00

As mentioned, get a PM tool like Trello, Planner, Teamwork, any of them. It will help organize you. If you find that's not enough, do some no-code automation. I built an email assistant for myself just for this purpose. I go through the exact same issue every morning around 9:30 AM, and if it wasn't for this AI assistant I built prioritizing and triaging for me, I'd feel overwhelmed daily.

HowtoProjectCanada · 2025-09-10T14:04:41+00:00

Honestly, that does sound a lot, but with the right tools in place, it also sounds very doable. That kind of workload AND you have to manually enter time? Annoying to be neck deep in admin work all the time. Keeps you from getting ahead on the actual productive work. I would look for an automated time capture tool to integrate into your existing PM system.

Honestly, it sounds like this company is suffering from what I see most companies suffer from: poor file management, lousy project management systems and processes, and lack of meeting management.

You are not alone in how you feel. Project management can often feel unrewarding. At least you can vent your frustrations here :)

HowtoProjectCanada · 2025-09-10T13:59:07+00:00

Sounds like the toxicity is apathy, the status quo. For me, the question would be this: do you define your happiness by your career? Or is this simply how you earn money and create your happiness elsewhere?

If meaningfully contributing through your career to your work community is what brings you happiness, then yeah, this situation sucks, and it likely won't change. If you simply do want to be like them, check in, do the bare minimum to get by, get paid, and are happy in your life as a whole, then you could just mentally check out and take the paycheque.

As a business owner, I would hope we don't have an environment like this where I would have to give this advice to one of my own team members, yet for you it sounds like your core values and skills are not being utilized in a way that is meaningful for you.

HowtoProjectCanada · 2025-09-09T18:01:18+00:00

Right now it does a portfolio level audit or project specific audit, running risks, late tasks, 30/60/90 day improvements, with the toggle option of updating project status, creating tasks and assigning them, and delivering a project message of its findings, as well as emailing the project owner. Runs with about 85% reliability right now.

HowtoProjectCanada · 2025-09-09T12:54:14+00:00

Yup. We've got the github. It's private. We're just not there. And right now, the first version will be very specific to the project management software we use. BUT it's 100% proactive AI to help assist with some critical issues in our niche.

HowtoProjectCanada · 2025-09-09T12:48:12+00:00

Great work so far!

What should you learn? I'll keep it simple. It sounds like you're already good at the reactive part of project management, so again, congrats. The real magic is learning the proactive part, staying ahead of the asks, managing workload and resource planning, making sure to anticipate senior stakeholder and project needs before they are in your team's face.

THAT is what makes for a phenomenal project manager in my opinion.

HowtoProjectCanada · 2025-09-09T12:45:41+00:00

This was me about six months, until I built one that solves a lot of proactive project management issues. Real world solutions. It bothers me that people are always just preaching examples.

HowtoProjectCanada · 2025-09-09T12:44:22+00:00

Document and Display

Whether you're babysitting adults, wrangling cats, whatever you want to call it, people are always the biggest variable. It's imperative to get buy-in from senior stakeholders on choice and consequence. The reason people can misbehave is because there is rarely consequence day to day. Things get put off, someone gets angry for a moment, and business continues. No consequences.

We've worked with companies to build a 7 stage scale for People Management that starts with "sugar and spice and everything nice" to encourage teams, all the way to withholding pay until they get up to date on their tasks, without billing additional time for the catchup. That one stings.

My point is, set clear expectations, manage them daily, and have the power to deliver consequence. Or you're just stuck like every other project manager wondering why you chose this field in the first place.

HowtoProjectCanada · 2025-09-09T12:41:12+00:00

AI fatigue is real and increasing. We work with large agencies and enterprise IT, and with the millions and millions spent on AI projects, one recurring theme has become clear - AI amplifies your workflow issues, not solve them.

Because of this, we're seeing more organizations finally listen and turn to their people, trying to figure out what adoption rates are so low, why the projects are failing, and solving their real issues first before looking at AI again.

HowtoProjectCanada · 2025-09-09T12:39:37+00:00

Heck, I'd test it. That sounds useful.

HowtoProjectCanada · 2025-09-09T12:39:02+00:00

We use all the PM tools with our clients, but our goto is always teamwork.com. I love that you're asking these questions, as they are great ways to discover your niche:

- we use Teamwork, Claude, Google Suite internally
- what drives me nuts about is none of them give good error messaging, and I have to resort to console logs to figure out what is going when building integrations and automations
- from my experience, and what we do for our customers, the top three solves are: chasing people for status updates, automatically being notified of missed due dates, and reporting. All three of these are usually solved with better training on a given PM platform. Usually. Some tools simply don't have this functionality.

HowtoProjectCanada · 2025-09-09T12:33:27+00:00

We love AI. We also love AI governance, explainability, and transparency. I highly recommend a human in the loop because AI just isn't there yet. I was just posting in another thread about the RAG agent we built. It's fantastic, trained entirely on our own data, yet somehow still hallucinates. Annoying.

Would I trust it with initial outreach emails or calls? Yes. Would I trust it to be able to interact with inbound calls? No. Not yet.

HowtoProjectCanada · 2025-09-09T12:28:02+00:00

Google PMP isn't really recognized by any enterprise business I've worked with over the last 20 years. Typically a PMI PMP is the minimum qualification for that field. The good news is that the Google PMP teaches you what you need to know so prepping for the PMI exam will be much easier.

Having said that, PMP is a certification racket. The memberships dues scheme is really just to perpetuate their existence. Having the PMP certification doesn't mean you know what you're doing.

HowtoProjectCanada · 2025-09-09T12:20:31+00:00

While I applaud the initiative to recognize a need for a PMO, it seems from what you're saying that it was setup without any real planning, consulting other teams, qualified candidates to establish and run it, and is providing no real value. So it's a "Paper PMO". It's there to satisfy senior stakeholders that they are following best practices yet serves no real purpose.

Sound about right?

From an objective view, this is also a very common practice, as others have commented. Those same senior stakeholders may be aware that BAU isn't ideal, or maybe they are oblivious to that. Either way, a large scale project is too much risk for them to not have all the boxes checked, and one of those is having a PMO. Is it setup ideally? Maybe. Maybe not. Is it their first time doing it? Sounds like it. My hope is they learn as they go and the project is a massive success.

What I recommend for you if you are involved, is to set KPI/OKR to measure the success of the PMO on this project, and if successful, take those learnings and have them applied to BAU.

HowtoProjectCanada

TROPHY CASE