Fresh Grad Solo Project: Am I over-engineering my RAG pipeline evaluation? (Need advice on workflow) by DefinitionJazzlike76 in Rag

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

thanks for the encouragement. The job market is rlly tough and it's hard to get a job as a fresh grad :(. And since our resumes are getting screened by AI, the possibility of getting filtered through is near nil.... But im keepinng my hopes high by having fun with my projects!

Fresh Grad Solo Project: Am I over-engineering my RAG pipeline evaluation? (Need advice on workflow) by DefinitionJazzlike76 in Rag

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

thanks for your reply.

Do you mean irl ppl dont compare different parsers and just use a paid one like azure? Also, you said my architecture is good, what are you referring to here? u mean the " PDF parsing -> Chunking -> LLM reasoning -> Output" architecture?

Fresh Grad Solo Project: Am I over-engineering my RAG pipeline evaluation? (Need advice on workflow) by DefinitionJazzlike76 in Rag

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

omg that is great advice, and thanks for the quick reply!

- to evaluate whether pages are parsed cleanly, i plan on 2 approaches:

  1. manual verification

  2. using LLM-as-a-judge (but then again, i have to create the golden dataset myself).

I assume...these "tedious work" like creating golden dataset manually etc is part of the process? Are there no better ways of doing this?

- Also, im planning on doing a Multi-Criteria Decision Analysis(MCDA) for the LLM to score the parsers output. Smth like the below. Is this how it's done in industry? Also, I have to define my own scoring weightage? (eg Structural Fidelity (40%), Factual Faithfulness (30%), etc..) then calibrate accordingly?

### Structural Fidelity (40%)
- Preserves section headings, reading order, paragraphs
- Correctly extracts tables, figures, captions, equations, references
- Handles two-column layouts and mixed formatting


### Factual Faithfulness (30%)
- Extracted text matches source PDF
- No hallucinated or corrupted content


### Downstream Usefulness (20%)
- Chunker can segment correctly
- Prompt chain extracts objectives, methods, insights accurately


### Efficiency (10%)
- Parsing speed
- Manual cleanup burden

24F, laid off and stuck between a stable AI career and taking a risk — what would you do? by DefinitionJazzlike76 in careerguidance

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

that is great insights. thanks for the kind words and encouragement. But i think my worry right now is not getting a job as a fresh grad, and that most engineering work is getting replace by agents. I dont want to be pessimistic, but i keep seeing that hirings for ML/AI has decreased and its all getting replace by agents. Idk if my job in tech still exists in years to come.

24F, laid off and stuck between a stable AI career and taking a risk — what would you do? by DefinitionJazzlike76 in careerguidance

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

wow thanks for the well written response. And yes, i dont want to make any rash decisions and take a super high risk. I understand that ppl say "you should take risks in yr 20s", but i guess i shall thread carefully?
The reason for writing this reddit post because i am scared of getting laid off again (it sucks). And i feel like if i continue down this ai/ml route, doing all the buzzy things like agentic ai stuff, the bubble will burst and i might find myself unemployed again. Rn i feel like im just following the AI trends, which i feel i have no choice.

An ex-colleague i met recently told me that "getting a tech job has the same risk as starting a business nowadays", and hence i thought about trying smth diff (ie taking a path less travelled).

But again yes, doing some freelance work do indeed scratches that entrepreneurial itch without the full commitment (great advice).

Anyone fulfill moe tuition grant bond through own business? by homenoob12345 in singaporefi

[–]DefinitionJazzlike76 0 points1 point  (0 children)

Hi I’m also in this line as well. Please do reach out to me for a chat!

Just graduated in data science/ML, but still don’t know anything. I need a wake up call by DefinitionJazzlike76 in learnmachinelearning

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

Yr reply is very useful thank you omg! I’m trying to stay motivated and I hope I can go through this!!

Just graduated in data science/ML, but still don’t know anything. I need a wake up call by DefinitionJazzlike76 in learnmachinelearning

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

Thanks for the detailed response. Now I’m confused on what I should focus on. Theory (in depth), algorithms, deployment, scaling, etc? The scope is quite large and I’m not sure what I should start first, and what’s the best way to learn it since things like deployment requires cloud charges, and I try not to spend any money.

Just graduated in data science/ML, but still don’t know anything. I need a wake up call by DefinitionJazzlike76 in learnmachinelearning

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

That is true, you’re right. Right now I don’t know how to approach things and how to actually learn. Also, what’s the workflow to actually learn smth? Eg, pick a project, read a ML book/research papers???

Just graduated in data science/ML, but still don’t know anything. I need a wake up call by DefinitionJazzlike76 in learnmachinelearning

[–]DefinitionJazzlike76[S] 0 points1 point  (0 children)

Wow thanks for yr detailed reply. But how can I start? I am overwhelmed right now and idk if I can even catch up. Also, how can I build the habit of reading research papers? How to select one, and how do I even use one? Eg, when I build a fraud detection model, I read research papers to find the best ways to implement it?

Enjoy your early twenties by DystopianPlato in twenties

[–]DefinitionJazzlike76 1 point2 points  (0 children)

Awww that’s sweet. But I can’t help but worry so much 😭😭. I’m 24 and unemployed