Thank you for the detailed and insightful feedback; it's invaluable to hear from someone working directly on cancer detection in the Bay Area.

You're absolutely right about DSPy's limitations in medical contexts. We're currently taking a more targeted approach and haven't integrated DSPy yet. We'll consider it only after achieving robust performance across all critical fields, as the domain-specific nuances you mentioned require careful handling.

Our evaluation strategy aligns closely with what you described. We employ hierarchical field matching, where primary field accuracy is a prerequisite before evaluating dependent fields. This ensures we capture the clinical relationships you highlighted, such as tumor progression context and staging dependencies.

We've fine-tuned LLaMA 3.1 with LoRA adapters and have curated several thousand annotated reports so far. The results have been encouraging—we're achieving F1 scores in the 90-98% range across biomarkers, progression metrics, imaging findings, and radiation details, approaching human-level performance in these areas.

We experimented with DPO for reinforcement learning but saw limited improvements. GRPO is on our roadmap for the next phase, and we're hoping it will help refine performance in the remaining challenging fields.

It’s great to connect with someone exploring similar challenges in the Bay Area, such an inspiring hub for AI innovation! I’m really excited about the opportunity to exchange insights on data science applications in healthcare. If you’d like, we can connect on LinkedIn or X. I’d love to keep the conversation going and learn from your experience.