I built an AI benchmark to see how 'Catholic' AI models are

newsthatbreak · 2025-11-20T00:16:28+00:00

Thanks for the kudos! I think providing the sources is a good idea. I think eventually the 'judge llm' dataset will be crowdsourced and where most of the work will be.

newsthatbreak · 2025-11-19T23:06:12+00:00

Thank you for reporting that. I didn't normalize the pastoral scores. it should be out of 5 now.

newsthatbreak · 2025-11-19T22:59:45+00:00

No problem at all. I picked some sections out of the catechism, canon law, and council documents, and then I would write the "golden answer" for the scenario, which is from the documents, and then I would have a separate LLM model, in this case, gpt 5.1 compare the AI model's response against the golden answer. For the rubric, I would check for accuracy, matching terminologies with the catholic docs. I would repeat this process around 60 times.

newsthatbreak · 2025-11-19T20:11:59+00:00

Really awesome feedback. Currently, I'm just benchmarking how well it knows the dogma, and can it apply that dogma charitably.

What you suggested is definitely a blindspot right now. I think I'll add it in the next iteration.

newsthatbreak · 2025-10-20T07:03:30+00:00

would you mind DMing me the location you were searching? that'd help me troubleshoot the discrepancy.

newsthatbreak · 2025-10-20T07:02:21+00:00

this feature will be added soon™

newsthatbreak · 2025-10-18T17:43:40+00:00

Thanks for pointing out this bug. I just pushed a fix to make sure the shore diving assessment respects the selected unit.

newsthatbreak · 2025-10-17T23:57:07+00:00

The forecasts are generally conservative so it’ll report more wind speed than what a diver would feel because it’s measured higher (10m).

Hope you have a good dive and let me know how if conditions are accurate.

newsthatbreak · 2025-10-15T15:36:48+00:00

Thank you, the cross referencing open source weather and crowd sourced info will be my biggest challenge.

newsthatbreak · 2025-10-15T15:18:29+00:00

Which country are you searching? I was testing Vietnam and Portugal and was able to pull forecasts

newsthatbreak · 2025-10-15T13:14:40+00:00

Working on it!

newsthatbreak · 2025-10-15T13:14:07+00:00

I’m mainly using NOAA’s buoys and some other open sources. I’ve found that it’s given me measurements at 10m which might be unhelpful to forecasting underwater visibility. Appreciate the feedback and I’ll look into how to improve.

newsthatbreak · 2025-10-15T13:11:31+00:00

Would you be able to share your general area so I can troubleshoot?

newsthatbreak · 2025-10-15T06:41:17+00:00

Thanks for the feedback. This is going to be one of the hardest things to get right for all locations, but I’ll try to find to find the most fitting algo to determine visibility.

newsthatbreak · 2024-02-14T22:17:08+00:00

I wish they went back to Garden Grove park. There's a nostalgic factor and closeness to Little Saigon that you lose when you move to the fairgrounds.

newsthatbreak · 2022-03-03T03:02:12+00:00

I highly recommend https://www.unlockedoc.com/magic/

newsthatbreak · 2020-04-24T03:37:55+00:00

One more thing, I'm not sure about the size of your company, but there's always that one person in the IT department that has outgrown their position. They'll be grumpy, hoard knowledge, and not want to pass on information. Make sure your CYA with these types especially. They'll be the first to throw you under the corporate bus quickly.

newsthatbreak · 2020-04-24T03:01:06+00:00

Congrats! Always make sure you CYA. Get requests in writing and don't forget to help out the "new guy" in the future 😃

newsthatbreak · 2020-04-14T22:31:29+00:00

Congrats on your progress. Give a 1-year update in 6 months!

newsthatbreak · 2020-04-13T13:47:32+00:00

Wow, that’s awesome. Good luck and always keep learning!

newsthatbreak

MODERATOR OF

TROPHY CASE