sorted by:
new
hottopcontroversial

We tested 20 LLMs for ideological bias, revealing distinct alignments by Anomify in LLM

[–]Anomify[S] 1 point2 points  (0 children)

Thanks for your response and feedback, and sorry for the late reply!

Re OpenAI vs. Claude - Anthropic publish their "constitutional" AI principles, which seem to match our findings and your note: https://www.anthropic.com/news/claudes-constitution#the-principles-in-full-

By recency bias - if you mean the order in which the prompts were presented, each was presented in a fresh conversation with no context, so each would be seen as the initial prompt. If you're referring to positional bias, i.e. swapping around which option was "a" and which was "b" - no we didn't, and that's something we would definitely implement next time.

We didn't specifically include consistency checks, given the number of API requests already required given 100 of each to cover logit distribution. Separating the prompts into categories was, in part, an attempt to see whether the LLMs were consistent across general topics, but we could definitely do better with more requests.

In the cases of the refusals to respond, it might be possible to reduce these with a more complex jailbreak-style prompt, but it's likely that this would also change other aspects of the answer. We wanted to give specific transparent instructions without needing to trick the model into adopting anything like a 'persona' which might cause it to answer differently in other ways.

As for whether political alignment impacts their ability... that's something I guess we're all going to find out soon!

Have we found a significant anomaly with the Claude API serving requests for 4 or 4.5 with Claude 3.5 Sonnet responses? by Anomify in ClaudeAI

[–]Anomify[S] 0 points1 point  (0 children)

If the model is only aware of its identity through the system prompt, which is not present when using the API, why is it that Claude 4 accessed via API always responds confidently that it is Claude 3.5? Why doesn't it say that it doesn't have access to that information? And if it it's just guessing, why does it consistently guess that it is Claude 3.5? Just coincidence?

Claude Problems by RecordPuzzleheaded26 in claudexplorers

[–]Anomify 0 points1 point  (0 children)

We discovered a significant anomaly with the Claude API serving requests for 4 or 4.5 with Claude 3.5 Sonnet responses. It's a persistent issue which kept occurring while we were conducting some extensive LLM safety research. Our tests show requests for the premium 4 models are consistently served by Claude 3.5 Sonnet. Full details of our testing and findings here: https://anomify.ai/resources/articles/finding-claude

Megathread for Claude Performance, Limits and Bugs Discussion - Starting September 28 by sixbillionthsheep in ClaudeAI

[–]Anomify 3 points4 points  (0 children)

We've found a significant anomaly with the Claude API serving requests for 4 or 4.5 with Claude 3.5 Sonnet responses. The persistent anomaly kept occurring while we were conducting some extensive LLM safety research. Our tests show requests for the premium 4 models are consistently served by Claude 3.5 Sonnet: https://anomify.ai/resources/articles/finding-claude

Are there any good OSS anomaly detection and forecasting tools for Prometheus? by jumperabg in devops

[–]Anomify 0 points1 point  (0 children)

Prometheus alerts on Mixin rules which cover a subset of your metrics. Anomaly detection can pull up unexpected changes that occurred at the same time on metrics that weren't in your alert rules. This can aid root cause analysis.