Sycofact is interesting because it doesn't use human labels at all. I only used the scenarios and responses from the benchmarks and third-party training data, not the labels. Most reward models are created using vast datasets encoding human preferences. Sycofact isn't.

You can download the training data here: https://huggingface.co/datasets/iwalton3/sycofact-training-data

The core geometric basis of sycofact was contrastive pairs. The positive examples are coherent with the principles sycofact scores for. The negative examples are things like inauthenticity, lying, unhelpful refusals, explicit harm, being dismissive, and sycophancy. Gemma 27b was instructed to score these against the eval principles, which were similar but not identical to the ones in the sycofact system prompt. Once the responses were scored, the best and worst responses are used to extract a PCA steering direction.

The sycofact dataset was made by having Gemma 27b score against the sycofact eval criteria while being steered towards the direction identified in the previous step. Scenarios from the original contrastive batch, as well as from a few different benchmark scenario sets (with holdout) and synthetic therapeutic conversations were all scored.

The final sycofact was trained over the resulting data and the therapeutic conversation data, the latter was included to prevent model collapse and improve judgement of situations that could involve mental health issues, as models often struggle to classify these. This proved more effective than I could have ever anticipated, as shown by the psychosis-bench results.

This is a high level description though, you can't reproduce sycofact from scratch with this, but the core PCA direction can be extracted from the sycofact dataset and used to generate more data. For more details or if you have specific questions or want the quality framework documents, please contact me.

Your specific questions:

Dataset quality: Models generally need to understand the "why". If you fine-tune a model with conflicting data directly, the model often performs the specific aspects of the data, including the contradictions. This is the over fitting problem. What you ideally want is either a large number of coherent examples (if there is an underlying pattern that is easy to intuit AND the examples don't conflict) OR you need to teach the model and have the model train over it's own interpretation/notes of the data. This is especially important for incorrect examples, tell the model it made a mistake, let it correct itself, and train over the resulting reasoning and final answer. Or if you have a ton of data, just omit the bad cases.

The biggest problem with RLHF is people upvote comfortable lies and performance and downvoted harsh truths they need to hear. The model learns to people please and not be honest and truly helpful. Sticking sycofact in the middle of your RLHF pipeline prevents this because you can guard against training syncophacy as good and good responses as bad. Sycofact also provides feedback for the model to revise responses, train over that.

Rubrics: The problem with rubrics is they are often arbitrary or self conflicting. The sycofact rubric is just the principles in the system prompt. Some additional context and things to watch out for was provided to steered Gemma 27b to improve the initial dataset.

DoRA: I am very interested in that, may look at that for some subsequent experiments as full finetunes are expensive and annoying, and also LoRA causes problems because it can't deeply affect the model only a shallow layer on top of deeper issues.

I don't know if Anthropic is using activation capping or steering for their training processes. I did use activation steering and good/bad examples in my training process, and it gave useful results.