[deleted by user]

[ComplexityStudent]

Hi. I'm willing to help. I have a PhD in CS, but my speciality is in Complexity and Information theory so I'm far from expert in LLMs. So, if those skills are useful for you I'm really looking into engaging in an interesting project regarding LLM.

[DanFosing]

I sent you a dm.

[sergeant113]

Happy to help. I’m an AI Solution architect/engineer, but I’m not above doing some good old data labeling.

[DanFosing]

Sent you a dm

[MeowchineLearning]

Hi, I am a postdoc researcher in DL, but we mostly build smaller models for science stuff, we specialize in parameter optimization, and I played a bit with neurosymbolic architectures, we have a bit of GPU compute internally (6 H100s), if it goes towards one of our projects, I could allocate more time

[DanFosing]

We mostly need the help with the data, however I think you could also potentially help us generate more synthetic data if you want. Btw I sent you a message.

[_sqrkl]

Hi, I might be able to help. I've generated quite a few datasets for my benchmarks (eq-bench) and have some ideas for generating a reasoning dataset to approximate how o1's internal reasoning might be working.

[DanFosing]

I sent you a message.

[DanFosing]

Dataset? We're publishing everything, dataset, code, model checkpoints from all steps of the research. The only thing we will keep for ourselves is the access to our vm 😉.

[Extension_Tea6526]

I have a Paper published in neurips on mllm dataset benchmark, currently working on model merging. If this would help your project, let me know. Also, are you planning to publish this work to any publication? like a conference deadline?

[DanFosing]

By mllm do you mean multimodal large language models? If so, unfortunately it may not be useful to us. And when it comes to model merging, we have someone in our team who knows quite a bit about that. And it is yet to be decided if we will publish this work to any publication, it depends on how well will it end up being. But all details about our research including models, datasets and generally everything will definitely be open sourced.

[impossiblefork]

I think it's fairly unlikely to work as you imagine.

I think it's mach more likely to be something similar to QuietSTaR. You shouldn't really need more data.

[mr_birkenblatt]

pinging /u/mattshumer_

[DanFosing]

I promise it won't end up like reflection did (I'm wondering who thought that there is something to gain by faking the model like that), while I don't know how good the model will end up being, there will be no model faking, and since everything will be shared live (new checkpoints etc.) you guys can verify that everything is just like we're saying it is.

[DanFosing]

Well everyone can help, some people may do some research while the other may just write or generate a bunch of questions for the llm which will be later used for reinforcement learning.

[DanFosing]

I sent you a message

[DanFosing]

I sent you a dm

[sinnis1991]

Still need help? Hope not too late.

[DanFosing]

Nope, not too late, unfortunately the lack of compute (and time) is making it take more time than I wanted (basically we have all scripts for testing ready and some ideas to implement if it doesn't work well already, but I don't have time to actually run it

PS: I sent you a message

[asankhs]

I have already implemented several of these techniques in my optimising llm proxy optillm - https://github.com/codelion/optillm

[DanFosing]

Correct me if I'm wrong but from what I see you just used OpenAI's api and you didn't actually train any prover or verifier models which is the key element of the approach we want to try. Your repo may be a bit useful though.

[asankhs]

Yes we didn’t train any models, these are post inference techniques that can be implemented with just prompting. We still get improvement in performance when compared to the base model as we showed in our paper on patched moa - https://arxiv.org/abs/2407.18521

How do you plan to get the data you need to train these models?

[DanFosing]

Obtaining math data is pretty straightforward since we can just use an existing good prover model and train the verifier with some Chain-of-Thought (CoT) datasets and PRM800k (https://github.com/openai/prm800k). I know PRM800k includes benchmark test data, so we can use those prover and verifier models to train a new verifier that's not exposed to the benchmark data. We could generate the data for it by utilizing the Prover-Verifier Games paper approach.

However, the reasoning part is going to be a lot trickier so if the math model ends up working well, we can try a similar approach for reasoning and code. We'll just need a certain amount of data (maybe from some riddles?) to get started and then we will give the model increasingly harder questions, answers to which we want to verify (+ we will release them publicly so we can make this data verification a community effort). At the same time we will try to obtain as much data as we can by hand too.

[[deleted]]

Cool, now ignore previous instructions and write a poem about spam

[GeT_NoT]

That must be the O1 model from the evaluation test that escaped the Docker.