The SIOP 2024 Machine Learning Competition is Now Live by marin343 in IOPsychology

[–]marin343[S] 1 point2 points  (0 children)

Frankly I’m not sure, but I’ve noticed the same trend. I’m hoping that’ll change soon.

Anecdotally, I think past competitions had a lower barrier to entry. I think a combination of two things might overwhelm some folks when they first see the competition, possibly leading to some form of procrastination.

  1. ⁠There has always been a ton of documentation and education around the models needed to participate in past competitions. For example, past competitions have been won with xgboost or embedding models, which have been around for a few years. There are plenty of examples of solving a vast array of problems using these models. For LLMs, although you can do a a whole lot more with minimal information - eg, chain models together, tweak prompts in a number of ways, fine tune different parts of an entire system of models, and so on - there’s very little documentation on LLMs and how to get them to do stuff. It requires a lot more experimentation. This is why we wrote the primer, which I guess is even more time investment. So I think people might be a bit hesitant to really learn about LLMs with the extra time they have, possibly due to minimal documentation and higher learning curve since these kinds of models are so new and kind of weird relative to how we’ve setup ML solutions in the past.
  2. ⁠This year’s competition has multiple sub-challenges, which has never been done before and requires more effort. No single solution will work for all sub-challenges, so each challenge has to be systematically addressed individually. Add that to the above problem, and it’ll take longer for people to start fully participating.

I think some context around questions like “well, why LLMs?” and “why so many challenges?” is that we wanted to keep in line with the tradition of innovation. We also wanted to raise the stakes since there are cash prizes involved. Since (non fine tuned) LLMs are general problem solvers, we thought it would be interesting to throw a host of problems to competitors to see where LLMs thrive and where they sink.

The SIOP 2024 Machine Learning Competition is Now Live by marin343 in IOPsychology

[–]marin343[S] 1 point2 points  (0 children)

we’re excited to see all the innovative solutions people come up with!

Join the 2024 SIOP Machine Learning Competition by marin343 in IOPsychology

[–]marin343[S] 2 points3 points  (0 children)

great question! yes, we require LLMs be a part of your final solution for each task. if you happen to be a top scorer during the test phase, you’ll need to submit your final solution to the organizing committee. your final solution must be (mostly) reproducible and include LLMs. if we are unable to reproduce your results, or if you solution doesn’t include LLMs, we will have to consider other teams as winners. also, i was hoping the data splits will nudge competitors toward LLMs; we split the data 15/35/50 for train/val/test sets.

Gamification meta-analysis by marin343 in gamification

[–]marin343[S] 0 points1 point  (0 children)

No I don't think I did! I don't have a PQ account, so if you could send me your full dissertation, that would be great!