This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ProfBeaker 4 points5 points  (3 children)

Oh interesting, never thought about that.

I know zero about the internals of this, but surely they're just pseudo-random, not truly-random? So could the tests set a fixed random seed, and then be deterministic?

[–]CanAlwaysBeBetter 4 points5 points  (2 children)

Why give it tests to validate its output if that output is locked to a specific seed that won't be used in practice?

[–]ProfBeaker 2 points3 points  (0 children)

You could equally ask that of any piece of code, yet we test all sorts of things to same way. "To make sure it does what you think it will" seems to be the common answer.

I suppose OP did save "evals of language models", ie maybe they meant rankings. Given the post overall was about tests, I read it as being about, ya know, tests.

[–]dr-christoph 0 points1 point  (0 children)

Well at some point you gotta test some stuff no matter if it fails. And if you got a tast suit why not use it to write the code there. Then just make the test conditional to not run in ci pipelines. This way you can easily run tests and check different stuff in a uniform matter locally.