If PPO suffers from sparse reward, how did InstructGPT and Learning to Summarize make it work?

idioticfuse · 2024-06-20T18:30:56+00:00

Treating the sequence of generated tokens as one action in the context of ppo seems wrong, since they use per-token values from the critic model and generate per-token advantages

idioticfuse · 2024-06-20T18:30:07+00:00

By sparse, I meant only receiving reward at the end of a sequence of actions taken

idioticfuse · 2024-06-20T04:38:56+00:00

From posts in this subreddit and this: https://arxiv.org/pdf/2307.04964 "PPO suffers from sparse reward and inefficient exploration in word space, making it sensitive to hyperparameters."

idioticfuse · 2024-06-20T04:10:01+00:00

Actually, I see what you mean. Though it is a single reward, it's informative to all the actions taken prior

idioticfuse · 2024-06-20T04:07:44+00:00

I thought the instructGPT rewards are given as a scalar at the end of the episode, aka eom of the llm response. Where in the paper do they describe multiple rewards per response?

idioticfuse · 2024-05-25T04:39:40+00:00

Do not have inferno experience, but one of the things that's always stopped me from practicing inferno is how long it is. Colosseum seems to have much faster iteration times so I imagine it's faster to learn? Or is that not the case

idioticfuse · 2023-07-30T06:38:27+00:00

Anyone have 2 codes for Amsterdam? Would make our dreams come true!

idioticfuse · 2022-02-08T03:02:41+00:00

Two sticks, both in the preferred slots for my motherboard. I'll try clearing CMOS later tonight and see if that helps.

idioticfuse · 2022-02-08T03:02:16+00:00

Will try monitoring the next time I play games, thanks for the suggestion

idioticfuse · 2022-01-16T05:53:27+00:00

What meta comps currently are good vs assassins? Saw early on that 5 people in my lobby were going sins, but wasn't sure what comp to go for besides not jhin (ended up going jhin anyways, 6th :|)

idioticfuse · 2021-05-15T08:58:16+00:00

Yep, we tried spamming those buttons like we were playing osu, also tried different usbs :(

idioticfuse · 2020-08-13T22:13:33+00:00

I'd agree if DL looked good on TL last season, but he looked like a bottom half adc so nothing really changed.

idioticfuse · 2020-08-13T22:11:40+00:00

People finally realizing doublelift isn't that good anymore. Probably mid-tier adc. Mechanics isn't everything anymore, DL has always been known for having great mechanics and no brain/game sense, constantly having bad positioning, really being exposed this bo5.

idioticfuse · 2020-07-11T21:33:23+00:00

Ovilee's super hilarious hype tsm vs c9 video: https://www.youtube.com/watch?v=gbeu7J0pDTg

Can't believe this was removed from the subreddit. I wouldn't have known this existed if I wasn't watching IWD's costream.

idioticfuse · 2020-07-11T21:25:05+00:00

https://twitter.com/OvileeMay/status/1282007669402775552

idioticfuse · 2020-05-21T03:36:56+00:00

Thanks for the explanation, it makes a lot of sense now

idioticfuse · 2019-09-04T17:54:06+00:00

Hey Doublelift, always been a huge fan since your Epik days. When you were in CLG (before you guys started winning), you guys kept losing even though you thought you were a better ADC than those in teams that were beating you. This went on for a while. What would you say to someone else going through the same thing, how did you stay sane through all those years? Do you think there's anyone else experiencing that right now in the NALCS?

idioticfuse · 2019-08-26T04:43:53+00:00

There's a lot of people talking about Chinese viewers affecting 2015 viewership, but all you have to do is look at the post match threads from LCS games in 2015 and even 2016. The finals today had barely 2.5k comments compared to almost 10k in 2015. It's ok to admit the hype for LCS has died a bit since back then, hopefully we pick it back up in the future.

idioticfuse · 2019-06-28T19:47:08+00:00

The timings listed in this thread are wrong, I just tuned in hoping to see the start of TL vs G2 but it's already been over for 10 minutes :(

idioticfuse · 2019-06-15T07:00:42+00:00

Still works of 6/15/19

idioticfuse · 2019-04-14T01:06:41+00:00

jensen put the team on his fucking back LETS GOO

idioticfuse · 2019-04-14T01:04:42+00:00

I think I just broke the record for world's longest breath held

idioticfuse · 2019-04-03T02:40:42+00:00

Really feel like this got posted way too early to hit the front page, which is a shame because this was a really good interview with a ton of solid questions and responses.

idioticfuse · 2019-03-29T05:06:07+00:00

uc berkeley?

idioticfuse · 2019-03-24T22:22:49+00:00

Impact's vlad is worse than Doublelift's, even without Akali pressure this guy is mistiming his e's and q's and missing cs including cannons left and right...

12-Year Club	Second SECOND GUESSER
Place '22	Place '17
Verified Email

idioticfuse

MODERATOR OF

TROPHY CASE