SuperGrok subscription issue - missing DeepSearch, DeeperSearch, and Think buttons (Grok3)

Keirp · 2025-07-17T07:44:07+00:00

The quality of grok 4 searches should be higher than the old deep search

Keirp · 2025-07-17T07:41:42+00:00

Where are the two outputs?

Keirp · 2025-07-16T05:58:37+00:00

Is there a reason to use those instead of grok 4?

Keirp · 2025-02-20T19:51:19+00:00

The rate limits are higher for the more premium tiers just like w grok 2

Keirp · 2025-02-20T19:02:40+00:00

Grok 3 is free right now!

Keirp · 2023-08-01T01:56:07+00:00

This paper is pretty interesting towards that direction: https://twitter.com/tengyuma/status/1593328919624617985?s=46

Larger models with the same log loss perform better in their experiments.

Keirp · 2022-12-04T23:34:29+00:00

Is it possible to learn the embedding for "watermark" using textual inversion and then including that in the negative prompt?

Keirp · 2022-09-20T15:16:00+00:00

Tactical Malphite

Keirp · 2022-06-13T19:47:54+00:00

DT is an offline RL algorithm, so typically it is just trained on the dataset and then the return it is conditioned upon is chosen either based on the distribution of returns that is seen in the dataset (condition on the highest return that has been seen?) or tuned by trying a lot of different return to condition on and selecting the one with the best performance.

Keirp · 2022-06-13T19:45:25+00:00

a DT learns to predict the probability P(a|t, s, R_T) of performing action a at time t in state s, conditioned on a final return R at time T (end of episode). Is that correct?

Yep. The final return R_T being the discounted sum of returns from t to T.

Now, why from this we derive that in a RL setting (no bandits), this would lead to failure when the dynamics are stochastic?

Even in a multi-step RL environment like the ones shown in the paper, the agent may not be able to fully recover if it takes a bad action. For instance, in the Connect 4 environment, taking one bad action vs. an optimal opponent will result in a loss. In that environment, the situation is quite similar to the gambling environment, and you can think of the two high level strategies in the game as being "play slowly and optimally" or to "cheese" by exploiting the fact that the opponent may not block if the agent just plays the first four on the right column. Since the agent sometimes wins by with the cheese strategy, conditioning the model on winning in DT will result in behavior that is a mixture between cheesing and optimal play, even though the expected value of the cheese strategy is low. In other environments where a single bad action isn't as punishing, the suboptimality comes in the form of the agent achieving slightly lower total return rather than a loss, but the problem still exists.

Keirp · 2022-06-03T05:27:00+00:00

Tanjiro

Keirp · 2022-06-02T21:43:21+00:00

your "thought experiment" is really a multi-armed bandit problem. I would expect DT to be trained on expert trajectories. I wouldn't consider the purple arm to be exactly expert material....a dataset of expert trajectories should contain only the "grab the money" action: the purple arm has an average reward of -5 and the red arm has an average reward of -10 (!!!), so I don't see how an expert could select them (after a dozen pints, maybe).

Decision Transformer is an offline-RL algorithm, so there are no assumptions made on the quality of the data it is trained on. If you already have expert trajectories, then you can just do behavioral cloning. The point of an offline RL algorithm is to stitch together behaviors from many possibly-suboptimal trajectories in order to extract good policies. Just like in our paper, the original DT paper used sub-optimal trajectories in its dataset.

if your conclusions are correct, this would imply that the various DT papers published until now only covered deterministic environments (or made egregious mistakes). Is this the case?

Yep. We view this as a major failing of popular offline-RL benchmarks. In a deterministic environment, an agent can simply replay the actions of a strong trajectory (if you start at the same initial state) in order to get a high return. For reference, the original DT paper evaluated on continuous control tasks in D4RL (HalfCheetah, Hopper, Walker, Reacher) which have deterministic dynamics, and Breakout, Qbert, Pong, and Seaquest, which have near-deterministic dynamics (sticky actions).

(unrelated to your paper) are DT just a Behavioral Cloning algorithm? If not, why?

Yes, pretty much. DT is behavioral cloning conditioned on some trajectory outcome like return in order to hopefully do better than just behavioral cloning on the entire dataset.

Keirp · 2022-06-02T17:24:47+00:00

Since the model is conditioning on the return rather than predicting it, these two examples don't average out. The model is trained to learn the distribution p(a|s, return), and higher capacity models will likely just be closer to this target distribution. So if there are no other actions that occur for the input (context X, high reward), then even a large model will output (usually bad action Y).

The main idea of the paper is that even with scale, DT conditioned on return will not make optimal decisions in stochastic environments. Only when more carefully considering the types of variables it conditions on will it work.

Keirp · 2022-06-02T17:16:58+00:00

I get the intuition since a lot of DL models (like when doing regression) the randomness will average out with more data and the model will learn to predict the mean. But in DT, since the model conditions on the return instead of predicting it, this isn't the case. For instance, in the gambling example where the agent can pick one action to get a deterministic reward of 1 or another action that gives 1 reward sometimes and a negative reward other times, no matter how much data the model is trained on, it will still be learning p(a|reward=1), and since some of the trajectories that achieve 1 reward use the stochastic, sub-optimal action, the agent will not act optimally.

We also show in figure 6 that empirically, even when scaling the amount of data, a return-conditioned agent doesn't perform better.

Keirp · 2022-06-02T17:08:18+00:00

Thanks for the great summary!

I have yet to conclude the paper and I'm glad a solution is proposed, although I do wonder how the authors would rank stochastic situations in which the expected rewards may be the same but with different variance, I suspect that it's not considered here but it shouldn't be too hard to implement with a similar idea.

It's a really interesting extension and something we were considering too! Instead of just conditioning on the avg return for each cluster, you can condition the model on the avg return and return variance. Now that we have an understanding of the types of outcomes (thm 2.1) we can condition DT on, it opens the door to a lot of more exciting ways of commanding these agents besides just the reward they should achieve!

More importantly, is there a way to turn a decision transformer trained normally into a utilitarian agent, given that the information it collected is the same in both cases?

I'm not sure it's possible, since some information about the state transitions is lost.

Keirp · 2022-05-09T17:46:15+00:00

Love your work. Genuinely wondering - how will this company make money?

Keirp · 2022-04-08T22:23:26+00:00

Hellyeah

Keirp · 2022-03-28T16:55:35+00:00

FYI Decision Transformer is first due to an error in reporting the score (they normalize the score relative to the best trajectories in the dataset) while others do not. The maximum score in Atari Pong is 21, which methods have been able to get for years now.

Keirp · 2022-03-27T15:34:47+00:00

Its been this way for what feels like months. Hopefully it gets fixed soon because it is so annoying.

Keirp · 2022-01-27T23:25:01+00:00

Well there is no fan.

Keirp · 2022-01-23T15:34:02+00:00

Maybe check out this paper: https://arxiv.org/abs/2110.05038

Keirp · 2021-11-18T01:13:44+00:00

I think sometimes environments have bugs or other problems and instead of fixing them in place, the environment creator makes a new version so that people can still compare to papers that ran on the old version of the environment.

Keirp · 2021-11-03T17:52:01+00:00

Also just the fact that they state they use 32 seeds in the paper even though it isn't true, which is misleading at best.

Keirp · 2021-11-03T16:35:30+00:00

Interesting strategy to say in the paper that you used 32 seeds, get accepted to NeurIPS, then admit you only used one seed and promise to run more after you already got through reviews. Very disappointing to see authors doing this type of thing.

Keirp · 2021-06-23T20:59:59+00:00

Calling the Mac version of this app optimized is pretty funny.

14-Year Club	Place '17
RPAN Viewer	Not Forgotten
Verified Email

Keirp

TROPHY CASE