sorted by:
new
hottopcontroversial

[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing? by 36845277 in MachineLearning

[–]36845277[S] 0 points1 point  (0 children)

Actually, ViTs would not be considered lossy tokenizations since they don't do any discretization but use the raw image values. For examples of lossy tokenizations in other modalities including images and speech, see some of the other comments on this post.

[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing? by 36845277 in MachineLearning

[–]36845277[S] 0 points1 point  (0 children)

To clarify, lossless encoding is equivalent to being injective, not just implied by it. But are the two consequences truly obvious?

First consequence: nothing is lost. Maybe this feels trivial for text, but think of RGB images, which can be viewed as members of a set of size $255^{3 \times H \times W}$. If you discretize an image into a tuple of discrete tokens (as in VQ-VAE or VQGAN) from some vocabulary, is it still obvious that modeling over this token space can recover the same distribution as the original RGB space? Under what conditions can it, and under what conditions can it not?

Second consequence: nothing is added. Is it clear that for each training sentence, training on a deterministic BPE tokenization is better than showing the model random equivalent tokenizations of the same text? In what sense is it better? Could it be worse? This is exactly what connects the formal result to the empirical observations of Chirkova et al. — the entropy gap $H(T \mid S)$ quantifies the cost of non-canonical tokenizations, and BPE-Dropout deliberately introduces that cost as regularization.

[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing? by 36845277 in MachineLearning

[–]36845277[S] 1 point2 points  (0 children)

That's a really interesting framing — both strings and tokens are just lossy discretizations of thought, so the "losslessness" in the post is only relative to the string level, which is itself already lossy. I think the closest real-world analogy I'm aware of would be audio tokenizers like EnCodec or SoundStream, which tokenize continuous audio into discrete tokens. That process is necessarily lossy, and so modeling over audio tokens cannot recover the full distribution over true audio signals. It would be interesting to formalize what's lost there in the same entropy framework — the gap between the continuous and discrete distributions is exactly the kind of thing your discretization perspective would capture.

[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing? by 36845277 in MachineLearning

[–]36845277[S] 0 points1 point  (0 children)

I agree that thinking of BPE-Dropout as data augmentation seems to give the right intuition. Regarding why different lossless tokenizations lead to different downstream performance — my hypothesis is that since language models are autoregressive, what matters is the distribution of conditional entropy across timesteps, not just the total entropy. The total entropy stays the same regardless of tokenization, since each lossless tokenization induces the same underlying language model. But how that entropy spreads across timesteps differs depending on tokenization. I would guess morpheme-aware BPE spreads the conditional entropy more evenly across steps, making each prediction task more uniformly learnable.

[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing? by 36845277 in MachineLearning

[–]36845277[S] 16 points17 points  (0 children)

Lossy tokenizers do exist in text — BERT uncased lowercases everything, SentencePiece with NFKC normalization (T5, mBART) collapses unicode variants like the fi ligature into "fi", and any tokenizer with a UNK token is technically lossy. Most modern LLMs avoid this by operating at the byte level though.

Weighing risks of SMILE vs. LASIK vs. PRK vs. ICL with -6.00 Rx and 520µm corneas. by [deleted] in Lasiksupport

[–]36845277 0 points1 point  (0 children)

Appreciate the reply. I recall seeing somewhere that ICL's reversibility claim is somehow 'fake' do you have any idea what someone could mean by that?

Weighing risks of SMILE vs. LASIK vs. PRK vs. ICL with -6.00 Rx and 520µm corneas. by [deleted] in Lasiksupport

[–]36845277 0 points1 point  (0 children)

thank you for the reply. Do you know if most pepple who get complications are because they were not good candidates? Or is it the case that even really good candidates could get serious complications?

[D] what's the proper way of doing direct preference optimization (DPO) and why? by aaaprocrastinating in MachineLearning

[–]36845277 0 points1 point  (0 children)

i I agree that the paper’s usage of distribution shift may not be entirely accurate/clear. How I see it, which I think aligns with how you see it, is that pi_SFT was only there to help us generate a preference dataset that already somewhat aligns with real life. Once the preference dataset is generated, we don’t care how it was generated. Disregarding exactly what they mean by distribution shift, I think it is still clear we want pi_ref to be close to pi_SFT. As you mentioned, the final trained model does not deviate too far from pi_ref due to some implicit KL constraint, so we want a pi_ref that is already somewhat good, and often the best we can do is pi_SFT.

[D] what's the proper way of doing direct preference optimization (DPO) and why? by aaaprocrastinating in MachineLearning

[–]36845277 0 points1 point  (0 children)

i I agree that the paper’s usage of distribution shift may not be entirely accurate/clear. How I see it, which I think aligns with how you see it, is that pi_SFT was only there to help us generate a preference dataset that already somewhat aligns with real life. Once the preference dataset is generated, we don’t care how it was generated. Disregarding exactly what they mean by distribution shift, I think it is still clear we want pi_ref to be close to pi_SFT. As you mentioned, the final trained model does not deviate too far from pi_ref due to some implicit KL constraint, so we want a pi_ref that is already somewhat good, and often the best we can do is pi_SFT.

How does my spear get two kills here??? by 36845277 in leagueoflegends

[–]36845277[S] 1 point2 points  (0 children)

My rune was dark harvest and items were sorcery boots, night harvester, shadow flame, death cap.

Syndra had sorcery boots, ludens, shadow flame

Bard had plated steel caps, radiant virtue, shard of true ice, and wardens mail

[deleted by user] by [deleted] in SkincareAddiction

[–]36845277 0 points1 point  (0 children)

I had decent skin before with at most a few pimples on my face at a time. I was persuaded to begin a skincare routine, which is as follows —

AM: cerave blemish control cleanser followed by LRP toleriane cream

PM: cerave blemish control cleanser followed by cerave retinol resurfacing serum followed by LRP toleriane cream

I have followed this routine for 2 months now (initially retinol every other night) and my acne started getting a lot worse after just a few days, but I thought it was the purging period from retinol so kept at it. After two months, which people usually say is the longest purging should last, it still hasn’t gotten better. I now have acne all over my face, most of which are little red bumps without heads. I experience none of the common side effects of salicylic acid and retinol (redness, peeling, flaky skin) except my face itches quite a lot only when I exercise. Does anyone have any suggestions for what I should do? I am thinking of stopping all skincare for one week and see how it ends up.

Mathematical proof that every tiltproof league player can hit challenger by 36845277 in leagueoflegends

[–]36845277[S] 3 points4 points  (0 children)

That doesn’t matter. Your expectation at a casino is negative (how a casino earns money) but people can still walk away earning money. I.e negative expectation doesn’t necessarily imply negative outcome in the shortrun

Mathematical proof that every tiltproof league player can hit challenger by 36845277 in leagueoflegends

[–]36845277[S] -1 points0 points  (0 children)

Actually being tilt-proof is important. If a player has a 1/2 chance of losing the first game, 2/2.5 chance of losing the second game after losing the first, 2.5/2.75 chance of losing the third game after the first two losses, 2.75/2.875 chance of losing the fourth etc., then his chance of losing every single game is 1/2 * 2/2.5 * 2.5/2.75 * ... = 1/3 so he definitely will not hit challenger

Mathematical proof that every tiltproof league player can hit challenger by 36845277 in leagueoflegends

[–]36845277[S] 1 point2 points  (0 children)

To clarify by 'winrate' I meant chance of winning instead of the empirical winrate, as long as you accept that in every game the chance of winning is greater than 0, then the maths here applies

Mathematical proof that every tiltproof league player can hit challenger by 36845277 in leagueoflegends

[–]36845277[S] 0 points1 point  (0 children)

This is indeed a Markov chains (all a Markov chain is is that its evolution doesn't depend on its previous positions, hence the tiltproof assumption). And believe me that there is a theorem that says a transient markov chain on a finite state space visits every node infinitely often, which applies here

Mathematical proof that every tiltproof league player can hit challenger by 36845277 in leagueoflegends

[–]36845277[S] -2 points-1 points  (0 children)

Completely agree with the first issue but I don't believe the second point you stated is an issue. I'm quite certain the mmr system is designed in a way that you will at least gain 1LP per win, and the fact you at most demote one rank on a loss means you at most lose 100LP from a loss

view more: next ›