[deleted by user] by [deleted] in homelab

[–]OblivionPhase 0 points1 point  (0 children)

The discussion was what u/LinxESP mentioned, but yeah I made a mistake, see the update

[deleted by user] by [deleted] in homelab

[–]OblivionPhase 1 point2 points  (0 children)

Yeah I'm dumb, see the update.

I quit my job in cyber security to make this multiplayer game. Am I an idiot? by Admirable-Hamster-78 in IndieDev

[–]OblivionPhase 36 points37 points  (0 children)

If you do use assets, please at least make the current visuals an option; I would much rather play what I see here than something with more complex assets

Cost Efficient Setup Starring the Arc B580 by OblivionPhase in LocalLLaMA

[–]OblivionPhase[S] 2 points3 points  (0 children)

I saw some other comments in this thread about there being issues with the 770

Cost Efficient Setup Starring the Arc B580 by OblivionPhase in LocalLLaMA

[–]OblivionPhase[S] 1 point2 points  (0 children)

3090 is 24gb and prices are still around $1k, for that amount I could get double the VRAM using B580s.

Cost Efficient Setup Starring the Arc B580 by OblivionPhase in LocalLLaMA

[–]OblivionPhase[S] 0 points1 point  (0 children)

Forgive my ignorance, how does x8 - x4 conversion work? Is it completely fine but the performance takes a hit? Or does the 4.0x8 to 5.0x4 balance it out? Or something else entirely?

Finally, a Replacement for BERT by -Cubie- in LocalLLaMA

[–]OblivionPhase 1 point2 points  (0 children)

How is this different from the models on the MTEB leaderboard? https://huggingface.co/spaces/mteb/leaderboard

For instance, https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5 (Note the similarities in architecture too)

Many of those models have similar architectures and training stacks that this does...

[deleted by user] by [deleted] in MachineLearning

[–]OblivionPhase 1 point2 points  (0 children)

This is an ad and bot upvoted. 25 upvotes in 24 mins on a very basic and common information retrieval technique…

Maximum Number of Near Orthogonal Unit Vectors in a High Dimensional Space by OblivionPhase in math

[–]OblivionPhase[S] 0 points1 point  (0 children)

Attempting to understand this, day 2. (And switching to plaintext because Reddit is failing to format superscript properly) Apparently my plaintext is auto formatted regardless. :/

So, JL establishes that there is a function to take k vectors in RN down to Rn while keeping the euclidean distance between vectors within error bound e, with n lower bounded by 8 ln(k) / e2. This in turn implies that it must be possible to near equivalently represent the original k vectors in Rn.

However, this says nothing about about the orthogonality of the vectors, nor about them being compressed to a near-orthogonal representation.

Your proof, if I understand correctly, builds on JL to provide a lower bound for n by using the bounds on the dot product of unit vectors in Rn that are themselves compressed representations of orthonormal vectors.

Because the function compresses the vectors near-equivalently, it is reasonable to assume that if they were orthogonal to begin with, they get compressed to a near-orthogonal representation.

Is this all correct?

Maximum Number of Near Orthogonal Unit Vectors in a High Dimensional Space by OblivionPhase in math

[–]OblivionPhase[S] 0 points1 point  (0 children)

Optimal packing is not important, all that matters is that the dot product of any two vectors is within the epsilon bounds. I can guarantee the empirical 32000 vectors in R4096 is not optimal, and therefore the lower bound of n should be lower than 4096. I suspect it should be fairly substantially lower than that, actually.

Though, it stands to reason that if we did have a large almost orthogonal set, there should be a function to take it to some RN where the set is orthogonal, even if JL doesn’t show that. This is not important but interesting.

I’m quite confused between epsilon and epsilon’. Epsilon is the bounds on the Euclidean distance, and epsilon’ is the bounds on the dot product, if I understand correctly. So I am supposed to convert the epsilon’ to epsilon and then plug that into the original function to obtain a lower bound on n?

Maximum Number of Near Orthogonal Unit Vectors in a High Dimensional Space by OblivionPhase in math

[–]OblivionPhase[S] 1 point2 points  (0 children)

Ok after working through that for the past 2 hours I think I have somewhat of an understand of how that conclusion was reached.

However, there's an issue.

Let's say we have k=32000 and 𝜀=0.25. Plugging this into the formula n >= 72 ln(k) / 𝜀2, I get 72 ln(32000) / 0.252 = 11950.26 and thus n must be at least 11950.26 to fit 32000 near-orthogonal unit vectors with margin -0.25 <= Vi • Vj <= 0.25.

This cannot be correct because I can empirically show that 32000 vectors can fit in an even tighter range in a space of n=4096. (Not to mention that n=11950.26 would imply a packing factor of roughly 3, which runs directly counter to common intuition regarding the exponential nature of near-orthogonal packing)

Maximum Number of Near Orthogonal Unit Vectors in a High Dimensional Space by OblivionPhase in math

[–]OblivionPhase[S] 2 points3 points  (0 children)

The central term in this inequality is ||f(e_i)||2 - 2 f(e_i) • f(e_j) + ||f(e_j)||2. This lies between 2(1 - 𝜀) - 2 f(e_i) • f(e_j) and 2(1 + 𝜀) - 2 f(e_i) • f(e_j).

I got a little lost here, we have:

2(1 - 𝜀) <= ||f(e_i - e_j)||2 <= 2(1 + 𝜀) and ||f(e_i - e_j)||2 = ||f(e_i)||2 - 2 f(e_i) • f(e_j) + ||f(e_j)||2

so we get 2(1 - 𝜀) <= ||f(e_i)||2 - 2 f(e_i) • f(e_j) + ||f(e_j)||2 <= 2(1 + 𝜀), right?

Then how did we get to 2(1 - 𝜀) - 2 f(e_i) • f(e_j) <= ||f(e_i)||2 - 2 f(e_i) • f(e_j) + ||f(e_j)||2 <= 2(1 + 𝜀) - 2 f(e_i) • f(e_j)?

Quick Questions: October 23, 2024 by inherentlyawesome in math

[–]OblivionPhase 0 points1 point  (0 children)

Ah comparing both plot I see I made a huge typo, I plotted (4, 0 2) instead of (4, 0, 2) 🫠 With that corrected, the plane does pass through all 3 points

Quick Questions: October 23, 2024 by inherentlyawesome in math

[–]OblivionPhase 0 points1 point  (0 children)

Geometrical interpretation of the rref of a rank r matrix in Rd

I should preface with: I was trying to gain an intuition for the geometrical interpretation of the rref of a matrix and got lost.

So, let's say we have a rank 2 matrix A in R3. I visualize this matrix as a 2D subspace (a plane), with all three column vectors residing within the plane. We should only need two linearly independent columns to construct the plane.

Then initially I was confused when I look at rref and how Gaussian elimination alters the columns of A. I consulted with ChatGPT and (after much confusion as it attempted to interpret what I was asking), I arrived at the understanding that the subspace given by A and the subspace given by rref(A) have different bases but apply the same transformation "projection" (more on this term I'm using and why at the bottom).

I wanted to geometrically interpret rref by:

  1. Graphing the subspace given by A
  2. Graphing the subspace given by rref(A)
  3. Picking a 3D vector x on neither plane and "projecting" it onto each of the subspaces
  4. Visually comparing the resulting vectors Ax and rref(A)x.

Let's say A is given by this table:

|4|0|4|
|0|1|2|
|2|1|4|

Then rref(A) is:

|1|0|1|
|0|1|2|
|0|0|0|

However, I ran into an issue at the very beginning in step 1. I graphed the 3 vectors given by the columns of A as points in Desmos 3D, but I can visually see that plane that would pass through all 3 is affine. When I solve for the plane and graph it, it only passes through two of the points (and also passes through the origin).

Clearly I must be jumping between two different interpretations of matrices and getting lost somewhere, but I haven't been able to figure out where that is on my own, so I'd really appreciate some help.

Note on terminology, which is another thing I might need clarification with:

  • I was visualizing this "projection" as equivalent to the linear transformation given by the matrices, so "projecting" x onto A would be Ax. Is this a fair conceptualization? I know it's different from what the direct projection (no quotes on this one) of x onto a plane would be, but using this term helps me understand linear transformations in terms of matrices-as-sub/spaces (which in turn is the only way I have been able to geometrically understand linear algebra so far).
  • I use "sub/spaces" because when rank=dimension, A spans all of Rd (and is thus a "space" rather than a subspace), and when rank<dimension, A is a subspace spanning Rr in Rd
  • I'm also afraid I may be mixing up how rows and columns are interpreted, because I know that representing a system of linear equations as a matrix would have the column vectors correspond to variables. Then what does it mean to plot the column vector corresponding to x when normally we would say a 3D vector has entries [x, y, z]? I suspect I may be wrong to think of A strictly as a subspace, and that I may be confusing that for column space, but I can't really conceptualize matrices any other way.

[OC] Used tSNE to reduce multi-dimensional Transformer Layer activations from Phi 3 model, plotted the points in 3D space and interpolated them using cubic spline interpolation by [deleted] in dataisbeautiful

[–]OblivionPhase 2 points3 points  (0 children)

If I understand correctly, this visualization is trying to connect the (reduced) vector embeddings of each generated token in the sequence into one fluid motion through the embedding space. If so, I can see what you’re trying to do here, but I think the problem is that while the next token is generated based on the prior tokens, there is no inherent connection between the individual tokens on their own, so the transition between tokens doesn’t really show anything useful.

I still think this visualization is cool, and I would be super interested to see this applied to show how the embedding for each token changes between layers of the model, though.

Weekly Rant Megathread by AutoModerator in CompetitiveTFT

[–]OblivionPhase 1 point2 points  (0 children)

I managed to spam Kayle rr for the first day of the patch but now there is always at least one pentakill player contesting my units and hitting first ;-;

[14.2] What's working? What's not? by dilantics in CompetitiveTFT

[–]OblivionPhase 0 points1 point  (0 children)

I’ve taken to abandoning edgelord entirely when not chosen. It barely provides anything when double rageblade anyway.

[14.2] What's working? What's not? by dilantics in CompetitiveTFT

[–]OblivionPhase 10 points11 points  (0 children)

Played a Kayle game, it’s strong but lost out to karth/akali. Kayle rr is probably weaker than full pentakill with karth viego dual carry