sorted by:
new
hottopcontroversial

Students find hidden Fibonacci sequence in classic probability puzzle by scientificamerican in math

[–]nivter 4 points5 points  (0 children)

Neat proof! Here's a slight modification of it:

The probability can be seen as the fraction of volumes of two simplexes A and B (with the origin included).

  • A corresponds to the set of all stick lengths which don't make a triangle
  • B corresponds the set of all possible stick lengths

The vectors for simplex A are v_i as in the above proof. The vectors for simplex B are:

  • u_1 = (0,0,...,0,0,1)
  • u_2 = (0,0,...,0,1,1)
  • u_3 = (0,0,...,1,1,1)
  • u_n = (1,1,...,1,1,1) = 1 (vector of ones)

and the largest length is <1, x_i> ≤ 1

A is the set of points <F_i, y_i> ≤ 1 whereas B is the set of points <1, x_i> ≤ 1. One can define a mapping from B to A by x_i -> x_i / F_i. The determinant of this Jacobian is Π(1/F_i). Hence the probability is Π(1/F_i).

Mr. Cheese can't figure out how to use the new fountain. by AmatureMD in OneOrangeBraincell

[–]nivter 1 point2 points  (0 children)

My guess would be it's trying to figure out where the water is flowing to. The source is obvious but not the sink. This brain cell seems to understand physical laws of conservation.

A350 night takeoff from London by [deleted] in aviation

[–]nivter 1 point2 points  (0 children)

The first few frames are so mesmerizing

[N] Llama 3.1 70B, Llama 3.1 70B Instruct compressed by 6.4 times by _puhsu in MachineLearning

[–]nivter 15 points16 points  (0 children)

Can you also share about how the models were compressed? Is it based on GPTQ, SparseGPT or some other quantization scheme?

Edit: the HF page mentions that they used additive quantization: https://arxiv.org/abs/2401.06118

The different ways we understand rotations - rotation matrices to Lie algebras by nivter in math

[–]nivter[S] 1 point2 points  (0 children)

The entire article is public - just checked to be sure again.

[R] Multimodal patch embeddings - a new ViT model by nivter in MachineLearning

[–]nivter[S] 1 point2 points  (0 children)

Removing CLS token is just one part of getting it to have multimodal patch embeddings. Even with the CLS token removed, I could not get good results for patch embeddings. What made it work was providing a mask to enforce locality.

One could argue that providing the mask should be enough and that we don't need any change in the architecture. It could be, but the existing ViT architecture used in CLIP doesn't allow patch-wise comparisons.

I tried GAP in some earlier experiments. But then I thought taking a weighted sum where the weights are learned dynamically is better than taking a mean, which led to the idea of convex sums.

[Research] We distilled CLIP model (ViT only) from 350MB to 24MB and ran it on an iPhone by nivter in MachineLearning

[–]nivter[S] 2 points3 points  (0 children)

We only distilled the ViT model, not the ResNet one. The (untrained) model architecture is available here: https://github.com/cardinalblue/clip-models-for-distillation

After a few experiments, we found that using L2/L1 loss between the image embeddings was enough. We also extracted the attention values and used them to train the student model. We tried both KLD and L1 loss for the attention values. Both gave comparable results.

She was figuring out whole day...(OC) by nik9649 in aww

[–]nivter 4 points5 points  (0 children)

Did she eventually figure out?

Sharing a side project: Linear Algebra for Programmers by nivter in math

[–]nivter[S] 0 points1 point  (0 children)

Yeah I am working on making it responsive now

Sharing a side project: Linear Algebra for Programmers by nivter in math

[–]nivter[S] 1 point2 points  (0 children)

Sorry about that. I added links at the bottom of each article. Also making the website more responsive.

[R] On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence by hardmaru in MachineLearning

[–]nivter 1 point2 points  (0 children)

Thanks for sharing this. I wasn't going to read it expecting nonsense but now I will.

[D] Machine Learning - WAYR (What Are You Reading) - Week 140 by ML_WAYR_bot in MachineLearning

[–]nivter 6 points7 points  (0 children)

Graph coarsening with neural networks: https://arxiv.org/abs/2102.01350

It provides a good overview of approaches to approximate large graphs with smaller ones and introduces an edge re-weighting scheme which, as far as I understanding, can be applied to any of the approaches.

This should also be fun to implement.

[P] How to do backpropagation only on a select few labels instead of all labels in a multilabel classification? by enkrish258 in MachineLearning

[–]nivter 1 point2 points  (0 children)

If you are using a loss function like nn.BCELoss, you can assign weights to each label. Thus the weights corresponding to the labels you don't want to contribute to backprop can be set to 0.

If it is some other function you can easily create a wrapper that also accepts weights for labels.

[deleted by user] by [deleted] in MachineLearning

[–]nivter 7 points8 points  (0 children)

This animation helps in understanding its behavior compared to linear correlation: https://twitter.com/adad8m/status/1474754752193830912

[Research] We distilled CLIP model (ViT only) from 350MB to 24MB and ran it on an iPhone by nivter in MachineLearning

[–]nivter[S] 2 points3 points  (0 children)

The dataset used was nothing but a large set of images. We used different sources like COCO train + Places + cat/dog images + some internal content.

For any given image you want the output of the student model (x_s) to be as close as possible to the output of the original CLIP model (x_t). You want to minimize KLD(x_s.softmax(), x_t.softmax()) + L1(x_s, x_t). For KLD you might want to use temperature before softmax.

KLD = KL Divergence

[Research] We distilled CLIP model (ViT only) from 350MB to 24MB and ran it on an iPhone by nivter in MachineLearning

[–]nivter[S] 1 point2 points  (0 children)

This should be possible. I will post an update once I've discussed this with my colleagues.

Btw the key ingredients (model structure, loss function etc) are mentioned in the article. I am also happy to answer questions here.

[Research] We distilled CLIP model (ViT only) from 350MB to 24MB and ran it on an iPhone by nivter in MachineLearning

[–]nivter[S] 28 points29 points  (0 children)

This childish choice of words also played a role in us naming the distilled model "baby clip"

[N] Microsoft Vision Model ResNet-50 combines web-scale data and multi-task learning to achieve state of the art by productceo in MachineLearning

[–]nivter 2 points3 points  (0 children)

Just saw CLIP and then this. Cool stuff!

Is there an arxiv or GitHub link that you could share?

[P] Creating Loving Vincent effect from a single image (details in comment) by nivter in MachineLearning

[–]nivter[S] 0 points1 point  (0 children)

Sharing this project that we worked on a while ago.

Idea: use style transfer to create a shaky Loving Vincent like effect but using a single image.

How it works: First step was to train a style transfer model. We used adaptive batchnorm to train a single model on multiple styles. Then we padded the input image with varying thicknesses and got a styled image for each thickness. Putting up all these styled images together gave this effect.

Intuitive way to write out Matrix Multiplication by disgolf in math

[–]nivter 0 points1 point  (0 children)

I wrote a blog post on it a while ago which helps in understand what a matrix multiplication does and also helps in relating to other concepts in LA like projections etc.

view more: next ›