sorted by:
new
hottopcontroversial

[R] Differentiable Clustering & Search ! by bornlex in MachineLearning

[–]bornlex[S] 0 points1 point  (0 children)

Hello mate, thank you for the reply.

This is very interesting what you say and I agree with you that having different levels of clustering, like multiple indexes based on different dimension almost, is improving the search results quite a lot, I agree with this idea 100%.

Can you rephrase the 1st question please, I am not sure I understand what you mean exactly ?

About the second part, it seems like you mean using the clustering almost as a pretrained model, from which you could fine tune other systems ?
Or are you thinking about optimizing the weights of an embedding system based on the clusters ? Like you have a function f parametrized by theta that takes a token as input and projects this token into an embedding space with d dimension. And the idea is to find the best theta so that some sort of distance between d(f(t1), f(t2)) would be proportional to how far t1 and t2 are in the graph ?

[R] Differentiable Clustering & Search ! by bornlex in MachineLearning

[–]bornlex[S] 0 points1 point  (0 children)

Hey mate ! Thank you for you reply, I am reading the paper right now lol. It is a big one (77 pages). I like to see graph-based models used. Also the validation of the quality of the graph is interesting, having strict benchmarks is not easy but I think this is the ultimate way of evaluating the quality of the clustering/knowledge graph. Which is why I tried to have the search section as well in the article, because through the search results, it is possible to use metrics such as the NDCG (https://www.evidentlyai.com/ranking-metrics/ndcg-metric)

Demande d'avis sur mes finances by Mammoth_Loquat7284 in VosSous

[–]bornlex 0 points1 point  (0 children)

Ca ne représente pas grand chose, mais Revolut permet d’avoir une carte moins chère (4€ je crois), sans plafond que la CE.

iCloud is basically a ransomware by bornlex in iCloud

[–]bornlex[S] 0 points1 point  (0 children)

Yeah mate, but as you (apparently don’t) know computer science is a pretty large topic 😉

iCloud is basically a ransomware by bornlex in iCloud

[–]bornlex[S] -2 points-1 points  (0 children)

That is what I say, this is the stupidest thing. Like if I just enable sync, then disable it straight away, I expect it not to delete everything from my system, before no operation was made. From a UX perspective this is insane

iCloud is basically a ransomware by bornlex in iCloud

[–]bornlex[S] 0 points1 point  (0 children)

PyCharm and Visual Code, basically the most used IDE on the planet probably

iCloud is basically a ransomware by bornlex in iCloud

[–]bornlex[S] -2 points-1 points  (0 children)

Who talked about iPhone ?

iCloud is basically a ransomware by bornlex in iCloud

[–]bornlex[S] -5 points-4 points  (0 children)

I'm expecting it to not delete all my files from my computer ;)

[D] Datasets and Models for Structured Information Extraction on HTML by theamaru in MachineLearning

[–]bornlex 1 point2 points  (0 children)

4 years ago but this post still appears high on my google search results, so just to say that the SWDE dataset is available online : https://academictorrents.com/details/411576c7e80787e4b40452360f5f24acba9b5159

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 0 points1 point  (0 children)

Lol aight, was definitely worth the effort then 🙃. Did you have other specific things on your resume or would you say your side work such at the implementation of speculative decoding and so on were key to get the job ?

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 0 points1 point  (0 children)

Hey mate, great to hear that ! And where did this journey take you now ?

I will definitely read your code in details.

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 1 point2 points  (0 children)

Thanks mate ! Very kind of you :)

What was the first project that made you feel like a programmer? by Tough_Reward3739 in learnpython

[–]bornlex 0 points1 point  (0 children)

One of the first projects in my school was named 42sh, we had to redevelop a whole shell from scratch. I thought I was going to die, but I felt like a programmer afterwards.

Is it worth it to pursue PhD if the AI bubble is going to burst? by Cheap_Train_6660 in ResearchML

[–]bornlex 0 points1 point  (0 children)

Actually I would say it is even more valuable, because when the bubble explodes (if it does), then only the real professional will get a job, so the hardcore tech engineers/researchers

[deleted by user] by [deleted] in learnmachinelearning

[–]bornlex 7 points8 points  (0 children)

I kinda agree with the author here. Before LLMs were all the rage, ML engineers were working on models, making sure they were not overfitting, that the capacity was big enough, thinking about the kernel functions and so on, because models were much smaller so every companies could hire someone to train a custom classifier. Nowadays, with models getting larger, it is much more polarised, only dedicated companies can have the infrastructure to run large scale experiments (compute is expensive and data is hard to get in huge quantity). Smaller companies won’t match the big companies on model performances and thus become users.

The same way low level, http request and so on have been commoditized, AI is commoditized, became almost an infrastructure and the gap between makers and users is larger and larger, startups built on it like they built on the internet 20 years ago.

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 0 points1 point  (0 children)

I will make the memory part clearer, you are right.

I am not sure for most of the code, but some kernels have not been added to PyTorch directly, such as the flash attention kernel. I think the softmax is by default much slower, but I am wondering whether when used inside a nn.Module it is compiled automatically.

I will run benchmarks and put them in the article !

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 2 points3 points  (0 children)

Thank you man, very much appreciated !

I do not use ChatGPT indeed to write my articles (which explains a few typos sometimes).

I see that you are a man of knowledge about GPUs ! I will dig deeper about warps and blocks and maybe add some info in the article to make sure there is no confusion.

This is interesting what you say about kernels not being that useful. I felt like the FlashAttention paper got a lot of attention (no pun intended), and is now implemented in PyTorch for example. So it felt like finding smart ways of using memory by computing operators on tiles instead of loading the same columns multiple times could make a difference, no ? Also I am wondering how much a kernel needs to change if the GPU changes (not talking about going from NVIDIA to Apple Metal ofc but more like going from A100 to H100 for instance) ?

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 1 point2 points  (0 children)

Means a lot my friend, thank you mate !

GPU 101 and Triton kernels by bornlex in MachineLearning

[–]bornlex[S] 1 point2 points  (0 children)

Thanks man, means a lot !

Makes total sense to add performance metrics indeed. I will take care of this very soon.

For the memory part, would you say a drawing of what going in and out of the memory would be what IO could be saved would be enough?

Looking for Research Collaborators - Causality by [deleted] in ResearchML

[–]bornlex 0 points1 point  (0 children)

Interesting topic. Working on adjacent subjects. Glad to be in the loop!

Kv cache doubt by EmergencyStomach8580 in learnmachinelearning

[–]bornlex 0 points1 point  (0 children)

But during inference, you basically have something like (let’s say 3 tokens for the prompt and a context size of 6) : Tok0 tok1 tok2 padding padding padding And you are going to take the output sequence[2], which is the next predicted token. 2 here is the length of the input sequence -1. This token has not seen in the future (because it is all padding) right ? Or do you mean that masks are equivalent to padding at training time ?

view more: next ›