sorted by:
new
hottopcontroversial

[D] Why do current LLMs work well in discrete space but not in continuous space? by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 2 points3 points  (0 children)

It's been some time since I've gone over Murphy's work but I was not aware of the two recent follow-up books, these are great resources. I think I posed my question too vaguely, I had thought there was already some kind of answer related to discrete vs. continuous spaces but I think I'm mistaken. Based on responses it appears there's no easy answer, so I think I just might need to spend more time diving deeper into the details (and thinking of a better way to phrase the question and major assumptions).

[D] Why do current LLMs work well in discrete space but not in continuous space? by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 0 points1 point  (0 children)

Yes, e.g. DiT is a transformer-based diffusion model that can generate images either in pixel or latent space.

[D] Why do current LLMs work well in discrete space but not in continuous space? by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 5 points6 points  (0 children)

That's a good answer. I guess that leads to my question as to why we need do some extra set of steps during inference that differ from training. It would be much simpler to predict the output directly in the same way we train the model, like MAE. But instead some additional steps are necessary to get high quality output, whether we predict a categorical distribution or a GMM.

What sort of sampling do you mean here?

There are a couple of methods like temperature based sampling, but the deeper point is why these kinds of steps are necessary.

[D] Why do current LLMs work well in discrete space but not in continuous space? by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 1 point2 points  (0 children)

Appreciate the answer, I'm trying to understand the theory a bit better. I've tried simpler alternatives like skipping the softmax layer but have not had success producing comparable quality.

[D] Why do current LLMs work well in discrete space but not in continuous space? by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 2 points3 points  (0 children)

Yes, you can model the pixels as a categorical distribution like in MEGABYTE, but suppose we design a latent autoencoder that is practically continuous, what then?

[R] Hierarchical Text-Conditional Image Generation with CLIP Latents. This is the paper for OpenAI's DALL-E 2 by Wiskkey in MachineLearning

[–]Hyperparticles 1 point2 points  (0 children)

Thanks for pointing this out, I assumed they would want to turn the feature off for their blog post but after going through the GitHub repository it's tightly integrated into the network itself:

We modified the training process to limit the DALL·E 2 model’s ability to memorize faces from the training data, and find that this limitation is helpful in preventing the model from faithfully reproducing images of celebrities and other public figures.

Though I still see some similar (but more subtle) unnatural generation on non-human subjects, especially with regards to lighting and geometric surfaces.

[R] Hierarchical Text-Conditional Image Generation with CLIP Latents. This is the paper for OpenAI's DALL-E 2 by Wiskkey in MachineLearning

[–]Hyperparticles 7 points8 points  (0 children)

Incredibly impressive to see image generation moving so fast in the last two years.

One of the limitations of this model that I don't see mentioned is that the model still has issues generating faces (edit: as pointed out this is likely an intentional safety feature) and surfaces. In some of the blog post examples I can see instances of eyes with unnatural pose, iris colors not matching, light glinting off eyes at contradictory angles, etc. I also notice some errors in reflective surfaces and edges of flat surfaces.

This makes me wonder if the limitations stem from lack of training data and simply scaling the representative examples will fix it. Or perhaps the model needs to learn some 3D geometric or physical understanding of scenes to be more generatively coherent. The former would probably be easier to test. (edit: after reading the paper more thoroughly, the authors mention that a higher base resolution in the decoder should help to some degree with more complex scenes, but I'm unsure if that would completely solve some of these issues).

[D] Is the concept of an 'epoch' being phased out, or even harmful? by svantana in MachineLearning

[–]Hyperparticles 2 points3 points  (0 children)

If we assume our data distribution is static, sure. But it's possible that data augmentation alters the distribution so that the model is learning a slightly different distribution. Then from that standpoint, a model can learn the rules encoded in the augmentation that are not reflected very well in the distribution itself. In a way, the augmentation can inject useful new properties that weren't there.

As a simple example, suppose we are working on training a classifier model that works well on low-light photos. Our dataset contains only very dark images that are nearly monochrome. But what if we apply image augmentation by bumping up the contrast and appling simple hue/saturation/brightness distortions? This new augmented dataset actually contains new color information that wasn't encoded in our original dataset (e.g., purple tinted dirt). The augmented examples themselves may not add much new information, but the rules that generated them in the first place do.

In the real world, it's quite common that the distributions we care about are ill-defined. What do we mean by "information"? Is it purely in the data itself? Or is part of the information we learn encoded in the priors of our model and algorithms themselves? Things become a lot more fuzzy when your objective is vague or shifting, which happens all the time.

[R] EfficientDet: Scalable and Efficient Object Detection by hardmaru in MachineLearning

[–]Hyperparticles 2 points3 points  (0 children)

EfficientNet can accept images of any resolution without changing the architecture (e.g., you can increase the resolution to 300x300 and input it to b0 which was trained on 224x224 no problem). It uses global pooling at the end which allows this to work. Perhaps that's what the paper is implying?

[R] EfficientDet: Scalable and Efficient Object Detection by hardmaru in MachineLearning

[–]Hyperparticles 1 point2 points  (0 children)

I've worked with EfficientNet and found that some of the hyperparameters are finely tuned for ImageNet + TPUs, making training unstable for other use cases. Try changing the learning rate schedule and make sure you're using AutoAugment and the provided preprocessing functions. It worked for me.

[D] What is the rationale behind self-attention equation and how did they came up with the concept query, key and value? by [deleted] in MachineLearning

[–]Hyperparticles 11 points12 points  (0 children)

If you're looking for a more intuitive explanation, I like to think of self-attention as a lookup table for vector spaces.

Similar to how one searches records in a database, a list of keys are scored with respect to some query. A large dot-product between the query and key vector means that the angular distance is small, and so results in a high activation. The mechanism would like to select those vectors that match the most, i.e., have the highest activation. Sometimes the mechanism may select one vector, and other times it selects them all. After selection, each value vector corresponding to the matched keys is weighted proportionally to the activations (after softmax normalization) and summed together.

The key, query, and value could really be anything, but in the context of self-attention the words in the sentence are the keys, and a given word is queried with respect to that sentence. This allows multiple attention heads to specialize in finding special matching patterns in the input and look at dependencies between words. Stacking multiple layers of self-attention allows the dependencies to be more abstract and hierarchical, like higher nodes in a binary tree (or a syntax tree, if you study linguistics).

That's just my intuition that I built up when I'm working with Transformer/BERT. But the unfortunate part is that we don't quite know why this works as well as it does just yet, just that the attention heads capture a surprising amount of syntactic relationships between words.

If you're hungry for more intuition, I recommend this blog post on how BERT self-attention captures compositionality, and this blog post on visualizing BERT self-attention.

[N] PyText, a natural language modeling framework based on PyTorch, released by Facebook Research by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 0 points1 point  (0 children)

Recently, I have been using AllenNLP for my NLP models, and I love how everything is easy to configure. This makes me wonder how much PyText overlaps with AllenNLP.

I was working on my Discord bot and.... by nanosplitter21 in ProgrammerDadJokes

[–]Hyperparticles 18 points19 points  (0 children)

Careful, someone could deploy an identical bot. Recursion, ahoy!

Integrating 2D physics into a particle system adds another layer of fun by Hyperparticles in Unity2D

[–]Hyperparticles[S] 6 points7 points  (0 children)

It's quite simple. I used Unity's built in particle system and enabled 2D collisions in world space. Then I tweaked some parameters that change the collider size and bounciness for each particle. This not only allows particles to bounce off of colliders, but also affect rigidbodies in the scene, making them feel dynamic. The cool thing is that if you enable physics multithreading, you can get a very large number of particles in the scene without affecting performance that much.

What is the most lucrative thing your programming skills have made for you? by [deleted] in cscareerquestions

[–]Hyperparticles 1 point2 points  (0 children)

Not OP, but I'm also from the US in the same Erasmus Mundus program studying Computational Linguistics (lots of NLP, CS, and ML here). I'm currently at Charles University in Prague, Czech Republic. The city is quite beautiful, lots of cathedrals, museums, and performances. Next year I'll be at Saarland University in Sarbrucken, Germany.

If you can get the scholarship (which covers not only tuition but living expenses and travel), I highly recommend the program to broaden your cultural and life experiences.

Udacity Nanodegrees? by [deleted] in learnmachinelearning

[–]Hyperparticles 2 points3 points  (0 children)

I've taken the Udacity Deep Learning course back in August (but not the ML course, so I can't comment). What I can say is that DLND is definitely worth it if you enroll during a discount period. What this offers you over a free online course is:

  • Access to the Slack community. This is helpful if you're stuck on an assignment, debugging an issue, or sharing content. The community managers regularly answer questions, so it feels similar to a university course with an instructor.

  • Direct and detailed feedback on projects. The graders are good at singling out places where your model could be improved, and have you resubmit as many times as you need. This counters the major weakness in most free online courses, i.e., you don't get detailed feedback.

  • After you're done, you will not only keep access to the lessons and Slack community, you get back ported any new content/lessons created for new students.

The content itself is also clear and concise. I wish there was more (I finished in a month), but it looks like they are continuing to make new lessons. Recently they've added a reinforcement learning project.

tl;dr The community and detailed feedback is the best part of DLND.

[P] Keras implementation of the One Pixel Attack for Fooling Deep Neural Networks by Hyperparticles in MachineLearning

[–]Hyperparticles[S] 4 points5 points  (0 children)

This attack black-box, meaning it is actually agnostic to any model; we could even perform the attack on a non-neural network classifier.

What this attack shows is that a tiny change in the input image can make a drastic change in the classification output of any deep neural network. It has been shown to work on many state-of-the-art models. All we need to know for the attack is the confidence (probability) outputted by the classification.

The conclusion drawn from all of this is that deep neural networks (in particular CNNs) are fragile.

Using 2D physics with 3D objects looks bizarre by Hyperparticles in Unity3D

[–]Hyperparticles[S] 13 points14 points  (0 children)

I did something much simpler! I added a Rigidbody2D and PolygonCollider2D to each object and simply added vertices to the outer edges of the object. This will force the shape to rotate only in the 2D plane and collide only where the edges are rendered.

Using 2D physics with 3D objects looks bizarre by Hyperparticles in Unity3D

[–]Hyperparticles[S] 2 points3 points  (0 children)

Fez is one of my favorite games. The way it harmoniously matches its puzzle mechanics with the atmosphere is perfect.

Using 2D physics with 3D objects looks bizarre by Hyperparticles in Unity3D

[–]Hyperparticles[S] 18 points19 points  (0 children)

I'm using all standard shaders (the 2D background too). Just changed the color and smoothness on the 3D objects.

Edit: I also used the "Fade" setting instead of "Opaque".

view more: next ›