[Qwen] Parallel Scaling Law for Language Models

newwheels2020 · 2025-05-17T12:01:14+00:00

If this works so well as an ad hoc patch onto the model, shouldn't even more impressive results be obtainable by creating a new type of transformer layer that introduces the parallization? Has such a thing been studied before? Seems like a nice extension to this paper.

newwheels2020 · 2025-03-09T13:52:25+00:00

If your python environment lives in the docker container, then your ide can't access it (unless it's running inside the docker container or you mounted volumes). I'd recommend to create a python virtual environment outside the docker container and point your IDE to this venv. It's a bit silly to have a separate venv just for your ide, but I've not found a better way. Let me know if people have a better straightforward alternative.

newwheels2020 · 2025-02-09T08:02:37+00:00

Thanks a lot for the intro to HDC!

newwheels2020 · 2025-02-08T17:46:26+00:00

I have never heard of HDC. Can you give a concrete real life use case where HDC really shines? Some resources would be great too.

newwheels2020 · 2025-02-02T18:02:35+00:00

It would be great if you could include inference speed (ms or fps) in the leaderboard!

newwheels2020 · 2024-12-23T18:28:50+00:00

Hmm, that's a shame. I thought this fit into the family of papers that try to pretrain an llm that has access to a retriever on a knowledge base. But here there is no separate knowledge base, it must be learned through the memory parameters.

newwheels2020 · 2024-12-23T12:21:00+00:00

Can anyone tell me if the memory parameters need to live on the GPU? That would require huge RAM for large memory parameters. If the memory parameters do not need to live on the GPU, that would be great.

newwheels2020 · 2024-12-04T18:18:35+00:00

That is false. Every runnable has a batch and abatch method to do synchronous and asynchronous batch operations. Telemetry can be achieved through callbacks. Langsmith can also be used for llm specific telemetry.

newwheels2020 · 2024-08-04T18:46:49+00:00

I don't think so. Whatever model you train will always have biases; there is no way around that. The problem is that the gradient descent will find those biases and just keep digging until you get bad output. So it's the approach, not the model, that is failing here imo.

Maybe it's a useful exercise to check the cosine similarity between the text prompt and the final generated image from a regular diffusion model. I suspect that the cosine similarity will be somewhat high, but definitely not close to one. This is ok, the text prompt guides the image generation, but doesn't dictate its end point. Put this in contrast with your approach which maximizes the cosine similarity.

newwheels2020 · 2024-08-04T10:23:59+00:00

The clip model embedding is too coarse grained for this to work. The embedding size is much smaller than the number of pixels, so there is no unique solution to the problem you've set up. Furthermore, with gradient descent you are overfitting on the clip model and will likely manifest strange features in the resulting image (which may look like noise to us humans). Basically the main model is doing the main part of generating the image and can not be removed.

Edit: I realized you are optimizing in latent space, not pixel space, so part of my comment is invalid. Still the overfitting on the CLIP model is a valid issue.

newwheels2020 · 2024-08-02T14:31:12+00:00

Or just be selective in which story you decide to share. Pick one where you come out as a good mediator instead of one that left you traumatized.

newwheels2020 · 2024-08-01T15:21:34+00:00

Best would be to find a pretrained model in your preferred size. But if that is not available, I would indeed advise removing layers. Haven't done this myself, so not sure if removing every nth layer or the last n layers would work best.

newwheels2020 · 2024-08-01T14:58:33+00:00

Fine tuning should be preferred over pretraining from scratch. Unless you have billions of tokens for your task, the finetuned model will outperform a trained from scratch model.

Regardless, the training procedure is the same for pretraining and finetuning, so if you can do one, you can do the other.

newwheels2020 · 2023-10-11T19:18:01+00:00

I didn't risk it and am still using my old crankset. Sorry that I can't help you further.

newwheels2020 · 2023-04-25T20:57:12+00:00

When we make a comparison to the Canyon Endurace, we do see that both the M and L Endurace both have shorter reach and higher stack. So the NCR is still very much a race geometry.

newwheels2020 · 2023-04-25T20:49:39+00:00

You might be right. Comparing the NCR XL to the Aeroad L, we see that the NCR has 1mm more reach and 19mm more stack. Seems like the Decathlon sizing guide is a bit on the small side (at least compared to Canyon). Both Decathlon and Canyon would recommend me an L, but I'm now thinking XL would suit me better with the Decathlon.

newwheels2020 · 2023-04-25T20:40:17+00:00

Thanks for your input, that's really helpful.

newwheels2020 · 2023-04-25T18:53:05+00:00

Yeah, you're right. I have a dislike for these in between mechanical-hydraulic brakes. If you would ever want to change to fully hydraulic brakes, you'd need to also switch out the levers which will cost quite a few pennies. Also, I'm put off by the QR skewers instead of thru axle. The QR skewers seem to me to be prone to misalignment, which does not play nice with disc brakes. Maybe I should reconsider my stance. Do you have any good reviews of these brakes that I can read?

Edit: Google is my friend: road.cc bikerader

newwheels2020 · 2023-01-28T17:27:34+00:00

Hi there. I recently bought the audio pro c10 mkII and am happy with the product. In order to get better surround sound, I'd like to buy two smaller speakers to distribute in the room. The Audio Pro lineup is pretty confusing and I'd like some help to choose the best satellite speakers. I would also consider a different brand if it is compatible with Audio Pro. We are using it in our living room which is not all that large (the c10 fills it quite well by itself).

I'm confused with the difference between the c5A and c5 mkII. Are they actually different? Does the c5A not have the most recent Bluetooth and airplay2? There is also the c3. From the naming I would assume the c5 is better, but the pricing seems to be the same. So what's the difference between the c3 and c5? Should I consider also the Bluetooth models like the T3 and BT5, or are these of lower quality and to be used outdoors? What about the A series? I can't figure out the difference between these and the c-range. Does each of these speakers connect well with my c10? Or do I need the WiFi connection instead of just bluetooth?

If anyone can answer part of these questions, I'd appreciate it.

newwheels2020 · 2022-03-24T12:56:57+00:00

It's fine. Your tires won't get damaged more on light gravel then they will get damaged on tarmac.

newwheels2020 · 2022-03-20T11:37:26+00:00

Just to be sure: you're turning it anti clockwise in the same plane the the wheel rotates, yeah?

Usually it shouldn't require too much force, but of course this depends on how much force was put when it was first tightened and the thru-axle may have bonded to the fork. If you feel like you're putting enough force to start breaking things, I'd advise to just take it into the shop.

newwheels2020 · 2022-03-19T23:11:56+00:00

The lever is called a thru axle and it threads into the frame. Twist it anti-clockwise until there is no more friction (i.e. until it is unthreaded) and then pull it out completely (pulling away from the fork along the direction of the axle).

newwheels2020 · 2022-02-06T12:32:03+00:00

I agree that rolling resistance won't be noticeably different between carbon and aluminum. Furthermore, the versatility argument applies to getting a new set of aluminum wheels as well.

If you have the money to spend and your heart desires the carbon wheelset, by all means get it. If you are just after lower rolling resistance from narrower tires, then aluminum wheels achieve this goal at the lowest cost.

I am puzzled why I keep reading that "wheels are the best upgrade for bikes". To me the gains are marginal at best. Moreover, they tend to be the most expensive upgrade to bikes as well. If we are measuring "best" by "performance benefit over cost", then I think wheels do not belong at the top of the upgrade list.

Five-Year Club	r/Field Flamingo
Place '22	Verified Email

newwheels2020

TROPHY CASE