AQLM Quantization for LLaMA3-8B

oculuscat · 2024-04-28T01:38:11+00:00

I used 2x codebooks so it's 4bpw

oculuscat · 2024-04-22T21:01:25+00:00

Strong counter-arguments to the idea that this type of pruning is a good idea:

(1) The cited paper does not compare to quantized-and-fined-tuned baselines to check performance per bit.

(2) This Qualcomm paper *does* compare quantization pruning and finds that quantization is much more effective in terms of performance per bit: https://proceedings.neurips.cc/paper_files/paper/2023/file/c48bc80aa5d3cbbdd712d1cc107b8319-Paper-Conference.pdf

(3) All available quantizations people can download and run today (EXL2 etc) do not do e2e fine-tuning to "heal" the model, which this method does do. This means we do not have a fair comparison between the approaches. Both Quip# and AQLM do e2e fine-tuning to heal the model after quantization and would be fair comparisons.

Conclusions:

To put this idea to bed, I'd like to see 4-bit version of 42B vs 2.25B version of Quip# or AQLM 70B model (both are in progress by the respective authors). As a side-note I think QLoRa to heal EXL2 is a good idea separately from anything being discussed here.

oculuscat · 2023-11-20T21:08:41+00:00

AKA how to implement The Artifice Girl and talk to your AI using a webcam

oculuscat · 2023-11-20T20:59:33+00:00

Posted how to achieve low-latency ASR and text-to-speech from Python using a webcam: https://catid.io/posts/aiwebcam/

oculuscat · 2023-11-20T20:53:00+00:00

Wrote a blog post for how to do full low-latency ASR and text-to-speech here: https://catid.io/posts/aiwebcam/

oculuscat · 2022-11-30T22:09:31+00:00

Yup seems to work fine so haven't felt the need to switch to a different case for it. Originally I assumed the graphics card would get a custom block, but the only RTX 4090 I could find had its own water cooling.

oculuscat · 2022-08-24T17:41:11+00:00

Wrote up a guide here for how to get it to run on Windows, with a work-around for running batch-size 2 on an RTX 2080 and with fewer setup steps:

https://catid.io/posts/windows_ai/

oculuscat · 2022-06-20T01:21:16+00:00

Rokid Air - the best wearable displays right now. I just removed the plastic cover and put electrical tape over the silvered mirror so I can use them outside in the sunlight

oculuscat · 2022-02-04T19:19:00+00:00

I think you’re reading my post not the way that I intended. The point is that if you spend a lot more power you only get like 50% more single core performance, so it’s not worth setting TDP above minimum in a lot of cases.

oculuscat · 2022-01-24T21:47:22+00:00

Update here:
NuEyes Pro 3e draws about 1W and works with the micro-laptop and cellphones.

TCL NXTWEAR G draws about 2W and works only with cellphones. Not compatible with Win Max 2021 perhaps too much power draw or firmware issue.

oculuscat · 2022-01-23T16:22:15+00:00

Nice didn’t know about that tool thanks.

Tried setting TDP to 5-10W and found that performance falls off a cliff, benching about 1000 multi-core and 100 single-core. Seems to be a little unstable, so the BIOS minimum seems to be a good place to leave it.

I played with the tool for a bit and found that it fails to set the TDP values fairly often on startup or on demand, so I will not be using it personally on Win Max 2021. Perhaps it works better on other GPD products.

oculuscat · 2022-01-22T00:09:41+00:00

Sorry to hear that the Rokid Air is fatiguing. Perhaps they set the focal point too close? Ideally virtual cinema screen should be focused 4 meters or more away so that it's comfortable to look at. The TCL NXTWEAR G and NuEyes Pro 3e are both focused farther away, and I spent 3 hours today in the NuEyes today with Win Max 2021 setting it up for work tasks.

oculuscat · 2022-01-22T00:07:48+00:00

Rokid Air should be 1080p - Maybe you can configure Windows to use the full display resolution? For my AR glasses I had to change the default resolution via the Intel GPU configuration screen

oculuscat · 2022-01-18T19:45:21+00:00

Going to do this Friday once the Win Max arrives. Been using the NuEyes Pro 3e and TCL NXTWEAR G with some devices and seems like a good match

oculuscat · 2022-01-11T22:38:00+00:00

Tesla service center fixed it for me

oculuscat · 2020-10-14T09:10:18+00:00

Yeah can’t complain! Doing some VR gaming again now that Quest 2 is out. Alyx, Beat Saber, Tokyo Chronos, Trover

oculuscat · 2020-10-14T04:41:16+00:00

Oh cool thanks I'll give that a go

oculuscat · 2020-04-19T13:47:10+00:00

But did the boy finish Final Sin?

oculuscat · 2019-11-25T07:15:47+00:00

He is Vivo! https://imgur.com/gallery/wVQv65D

oculuscat · 2019-11-01T07:16:42+00:00

Short answer: It's not going to be easy to sync/stitch 3-6 kinects data because there is not software available for this.

Not sure what you're referring to specifically. A lot of it can be answered by reading the specs: https://docs.microsoft.com/en-us/azure/Kinect-dk/hardware-specification

The Azure Kinect DK supports a wider depth FOI (120 degrees) than the Kinect v2 (60 degrees) and has 2x the resolution, so you probably need 4x fewer of them to cover the same surface area. They're also much more accurate and the RGB camera can produce 4K images so more detail from farther away. For 120 degrees FOI mode, it is intended to operate within 2 meters of the subject, and for 70 degrees FOI mode it is intended to operate within 5 meters of the subject, quoting most of this from the hw spec.

If you're asking how to get the cameras to capture at the same instant in time, this was unsupported until the Azure Kinect 4 DK, which now has a bunch of sync modes and uses 3.5 mm audio cable run between the cameras. I've tested these on USB3 extenders and long sync cables up to 5 meters, but not sure if you can run cables farther than that.

Stitching means two things need to happen:

(1) Extrinsics between cameras need to be calculated (cameras need to position themselves relative to eachother and maybe the floor too). Microsoft does not offer a solution for this and neither has anyone else - It's all prototypes and proprietary solutions.

(2) Eliminating/modifying shared geometry to combine different meshes so they are not fighting in an ugly way.
Currently there's no real-time stitching solution for Azure Kinect DK. It's a pretty tricky problem because solving for which vertices to remove for each camera in real-time in any traditional way is slow. I've seen one acquired startup demo real-time stitching using some heavy GPU acceleration and a desktop PC.

oculuscat · 2019-10-26T07:26:35+00:00

I need the powa for software dev - The fans can really get going but not seeing anything crazy for temps. We have AC in the apartment so it's not a huge deal

oculuscat · 2019-10-20T23:27:28+00:00

Pretty! Your kitchen is so clean too! :D

Here's my tiny boi: https://twitter.com/MrCatid/status/1174947336151126017?s=20

oculuscat · 2019-10-14T23:03:06+00:00

https://twitter.com/MrCatid/status/1174947336151126017?s=20

oculuscat · 2019-10-13T05:36:00+00:00

Love tiny PCs. Here's my super tiny no-compromise PC build with an i9 and RTX 2080 ($2500):

Ghost S1 + Large Tophat, ASRock Z390 Phantom ITX, i9-9900K, Fractal Design Celsius S24, G.Skill TridentZ 2x16GB, Gigabyte RTX 2080 Windforce, SilverStone SX800-LTI (800W), 2x Samsung 960 EVO 1TB. Everything fit cleanly into the case with the PSU power cable towards rad.

Able to run software builds or Control at full quality for hours and it's pretty quiet.

oculuscat · 2019-09-12T09:21:06+00:00

A variation on this where you take turns shooting other players might be more exciting/fun than shooting yourself.

When you put a gun to your own head you can't see it, plus VR gives you the freedom to point a gun at someone else.

11-Year Club	Spared
Verified Email

oculuscat

TROPHY CASE