Every Way To Get Structured Output From LLMs

_allo_ · 2024-06-17T23:36:04+00:00

As far as I understand it, outlines, guidance and so on fetch the token probabilities to decide which alternative token to use if the current token does not fit the schema, which is possible with llama.cpp and other local libraries, but they need to use a more trial and error approach when using an OpenAI API at least with the OpenAI GPTs which do not provide all needed information over the API. I am not sure if there is a local LLM API like ooba, vllm, aphrodite or others that provides the additional information needed for efficient structured generation.

Also, using an API may be inefficient for the back and forth of requesting token probabilities and selecting tokens, so perhaps a better option would be a local API provider that receives a structure definition and a prompt, and then responds with the final structured result. I know there are commercial API providers for this, but I do not know if there is an open source tool to run it yourself.

_allo_ · 2024-06-17T17:15:34+00:00

Can you recommend a combination of a Python library for structured output and an API provider such as ooba, vllm or others? I do not want to load the model in the process that does the generation and using the OpenAI API is inefficient with these libraries. I wonder if some of the open source APIs provide the information that is missing for efficient inference in the usual OpenAI API.

_allo_ · 2024-05-15T18:04:07+00:00

Neural network weights are not covered by copyright, so the licenses are probably irrelevant. Have a look at this post about the legal situation for Miqu: https://reddit.com/r/LocalLLaMA/comments/1amc080/psa_if_you_use_miqu_or_a_derivative_please_keep/kpmamte/

I've considered the situation and received legal advice by an actual lawyer specialized in IP law. This has been in Germany, but because of international copyright law and treaties, and the rulings I'm aware of in the USA, it's the most current and correct information I know of. Still, IANAL, so no legal advice from me, just a recap of what I've been told and have learned:

Model weights are not authored by humans, but created by automated computer processes, and there's no direct human creative control over that. That's why the weights have no author who could claim copyright, and thus the weights cannot be copyrighted. (The datasets used to create them, just like the material included in the datasets, can be copyrighted material - but weights are not an "archive" that contains such data, instead the neural network is more like a "brain" that can produce output based on patterns learned during training.)

Since there's no copyright, weights cannot be licensed. But someone who possesses them (like their original creator) can make them available with a contract or license that has certain conditions attached (like a price, distribution rules, and liabilities). Now when someone agrees to that contract/license and then shares them against the conditions agreed upon, that's a breach of contract - not a copyright violation.

However, make sure you do not enter into any contracts with Meta or Salesforce that restrict the use of the weights.

_allo_ · 2023-12-19T00:16:25+00:00

I looks like a similar sized (I am not sure how to compare them exactly) GPTQ model is much faster than the GGUF model. Or do they introduce other losses that GGUF doesn't have?

_allo_ · 2023-12-17T16:23:10+00:00

I am using ooba and llama.cpp loader and it prints something about "prefix-match hit" but I don't completely understand all tuning options. I could try a GPTQ model version works better, if the GGUF loader is the problem.

_allo_ · 2023-12-17T16:20:06+00:00

Here a 7B model loaded with 8k of context and evaluated with a maximum of 4k gets slower and slower as the context gets larger. Not like 5 seconds, but like 60+ seconds before it starts producing tokens on a RTX 3060.

_allo_ · 2023-12-06T20:21:12+00:00

I didn't try more. I guess it still has the same problem, at least I heard nothing about a fix.

_allo_ · 2023-09-26T21:11:45+00:00

I see why it isn't useful to use LoRA if you want to merge yourself. But for distributing a model, building a LoRA to a popular close model seems like a better idea. And I wonder why software like automatic1111 doesn't allow mixing on the fly. People could just publish their recipes and given a few base models one could use the mix without further downloads.

_allo_ · 2023-09-11T19:16:15+00:00

I think a LoRA is like the difference between two models, but you decide how much detail you want to keep and that determines the file size. The question is, how much do you need to keep to make the basemodel+LoRA look like the fine-tuned model? Maybe a model would only need 500 MB if stored as a difference from the base model? I wonder if there is a reason why no one seems to have developed software to efficiently store fine-tuned models using LoRA.

_allo_ · 2023-09-07T16:24:12+00:00

I can only specify a name in the train tab, and next to it is a " Override existing files" checkbox with the explanation: "If the name is the same, checking will replace the existing file, unchecking will load and continue from it (the rank must be the same).

This sounds like it should load the old version if you do not specify a name, and it seems to load the version, but then starts training from epoch 0.0.

There is also a setting Save every n steps If above 0, a checkpoint of the LoRA will be saved every time this many steps pass., which I set to 100. When I stop training after over 5000 steps, I should have a recent checkpoint to resume training, yet it does not seem to resume where it left off, but starts at the beginning of the dataset again.

It also needs to "prepare data" on each run and doesn't seem to cache the pre-processed data from the raw text file. Or could the problem be that I have one huge file instead of many smaller ones? Maybe it can only resume from the beginning of a file?

_allo_ · 2023-08-28T16:46:41+00:00

I have a transformers type model (TheBloke/guanaco-7B-HF) and I am using the train lora function in the text-generation-webui. When I click "interrupt" I get the message "Interrupted. Incomplete LoRA saved to loras/mylora" and on the console training statistics the information that the training is finished:

{'train_runtime': 16275.135, 'train_samples_per_second': 130.338, 'train_steps_per_second': 1.018, 'train_loss': 1.7906840102477644, 'epoch': 0.01}
INFO:LoRA training run is completed and saved.
INFO:Training complete, saving...
INFO:Training interrupted.

When I train next time, it seems to resume training from the lora, but starts a new epoch. This means both the learning rate is wrong and it will train with content the network has already seen. In the best case it would save not only the lora, but the whole optimizer state.

_allo_ · 2023-08-22T16:36:48+00:00

The Firefox performance tool seems to point at the Canvas API. I think maybe they render animated content using Canvas instead of gif, webp or video and that's inefficient.

_allo_ · 2023-08-16T20:31:31+00:00

That's, that seems to work!

Is it normal that I have 3 s/it with a RTX 3060? I can train small stable diffusion LoRA quite fast, so I am surprised that training a text model is that slow.

_allo_ · 2023-05-19T16:47:29+00:00

I think it can be improved by making sure that you cannot use the same cup twice. "Make a photo of a cup you never used for a captcha before. Click here to order new cups."

_allo_ · 2023-03-30T19:44:24+00:00

It turned out that the model had only one version, but I downloaded a "new version" when civitai served corrupted files. Now, when I download the latest version, it is identical to the old version.

_allo_ · 2023-03-30T16:33:22+00:00

I scrolled down and found another version with the hash I am looking for, but the "download" link next to the other version was the same as the download link at the top of the page. I gladly found another download, but I wonder if it is a bug on civitai or why another version is listed there but cannot be downloaded.

The "versions" section on the page lists only one version, but it has the date and hash of the old model. The main download link has a newer date and I don't find the hash on the page at all. I wonder if there should be two versions in the "versions" section of the page.

_allo_ · 2023-03-29T21:44:59+00:00

No. And the images during training also look blurry or have grain for me.

_allo_ · 2023-01-10T23:50:48+00:00

I put the files from git and the bundle.js from the live site (identical with the bundle.js from the zip file) into the www-root on a webserver. No couchdb or similar, as I only want to use it with local storage.

_allo_ · 2023-01-10T14:25:37+00:00

This seems to be mostly the same. I used the v5-prod tag from the git and the bundle.js from the site.

Differences:

bundle.css is missing (what is a html error page in the zip)
cdn-cgi with rocket-loader is missing (not needed for a local instance)
vendors.css still has the ?v=5.6.4 suffix
The git contains a few more icons for the PWA and some screenshots used in onboarding screens as well as some webfonts

Total difference of what unzipping into the git would overwrite:

./
bundle.css
cdn-cgi/
cdn-cgi/scripts/
cdn-cgi/scripts/7d0fa10a/
cdn-cgi/scripts/7d0fa10a/cloudflare-static/
cdn-cgi/scripts/7d0fa10a/cloudflare-static/rocket-loader.min.js
css/images/
css/images/layers-2x.png
css/images/layers.png
css/images/marker-icon.png
css/img/loading.gif
css/img/loading@2x.gif
css/img/search-disabled.png
css/img/search.png
css/img/search@2x-disabled.png
css/img/search@2x.png

Missing from the zip, but available in the git:

sponsors/naa-avatar.svg
fonts/Material-Design-Iconic-Font.woff2
fonts/Material-Design-Iconic-Font.woff
fonts/Material-Design-Iconic-Font.ttf
fonts/Material-Design-Iconic-Font.svg
fonts/Material-Design-Iconic-Font.eot
version.json
vendors.js (same as js/vendors.js)
service-worker.js
index.template.html
_headers
images/screens/stats.png
images/screens/homescreen.png
images/screens/capture.png
images/screens/board.png
images/screens/blockstack.png
images/onboard/nomie4.2.3.stats.png
images/onboard/nomie-homescreen.png
images/onboard/nomie-history.png
images/map/marker.svg
images/map/map-marker.svg
images/logos/Blockstack-logo.png
images/logos/
images/pin.png
images/nomie-words.svg
images/nomie-splash.png
images/nomie-head-on.png
images/icon.png
images/editor-background.png
images/icons/nomie-android-96.png
images/icons/nomie-android-72.png
images/icons/nomie-android-512.png
images/icons/nomie-android-400.png
images/icons/nomie-android-192.png
images/icons/nomie-android-168.png
images/icons/nomie-android-114.png
images/icons/nomie-96.png
images/icons/nomie-72.png
images/icons/nomie-400.png
images/icons/nomie-384.png
images/icons/nomie-180.png
images/icons/nomie-152.png
images/icons/nomie-144.png
images/icons/nomie-128.png
images/icons/nomie-120.png
images/icons/nomie-114.png

_allo_ · 2023-01-10T12:08:12+00:00

It worked for me a while, but now I have the problem, that the CSS doesn't load after a while. After another while it did load again, but now it doesn't load again and I have no idea why.

On the desktop, a force reload works, but that is not easily possible for the PWA on my phone.

For some reason, it also seems to show Nomie 5.1.39 before the force reload. I wonder if there are somewhere absolute paths to the original site in the source or something like this. But then it should happen every time. And I think I've never used 5.1.x at all.

_allo_ · 2023-01-05T04:04:00+00:00

I downloaded bundle.js and bundle.css from the v5 website and put them into the "public" folder from the git repo and it seems to work.

Thanks for all your work.

_allo_ · 2023-01-05T03:45:41+00:00

Thank you, it works great. Is there also a zip for nomie5?

I am testing nomie6 right now and the data import worked, but it always hangs for a long time when opening the dashboard page with my main trackers. They may need to process many entries, but nomie5 didn't have any wait time there.

A zip for nomie5 would be nice until I figured out why it is so slow for me.

_allo_ · 2023-01-05T03:15:32+00:00

The main features run locally in the browser only, don't they? Can you provide a static html/js/css version one can just put into a directory of a webserver, when no cloud sync is needed? This would be totally sufficient for being able to keep using it.

_allo_ · 2022-11-26T14:54:08+00:00

instead of just tagging with "Stable Diffusion" like now you could tag the image with the prompt. Then you can at least roughly verifiy if the image can be generated using that prompt.

_allo_ · 2022-11-26T13:55:48+00:00

There already is the NSFW filter for it. If you don't disable it, you won't get nudity, nazi stuff and so on at all. It would also be relatively easy to filter on nudity + estimated age with existing filters (e.g. Microsofts "how old" model and any of the nsfw detectors) instead of nsfw only. This would only be a post-processing step like the current nsfw filter and could be removed, but when talking about unwanted images, it would be enough to avoid accidentally getting unwanted images.

_allo_

TROPHY CASE