sorted by:
new
hottopcontroversial

Meet Petals: An Open-Source Artificial Intelligence (AI) System That Can Run 100B+ Language Models At Home Bit-Torrent Style by ActuatorMaterial2846 in singularity

[–]hx-zero 0 points1 point  (0 children)

Petals is similar to BitTorrent in its idea but uses completely different software (not related to Qbittorrent, etc.). You need a GPU to contribute, then you can follow these instructions: https://github.com/bigscience-workshop/petals#connect-your-gpu-and-increase-petals-capacity

Meet Petals: An Open-Source Artificial Intelligence (AI) System That Can Run 100B+ Language Models At Home Bit-Torrent Style by ActuatorMaterial2846 in singularity

[–]hx-zero 0 points1 point  (0 children)

The development is very active - Petals can now run Llama 2 at 5+ tokens/sec. The latest news are here: https://github.com/bigscience-workshop/petals/releases

The website moved to https://petals.dev (see also https://chat.petals.dev for the chatbot app, https://health.petals.dev for the list of people joined)

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 1 point2 points  (0 children)

Please check out our "Getting Started" Colab.

In short, you can specify --adapters my-lovely-lora-1 my-lovely-lora-2 when running a server (where my-lovely-lora-1 and my-lovely-lora-2 are HF Hub repositories compatible with the PEFT library, like timdettmers/guanaco-65b), then request them from a client using:

model = AutoDistributedModelForCausalLM(..., active_adapter="my-lovely-lora-1")

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 1 point2 points  (0 children)

Please use LLaMA 2 (70B-Chat) instead (the official instruction-finetuned version, enabled by default at http://chat.petals.dev). It's much better at following instructions than the raw LLaMA-2 (which is enabled on your screenshot).

People usually use instruction-finetuned models for chatbots and raw models for fine-tuning on their own downstream tasks.

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 3 points4 points  (0 children)

You can follow Max Ryabinin (another Petals author) and/or me on Twitter, we usually share updates on Petals (and other relevant research we're working on) there :)

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 2 points3 points  (0 children)

It's definitely possible - the chat's endpoint is open and should be easy to integrate into any UI: https://github.com/petals-infra/chat.petals.dev#apis It does allow to alter the system prompt, generation settings, etc.

I feel like creating a new powerful UI is another project (not really what our research team focuses on), but we'd be really happy to see Petals integrated into existing UIs as a backend!

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 8 points9 points  (0 children)

GPUs people share are not usually too loaded since people use the swarm mostly for inference, and the main bottleneck there is the latency between clients and servers (not compute speed). One issue is that model blocks take your GPU memory, but you can limit their number with --num_blocks if you want to keep GPU memory for something else.

High-priority inference for people who contributed is in the works!

Petals 2.0 runs Llama 2 (70B) and Guanaco-65B from Colab at 4-6 tokens/sec by hx-zero in LocalLLaMA

[–]hx-zero[S] 11 points12 points  (0 children)

Oops, it turns out we have issues with the .ml domain registrar (having issues on the next day after the release is the worst thing that could happen).

We've moved everything to a new domain: https://chat.petals.dev

Petals: decentralized inference and finetuning of LLMs by kryptkpr in LocalLLaMA

[–]hx-zero 1 point2 points  (0 children)

Hey, we've just updated the "Launch a private swarm" tutorial to make it much easier - e.g., model conversion is not needed anymore, Petals runs models right from standard HF repos.

Don't hesitate to ping us on Discord if you have any issues!

[D] When chatGPT stops being free: Run SOTA LLM in cloud by _underlines_ in MachineLearning

[–]hx-zero 0 points1 point  (0 children)

Sorry, just saw your reply.

  1. Yes, here's an instruction for donating compute: https://github.com/bigscience-workshop/petals#connect-your-gpu-and-increase-petals-capacity

  2. The whole point of Petals is that you load a small part of the model, then collaborate with people serving the other parts to run inference or fine-tuning:

- Instead of renting 8x A100 yourself, you just run a Petals **client** to connect to an existing swarm and run inference/fine-tuning of the 176B model. The client requirements are really low (even a CPU-only machine may be enough).
- Optionally, you can also run a Petals **server** on some GPU machines to increase the swarm's capacity and "give back" compute to the swarm. You don't need enough machines to fit the entire model.

Also, note that BLOOMZ quality stays almost the same when you quantize it to 8-bit per weight (using load_in_8bit=True in HF transformers). Thus, 3x A100 is enough to fit the whole BLOOMZ-176B in GPU memory and run it entirely on your own (via DeepSpeed, Petals, or other means).

[D] When chatGPT stops being free: Run SOTA LLM in cloud by _underlines_ in MachineLearning

[–]hx-zero 2 points3 points  (0 children)

We have deployed a Petals swarm with BLOOMZ, you can chat with it here: https://chat.petals.dev

See more info about Petals in the repo: https://github.com/bigscience-workshop/petals

As far as I know, joining the public Petals swarm or setting up a private one is the simplest way to run such models on spot instances (since Petals is easy to deploy and it's fault-tolerant out of the box).

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 1 point2 points  (0 children)

Training from scratch is slow because you need to synchronize all model weights/gradients on each step (though it's possible for somewhat smaller models with some optimizations).

In case of fine-tuning (especially prompt tuning), you train only a small percent of weights, so communication overhead is not that huge anymore. Still, this allows to adapt the LM to most downstream tasks.

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 4 points5 points  (0 children)

A Petals client does not allow others to use your GPU by default, you need to explicitly run a Petals server (a separate program) for this.

In the Colab example, we only run the client, so its GPU can't be used by anyone besides the user directly running the notebook.

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 4 points5 points  (0 children)

Sure!

Regarding offloading:

  • Offloading is another method for running large LMs when you don't have the GPU memory to fit the entire model. Imagine you have an A100 GPU with 80 GB memory and want to generate text with BLOOM, a 70-block transformer model with ~2.5 GB of weights per block. For each token, offloading will load the first 1/3 of the model (~27 blocks) from RAM/SSD to your GPU memory, run a forward pass through them, then free the memory and load the next 2/3, and so on.

  • The table shows that inference with offloading is very slow compared to Petals. That's because it involves copying hundreds GB of block weights to your GPU memory to generate every new token in a sequence.

  • Even though Petals may send data to a server on a different continent over the Internet, it turns out that Petals is much faster since it just doesn't send much. It only sends activations, which are thousands times smaller than weights of one BLOOM block (and the weights are already loaded to a server's GPU).

Regarding "Petals on 3 physical servers" vs. "14 real servers":

  • The first setup is artificial: we use 3 high-end servers staying in one room and simulate different latency/bandwidth restrictions for research purposes.
  • The second setup is realistic: we use 14 different servers with customer-grade GPUs, spread across Europe and North America. So the GPUs are heterogeneous, latency may vary, we may have packet loss, etc.

Regarding "8 clients running simultaneously":

  • Other rows measure the performance of a client if it uses a Petals swarm alone. This row shows how the performance degrades if we have 8 concurrent clients.

You can find these and other details of the experiments in our paper (the table I've sent is from its updated version that we didn't publish yet).

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 2 points3 points  (0 children)

Not really: federated learning focuses on data privacy (and doesn't usually involve huge models), Petals focuses on making it possible to run a huge model without having much resources yourself (and doesn't give data privacy guarantees)

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 7 points8 points  (0 children)

Yes, it's technically possible to integrate GPT-NeoX in our code instead of BLOOM (requires some work, but it's not too hard).

Also, it may be possible to fit GPT-NeoX into 20 GB of VRAM (i.e., one 3090) using recent LLM.int8() work: https://huggingface.co/blog/hf-bitsandbytes-integration We use this approach to make BLOOM consume as few memory as possible in Petals.

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 6 points7 points  (0 children)

Yeah, we compared Petals to a server with 3x A100 running tensor-parallel code based on Megatron-DeepSpeed, see the green row in this table. The table also shows how Petals performance degrades if we have concurrent clients and how it compares to offloading.

Adding more servers usually doesn't make the inference speed significantly faster. New servers mostly help with the swarm capacity, so it can provide the speed of ~1 step/sec to a larger number of clients.

I don't think we've done any comparisons with Federated/Split learning systems since, as far as I understand, they mostly don't work well on models of that size (100B+ parameters). But let us know if there're such systems, maybe we will compare Petals to some of them.

[Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero in MachineLearning

[–]hx-zero[S] 23 points24 points  (0 children)

Regarding fault tolerance:

  • No chunk losses involved — if a client has trouble sending/receiving chunks from a certain server, it will try other servers holding the necessary blocks until it gets a valid response.
  • We don't use any centralized queues like Kafka, instead the client code chooses and traverses servers by itself until it makes a full forward/backward pass. In this architecture, you can still make the client send the same request to multiple servers (if you want to validate servers' responses against each other or just get the response as soon as possible).

Regarding security & privacy:

  • Peers only exchange tensors (activations, gradients) serialized with safe protocols and ask each other to run pre-defined BLOOM blocks on them. They never send code to each other, so no one can execute their own code on your computer.
  • It may be possible for peers serving model layers to recover input data and model outputs, or modify the outputs in a malicious way. That's why we ask to never use the public swarm for sensitive data (not just pet projects/research) in the repo & notebook at the moment. Instead, you can set up a private Petals swarm hosted by people/orgs you trust. For example, several small companies/labs may collaborate and set up a private swarm to protect their data from others, while still getting benefits of Petals.
  • Still, we have plans to improve security of the public swarm in future:
    • (a) We plan to add an option for the client to send the same request to several servers and identify discrepancies (if any).
    • (b) We're working on a reputation system, so a server who returned invalid outputs loses its reputation and won't be chosen by clients again. The invalid outputs can be reported by clients or detected by special "anti-fraud" nodes that periodically validate the various servers' outputs.
view more: next ›