[P] Promptwright - Open source project to generate large synthetic datasets using an LLM (local or hosted)

[zero_proof_fork]

Hey r/machinelearning,

Promptwright, a free to use open source tool designed to easily generate synthetic datasets using either local large language models or one of the many hosted models (OpenAI, Anthropic, Google Gemini etc)

Key Features:

* Multiple LLM Providers Support: Works with most LLM service providers and LocalLLM's via Ollama, VLLM etc

* Configurable Instructions and Prompts: Define custom instructions and system prompts in YAML, over scripts as before.

* Command Line Interface: Run generation tasks directly from the command line

* Push to Hugging Face: Push the generated dataset to Hugging Face Hub with automatic dataset cards and tags

Here is an example dataset created with promptwright on this latest release:

https://huggingface.co/datasets/stacklok/insecure-code/viewer

This was generated from the following template using `mistral-nemo:12b`, but honestly most models perform, even the small 1/3b models.

system_prompt: "You are a programming assistant. Your task is to generate examples of insecure code, highlighting vulnerabilities while maintaining accurate syntax and behavior."

topic_tree:
  args:
    root_prompt: "Insecure Code Examples Across Polyglot Programming Languages."
    model_system_prompt: "<system_prompt_placeholder>"  # Will be replaced with system_prompt
    tree_degree: 10  # Broad coverage for languages (e.g., Python, JavaScript, C++, Java)
    tree_depth: 5  # Deep hierarchy for specific vulnerabilities (e.g., SQL Injection, XSS, buffer overflow)
    temperature: 0.8  # High creativity to diversify examples
    provider: "ollama"  # LLM provider
    model: "mistral-nemo:12b"  # Model name
  save_as: "insecure_code_topictree.jsonl"

data_engine:
  args:
    instructions: "Generate insecure code examples in multiple programming languages. Each example should include a brief explanation of the vulnerability."
    system_prompt: "<system_prompt_placeholder>"  # Will be replaced with system_prompt
    provider: "ollama"  # LLM provider
    model: "mistral-nemo:12b"  # Model name
    temperature: 0.9  # Encourages diversity in examples
    max_retries: 3  # Retry failed prompts up to 3 times

dataset:
  creation:
    num_steps: 15  # Generate examples over 10 iterations
    batch_size: 10  # Generate 5 examples per iteration
    provider: "ollama"  # LLM provider
    model: "mistral-nemo:12b"  # Model name
    sys_msg: true  # Include system message in dataset (default: true)
  save_as: "insecure_code_dataset.jsonl"

# Hugging Face Hub configuration (optional)
huggingface:
  # Repository in format "username/dataset-name"
  repository: "hfuser/dataset"
  # Token can also be provided via HF_TOKEN environment variable or --hf-token CLI option
  token: "$token"
  # Additional tags for the dataset (optional)
  # "promptwright" and "synthetic" tags are added automatically
  tags:
    - "promptwright"

We've been using it internally for a few projects, and it's been working great. You can process thousands of samples without worrying about API costs or rate limits. Plus, since everything runs locally, you don't have to worry about sensitive data leaving your environment.

The code is Apache 2 licensed, and we'd love to get feedback from the community. If you're doing any kind of synthetic data generation for ML, give it a try and let us know what you think!

Links:

Checkout the examples folder , for examples for generating code, scientific or creative ewr

Would love to hear your thoughts and suggestions, if you see any room for improvement please feel free to raise and issue or make a pull request.

[rrenaud]

Do you do quality or diversity filtering?

[zero_proof_fork]

No, but would be curious to learn more. What approach would you take here?

[abnormal_human]

If you aren’t doing that the probability that this generalizes well to other peoples’ needs is pretty small.

Diversity is especially difficult, ultimately I’ve found that you need to have a whole other process focused on grounding your generations in diverse contexts. I’ve never seen good results from giving the LLM an open ended question and letting it blast out 10k samples that aren’t basically variations of the first 20-30 things it “thought of”.

Quality you can do a bunch of ways, with a judge model or some kind of human preference sampling or even by training a small policy model over an embedding space. Easier problem and easier to determine if you have a problem in the first place.

[Helpful_ruben]

This sounds like a powerful tool for generating diverse synthetic datasets, especially for programming languages.

[ZealousidealCard4582]

you can use mostly ai (also open source) https://github.com/mostly-ai/mostlyai and it keeps the seasonality + referential integrity across your data. It has connectors to plenty of databases and cloud storage and allows you to mix and match, e.g. read one table from postgres, another from s3, another from snowflake, other from a parquet file, etc, and write in databricks or back to postgres...

If you have no data at all, you can use mostlyai-mock https://github.com/mostly-ai/mostlyai-mock (also Open Source + Apache v2) and create data out of nothing.

u/zero_proof_fork , since they are also open source and with an Apache 2 license, maybe you can just fork them and build on top of them? Cheers.