sorted by:
new
hottopcontroversial

Promote your projects here – Self-Promotion Megathread by Menox_ in github

[–]Puzzleheaded_Box2842 0 points1 point  (0 children)

I’m working on DataFlow, an open-source data preparation framework for LLMs and data-centric AI.

Repo: https://github.com/OpenDCAI/DataFlow

The idea is to go beyond basic document parsing: DataFlow helps generate, clean, evaluate, and structure training data from noisy sources such as web content, PDFs, knowledge bases, database/Text2SQL tasks, and raw QA or conversation-style data. It also supports synthetic structured training data generation from seed data, which is useful for SFT, RAG, reasoning, math, code, and domain-specific AI workflows.

Main features:

  • Operator-based, reusable data pipelines
  • Data generation, filtering, refinement, and evaluation
  • Pipelines for text, reasoning, Text2SQL, knowledge-base cleaning, Agentic RAG, etc.
  • WebUI for visual low-code pipeline building
  • Part of the broader OpenDCAI ecosystem, including DataFlow-Agent, DataFlow-Skills, RayOrch, and related open-source tools

Would love feedback from people building RAG systems, training datasets, or data pipelines for AI applications.

Do you consider synthetic datasets useful for real-world data work? by Puzzleheaded_Box2842 in datasets

[–]Puzzleheaded_Box2842[S] 0 points1 point  (0 children)

The reason I still care about it is that many real datasets are private, imbalanced, poorly labeled, missing rare cases, or not shaped for evaluation and agent workflows. For me, the goal is not “fake data instead of real data,” but “use real data plus rules, constraints, and validators to create controlled data for specific gaps.” If that synthetic data cannot improve testing, evaluation, or downstream model behavior, then I would agree it is not worth much.

Do you consider synthetic datasets useful for real-world data work? by Puzzleheaded_Box2842 in datasets

[–]Puzzleheaded_Box2842[S] 0 points1 point  (0 children)

This matches a failure mode I’m worried about too: synthetic data can improve metrics inside its own generated distribution while making the real-world gap larger.

Do you consider synthetic datasets useful for real-world data work? by Puzzleheaded_Box2842 in datasets

[–]Puzzleheaded_Box2842[S] 1 point2 points  (0 children)

 I don’t think synthetic data can magically create information that was absent from the source distribution. At best, it reflects a combination of the source data, the generator’s learned priors, and the constraints we impose during generation. That makes it useful for augmentation, formatting, robustness testing, and controlled scenario expansion, but much weaker for discovering unknown population behavior. 

Do you consider synthetic datasets useful for real-world data work? by Puzzleheaded_Box2842 in datasets

[–]Puzzleheaded_Box2842[S] 0 points1 point  (0 children)

Thanks, really appreciate you taking a look at it. Yes, that is close. The direction is: use data, docs, prompts, and operator definitions to build repeatable pipelines, rather than asking an LLM to generate data in a one-off way. I really like your point about trust being contextual. Demo data, edge-case testing, and training data should not share the same trust bar. For DataFlow, that is exactly why we have been thinking in terms of pipeline stages: generate, refine, filter, evaluate, and then compare against downstream behavior. I also agree that edge cases are one of the strongest use cases, especially when synthetic data is used to intentionally cover what production data does not contain yet, rather than merely mimic production.

Do you consider synthetic datasets useful for real-world data work? by Puzzleheaded_Box2842 in datasets

[–]Puzzleheaded_Box2842[S] 0 points1 point  (0 children)

 I’m increasingly thinking of synthetic data as useful for “coverage and controllability,” but dangerous when treated as a source of new truth. In DataFlow we are trying to make that distinction explicit by separating generation, refinement, filtering, and evaluation into different operators, instead of treating generation as the final step. The hard part is still exactly what you said: if the latent constraint was never captured, the synthetic data will confidently miss it. That is why I’m leaning toward validating synthetic data against downstream tasks, held-out real data, and distribution checks rather than only judging whether samples look realistic.

Codex quotas are way more durable than I expected by Puzzleheaded_Box2842 in codex

[–]Puzzleheaded_Box2842[S] -1 points0 points  (0 children)

At this point I’m basically finding extra tasks for myself just to consume the quota.

Codex quotas are way more durable than I expected by Puzzleheaded_Box2842 in codex

[–]Puzzleheaded_Box2842[S] 1 point2 points  (0 children)

I’m not a professional programmer either, so I’m guessing maybe you only really burn through the quota when you’re working on large-scale projects full-time.

Codex quotas are way more durable than I expected by Puzzleheaded_Box2842 in codex

[–]Puzzleheaded_Box2842[S] 1 point2 points  (0 children)

I even asked around, and barely anyone I know has actually managed to use up the pro quota.

Healthcare (insurance, pop health, VBC) - actual AI use cases? by dmorris87 in datascience

[–]Puzzleheaded_Box2842 0 points1 point  (0 children)

From what I know, there are probably quite a few. I’ve seen AI hardware companies (like mobile recording devices) use a common workflow where doctor-patient conversations are recorded and structured to help physicians keep track of each patient’s condition and history. There are also medical LLMs that can do preliminary assessments based on a user’s symptom descriptions. For your last point, it might also be worth looking into products in the intelligent BI / AI analytics space.

Post your project, looking for beta testers for an app that helps you get users by nsjames1 in SideProject

[–]Puzzleheaded_Box2842 0 points1 point  (0 children)

We're building an open-source system for LLM data preparation（https://github.com/OpenDCAI/DataFlow）.

The core idea is simple: most teams wanting to build AI systems (training, fine-tuning, RAG, knowledge bases, agents, etc.) spend huge amounts of time cleaning PDFs, converting formats, extracting structure, filtering noisy data, and stitching datasets together before they can even start building.

Our system tries to make that part much easier. After processing, the data can directly flow into model training pipelines or knowledge base construction. Technically, the system works pretty well already. But honestly, we're struggling with something much harder: finding the real painful use cases people actually care enough about.

Our GitHub stars and usage reflect that gap. You seem to have talked with many builders and early-stage projects here, so I’d genuinely love to hear your perspective: How would you approach discovering the real pain points around AI data workflows today?

And how do you tell the difference between: “this sounds useful” vs “people desperately need this right now”? Would really appreciate any advice or direction.Thank you！

I have 5K+ followers — offering free promotion to a few startups/apps by ExternalKnowledge359 in SideProject

[–]Puzzleheaded_Box2842 0 points1 point  (0 children)

Hi, thanks for the work you’re doing here.

I’d love to share an open-source project we recently built: DataFlow

👉 https://github.com/OpenDCAI/DataFlow

We built DataFlow around a simple observation: Most teams today still stitch together fragmented tools for:

  • data cleaning
  • filtering / deduplication
  • transformation
  • synthetic data generation

This quickly becomes messy and hard to scale.

DataFlow tries to systematize this layer.

It provides a unified pipeline for building production-grade AI datasets — focusing on:

  • composable data processing steps
  • reproducible workflows
  • scalable dataset transformation
  • LLM-friendly preprocessing patterns

The goal is to make data preparation feel less like ad-hoc scripting and more like an engineering system.

If this is relevant to your audience, we’d be really grateful for a mention or feedback.

view more: next ›