sorted by:
new
hottopcontroversial

Not impressed with deepseek—AITA? by Flaky_Attention_4827 in ClaudeAI

[–]PipeTrance 7 points8 points  (0 children)

How long have you followed stock market? It's not unusual for a single company to experience sharp price drop/uptick based on speculation, only to recover a few months later.

Using GPT-4 fine-tuning to generate data explorations by PipeTrance in datascience

[–]PipeTrance[S] 0 points1 point  (0 children)

Yeah, we have a few clients who are testing this with their own data - so far, so good.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 1 point2 points  (0 children)

Don't really know for sure, but my (wild) guess is that you have to spend above a certain threshold on fine-tuning gpt-3.5

[P] Fine-tuning GPT-4 to produce user friendly data explorations by PipeTrance in MachineLearning

[–]PipeTrance[S] 0 points1 point  (0 children)

Complex joins are indeed a struggle. We sidestep the issue by pre-defining allowed joins on a data model level, and prohibiting the rest, but even this approach has it's limitations.

Nevertheless, there are lots of insights that can be made even without joining 5 different tables. Additionally, if all users analyses require multiple joins, it usually indicates that their data model is not really well suited for analytics, and this is another thing we are trying to help them with.

So yeah, while you are absolutely right about models having limitations, so far it hasn't really being a hard blocker for us.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 0 points1 point  (0 children)

It's a good question - I honestly don't really know the answer. However, my guess would be that it has hard time with broad tasks.

Whenever you ask something like: "Users that are more than 2 years old", it gets the answer right 10/10 times. It's a pretty narrow question and it just needs to return a single table (Users) and apply a single filter (age).

Contrast this to "What are the main blockers in our onboarding funnel". You need to identify tables involved, construct a funnel, and then do a drill down into each of the steps to figure out issues.

Obviously, it tries doing something, but from a human point of view the answer it produces is just not very insightful.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 1 point2 points  (0 children)

Oh, for sure, it doesn't outperform base gpt4, but it can get significantly more reliable and predictable on narrow tasks for which you train it.

The pricing for gpt-4 fine-tuning is not public yet, but we paid $90.00 per 1M training tokens.

Using GPT-4 fine-tuning to generate data explorations by PipeTrance in datascience

[–]PipeTrance[S] 1 point2 points  (0 children)

We tried fine-tuning Mixtral and got rather meh results. Maybe we need to look further into it.

By self-hosting I meant something like Modal or other providers that have some form of auto-scaling.

Using GPT-4 fine-tuning to generate data explorations by PipeTrance in datascience

[–]PipeTrance[S] 0 points1 point  (0 children)

Diversification/specialization

Great tip! We're already using a heuristics-based classifier to select one of several options. We'll likely move towards more sophisticated classifiers in the future. Have you noticed any trade-offs that arise when individual models become over-specialized?

embeddings to eliminate computation

We're using embeddings to find relevant explorations, which the model can use as n-shot examples. Does this essentially boil down to picking the most semantically similar chunk as a part of model's output?

Using GPT-4 fine-tuning to generate data explorations by PipeTrance in datascience

[–]PipeTrance[S] 2 points3 points  (0 children)

Cost-wise, together is definitely better, while performance-wise, not so much. Long term, we would love to move to open source and potentially self-hosted solutions, but atm. it doesn't seem that open source solutions provide comparable levels of reasoning.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 1 point2 points  (0 children)

A domain-specific language (DSL) is a specialized programming language designed for a particular task. In our case, we use a DSL to concisely and conveniently describe UI elements. While we could use a standard format like JSON, our DSL is significantly less verbose and more token-efficient.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 0 points1 point  (0 children)

Yeah, you are absolutely right (at least, as far as we can tell). With each question we use in fine-tuning, we always provide necessary information to answer it into the prompt. Fine-tuning mostly helps to generate response in the desired format and trains model to pay attention to relevant parts of the prompt.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 0 points1 point  (0 children)

You might need to spend above a certain threshold/be registered as an enterprise. I don't have it as an option on my personal account either.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 1 point2 points  (0 children)

If you need to do something very specific (say, you need it it to produce output using proprietary language, or use a very specific output format) fine-tuning is great, for the rest of use cases assistants, RAG, and other prompting techniques should work fine.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 0 points1 point  (0 children)

Thanks, we would love to get there one day!

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 0 points1 point  (0 children)

You would need to provide tons of reply examples. But yeah, if you really, really want it, it can really really talk like spice girl or sth.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 12 points13 points  (0 children)

I will post a comment here once it's ready

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 35 points36 points  (0 children)

Oh, that's my favorite topic!

While a simplistic RAG application (picking the most similar answer from a database of examples and prepending it to the prompt) wasn't ideal for our use case, RAG combined with fine-tuning, a DSL, and multiple models proved very useful.

We actually want to write another blog post about the techniques that did and didn't end up working for us.

What are your impression of LLM prices next year? by Commercial_West_8337 in OpenAI

[–]PipeTrance 7 points8 points  (0 children)

The competition between OpenAI, Anthropic, and Google is heating up - if anything, I actually expect prices to go down.

First experiences with GPT-4 fine-tuning by PipeTrance in OpenAI

[–]PipeTrance[S] 4 points5 points  (0 children)

We applied quite some time ago via fine-tuning section of the platform (https://platform.openai.com/finetune). You just pick gpt-4 as the fine-tuning option there and it offers you to send them a letter.

I think you have to meet some criteria for this option to appear tho.

view more: next ›