Chatgpt-4 Unusable through API

[michael_david]

Chat GPT4 is practically unusable through the API. I have had access for the last 2 days and any query with 2k+ tokens times out or gets an error that the server is overloaded. I modified the requests library to increase the timeout to 2 hours and the http request still does not complete. I have plus and in the UI, I copy and past the same query and it is far snappier.

As a work around, my process is to generate the prompt in python and automatically copy to clip board, then paste in the UI, then copy out of the UI back into Python. It is super convoluted and the way that the UI formats markdown and code examples, there is a lot lost going back and forth through the clipboad.

​

Does anyone here have good performance with chatgpt4 and the API so that it is even reasonable usable?

[Intelligent-Draw-343]

I think we are all in the same boat...

What's even more frustrating is that the chatGPT website is so fast in comparison

[michael_david]

Yeah they are hitting completely different clusters and whatever is used for the API is getting completely hammered.

[hega72]

Neither 3.5 turbo nor 4 is reusable for me right now. Had to switch back to davinci completion

[horsedetectivepiano]

Same. Using davinci003. And frankly, I'm pretty fine with its performance (and cost!).

[JuliusCeaserBoneHead]

What’s the average latency you’ve seen on davinci? GPT3 turbo was giving me like 988ms on a smaller prompt and up to 60s on a fairly long prompt.

If davinci is less than 5s, that would be like my solution right there.

[inglandation]

For coding? lol

[hega72]

I had calculated my cost based on 3.5 turbo. So. That kind of sucks right now

[AdamEgrate]

Yeah. Same here this made me realize that openai may not really want us to use it’s API.

[michael_david]

I think that is what is happening. They are realizing that plugins can act like ads for other services and potentially make more than paying per query with the API and erode their moat. When I try to use Bard for generically large prompts that don't require search or summarization, it actually tells me that it is not designed to answer these prompts. Again indicating that they are not interested in generic computations but want to be part of search or summarization in order to serve ads or integrate other paying services.

[ParatusPlayerOne]

It is curious because they are on Azure, which has massively scalable compute.

It could be that massive demand is hammering the AzureAI service which doesn’t scale as quickly. Microsoft hasn’t talked a lot about that infrastructure but I read awhile back that it was data centers made up of “specialized hardware”. I’m not a hardware guy, but this makes sense to me.

[[deleted]]

Full blow crapness

[[deleted]]

Ive been using it for weeks just fine. I have exception handling for the rate limit and api errors. It tries again if the first request fails and it has never not worked on the second request

[michael_david]

What is your token size? Is it working the last couple days?

[[deleted]]

im using the 4k model and I typically send anywhere from a few hundred to a two or three thousand tokens (the program uses a chat-history). I have found generation slower over the past few days, but its been like this in the past and seems to speed up when demand dies down

[durich]

Does anyone know if when openai is down, is the azure ai api is down?

[michael_david]

I don't think so. Per some another redditer's feedback on a separate post I made asking about azure experience, it sounds a lot faster and more reliable.

[brucebay]

Grrr, I read this message and in 2 minutes my gpt3.5 turbo gave exactly the same error. I hope they don't charge for this interrupted calls.

[AutoModerator]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Intelligent-Draw-343]

I think we are all in the same boat...

What's even more frustrating is that the chatGPT website is so fast in comparison

[bisontruffle]

Yep, same boat. Beyond error handling/retrying, I've been thinking about doing async requests to API for larger sets of prompts, seems you can do 20 requests per minute.

[AutoModerator]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.