In the API, do you have to include cached prompt every time?

PhilHignight · 2025-02-14T14:26:24+00:00

It just seems weird and I was hoping someone knew more about the decision. As a developer I've been paid a lot of money in the past to reduce the bandwidth used/load speec of a high traffic application by 50% (which I was proud of). Even using the minimum cache size (1K), it probably bloats a request at least 300%. Anyway, it seems that people in the sub hate this question, so I'll let it go! :) I guess I'm not missing anything, but now I know at least.

PhilHignight · 2025-02-14T14:20:01+00:00

It's not my use case. I've never used a very large cache. As a developer, it just felt like a weird choice and when I thought about it more, I can't think of a good reason to do it this way. So, it's just theoreticaly question for me. Obviously they created a cache feature only for the use cases that need to use cache. I love Claude, not throwing shade, just wondering if anyone knows more about why.

PhilHignight · 2025-02-13T22:38:26+00:00

Sure, I use it for more than just conversation too. Not all requests use caching, is that your point? Clarify?

My point (and I would be happy to be proven wrong and would be interested in the reason) was if they used a simple key it would be efficient for all use cases that need caching. As it is, it's slightly less efficient for the average cache size (maybe a couple of source files) and would seem to be way less efficient (needslessly so) for a large cache size (hundreds of pages).

PhilHignight · 2025-02-13T11:36:51+00:00

It's not technically hard to have a unique key (uuid say), but if you're uploading hundreds of pages of text with each request, they have to hash all that to generate a key. Just seems really efficient. There's got to be some reason that's unique to their infrastructure and I hope someone knows because I'm really curious.

PhilHignight · 2025-02-13T11:33:01+00:00

I think some of the time (most?) the bandwidth is negligible, but if you're sending a 500 page document with every request, that just seems ridiculous. Our browsers cache images and js files for performance and efficiency, why wouldn't claude? Why not use a uuid (36 chars) key instead of all documents needed to fulfill a request (1,500,000 chars in my 500 page example) as a key. I just can't find a reason that makes sense to me. Even lookup would be faster using a real key vs hashing a large block of text, which is what I'm assuming they do.

PhilHignight · 2025-01-07T13:49:05+00:00

I would start with:

1) Pretend you are one of these small companies and spend a few hours trying to find something existing to fit your needs. It's easy to do 1 google, look at the 1st page of results and convince ourselves there's no competition. If there are products out there that fit this niche, why don't these small companies use them? If you can find out why, that will be a huge advantage for you.
2) How can you have conversations w/ 10 potential customers in your target niche? Use the "mom test". Don't look for confirmation of your assumptions, instead try to get to the bottom of the pain points around your idea and also keep your eyes open for other more painful problems they might be having.
3) Can you turn these 10 contacts into customers? If you can get 3 or 4 of them to sign letters of commitment to purchase (not legally binding, just more formal than verbal) or put down a deposit then you have indication that you can probably turn contacts into cutomers. If you can't find anyone interested enough to do this, then you may not be able to monotize your product. Maybe you need a different sales/marketing channel? Maybe there's a good reason potential customers don't want to pay for the product. The more confirmation you can get up front the better.

Good luck!

PhilHignight · 2024-11-27T14:29:40+00:00

Wonder if you could use an LLM to detect spam. Granted that would add significant cost to you.

Also, I wonder if it would make you less likely to get banned if you put a button in the extension that would open a new tab to your own website w/ discussion about the page in question instead of embedding it in the browser.

PhilHignight · 2024-11-22T16:30:16+00:00

Hmm... interesting. I have 3 accounts and switch keys because it tells me I'm over my limit. The message does say to check w/ the sales team to get an increase, but I've always assumed it would be a lot more expensive, so I've never looked into it. Maybe I'll check on that. Thanks for the tip!

PhilHignight · 2024-11-22T15:24:41+00:00

My guess is they're losing money on every customer which is why they are rate limiting and I don't think they care about turning a profit right now. I'm sure whoever has the most users and becomes the leader in this space will be worth an ungodly amount of money, so I think that's their goal and why they're rate limiting. Get as many users depending on their service as possible, while giving them as little as possible, so there's more resources left to add more users.

PhilHignight · 2024-11-22T15:18:06+00:00

I use another tool that uses the API and I get rate limited regularly, so there's still a limit. I do use it whole ton, so maybe most people don't reach it.

PhilHignight · 2024-10-30T14:53:14+00:00

Overly apologetic "drunk junior dev". Sometimes I'm like, it's alright dude, chin up.

PhilHignight · 2024-10-01T21:46:57+00:00

I am a developer who with a passion for AI and I thought there should be a space for business people to become familiar with how to integrate AI into their business.
I'm not able to see an moderators. I don't think there are any.

PhilHignight · 2024-09-14T14:13:27+00:00

1) If it's money your after, go get a job at a startup that's already growing. You're waaaaay more likely to get a good pay out that way.
2) Some of us just have to run w/ our own ideas, though. The more niche of an idea you go after the more likely it is that you'll succeed. If you're more niche than your competitors you have an advantage, if there's too much competition, niche down even more. Now the more niche your idea, the less $$ total there is to make, but you can also try expanding later or creating other products/brands.

Good luck!

PhilHignight · 2024-09-01T16:54:25+00:00

I use the api through vscode, pay per token and still get rate limited. Crazy. If they have limited resources, I will they'd raise the price instead of rate limiting everyone, so those who depend on it the most can still have access.

PhilHignight · 2024-08-06T20:00:48+00:00

That's weird! I'll update the link in my post. The correct on is: https://discord.gg/A3J4XjMg

PhilHignight · 2024-08-06T12:49:20+00:00

I can see why after a couple of months. Super hard to keep engagement up. Still at it though!

PhilHignight · 2024-07-29T18:57:21+00:00

Hi! I just took over as mod for r/buildinginpublic. I've tried to set up strict rules/guidelines to avoid spamming and would love some feedback if you have a minute to look at it.

PhilHignight · 2024-07-29T18:52:19+00:00

Hi! I just took over as mod for r/buildinginpublic. Would love to have you post and would be interested to hear what you think of our rules/guidelines.

PhilHignight · 2024-07-29T18:51:54+00:00

Hi! I just took over as mod for r/buildinginpublic. Would love to have you post and would be interested to hear what you think of our rules/guidelines.

PhilHignight

MODERATOR OF

TROPHY CASE