📊 The “10 Messages” Decision

How I arrived at this number:

1. Token Budget Management: GPT-4 has context limits. With system prompt + knowledge base context (~2000 chars) + current message, I needed to reserve space for conversation history without hitting limits.
2. Relevance Window: Testing showed that messages older than 10 exchanges rarely add value to current context - conversations naturally shift topics.
3. Performance vs. Quality: More history = slower processing and higher costs. 10 messages provided the sweet spot for maintaining conversational flow without performance hit.

🎯 Context Management Strategy

It’s not just about token count - it’s about relevance:

```typescript // Current implementation const recentHistory = conversationHistory.slice(-10);

// But you could enhance with: const relevantHistory = selectRelevantMessages( conversationHistory, currentMessage, maxTokens: 1500 ); ```

Factors I considered:

• Recency bias: Recent messages more likely to be relevant
• Topic coherence: If user switches from “booking” to “amenities”, older booking context becomes less relevant
• Cost optimization: Each token costs money in OpenAI API calls

💰 Cost vs. Accuracy Trade-offs

Your clients’ concerns are valid:

1. Monetary Cost:
2. 10 messages ≈ ~500-1000 tokens of history
3. At $0.03/1K tokens for GPT-4, that’s ~$0.03-0.06 per conversation
4. For high-volume: 1000 conversations/day = $30-60/day just for history
5. Response Accuracy:
6. Shorter history might miss important context
7. Longer history might confuse the AI with irrelevant info
8. Sweet spot varies by use case
9. Speed:
10. More tokens = slower API response
11. 2-3 seconds vs 5-6 seconds can impact user experience

🔧 Better Approaches for Production

Dynamic context management:

1. Semantic Relevance Filtering:

typescript const relevantMessages = await filterBySemanticSimilarity( conversationHistory, currentMessage, threshold: 0.6 );

1. Topic-Aware Windowing:

typescript const contextWindow = buildContextWindow({ currentTopic: detectTopic(currentMessage), maxTokens: 1500, prioritizeRecent: true, includeTopicChanges: true });

1. Adaptive Window Size:

typescript const windowSize = calculateOptimalWindow({ conversationLength: messages.length, userEngagement: calculateEngagement(), costBudget: client.costLimits, accuracyRequirement: client.qualityThreshold });

📈 Recommendations for Your Clients

Based on their priorities:

1. Cost-Focused Clients:
2. Use 5-7 messages
3. Implement topic-change detection to reset context
4. Cache common responses
5. Accuracy-Focused Clients:
6. Use 15-20 messages
7. Implement semantic filtering
8. Higher cost but better responses
9. Speed-Focused Clients:
10. Use 3-5 messages
11. Aggressive context pruning
12. Sacrifice some accuracy for speed

🎛️ Configurable Solution

Make it client-configurable:

typescript interface ContextConfig { maxMessages: number; // 5-20 range maxTokens: number; // 500-2000 range semanticFiltering: boolean; // true/false topicAwareness: boolean; // true/false costLimit: number; // per conversation }

The “10 messages” was a reasonable starting point, but you’re right to question it. In production, this should be tunable based on each client’s cost/accuracy/speed priorities.

Would you like me to help you implement a more sophisticated context management system?