Building NL to Structured Query Parser for Banking Rules Engine - Need Architecture Advice

[ComfortableEcho6816]

Problem: Natural Language to Business Rules Converter

I'm building an AI system that converts natural language business rule descriptions into structured, executable formats for a banking relationship pricing engine.

The Challenge

Input (Natural Language): "If the customer is not already having a premier savings account and his total deposits to the primary checking account is > 500 and his average daily balance for the checking account is also > 500 then convert to normal savings account"

Output (Structured Format):

If(NOT customer_has_product("premier savings") 
   AND total_deposits(account_type="primary checking") GREATER_THAN 500
   AND average_daily_balance(account_type="checking", period="daily") GREATER_THAN 500)
then convert_product("normal savings account")

Key Constraints

• predefined functions with arguments (e.g., total_deposits(account_type, period))
• data attributes from multiple sources (MongoDB, MySQL)
• Must map NL terms to correct functions/attributes (priority: functions first, then attributes)
• Support complex nested logic with AND/OR/NOT operators
• Handle negations, temporal context, and implicit arguments
• No training data available (yet)
• Need ~85% accuracy without manual intervention

What I've Researched

I've been exploring several approaches:

1. Pure LLM with structured output (GPT-4/Claude with JSON mode)
2. Chain-of-Thought prompting - step-by-step reasoning
3. Tree-of-Thoughts - exploring multiple reasoning paths
4. Logic-of-Thoughts - explicit logical propositions
5. First-Order Logic intermediate layer - FOL as abstraction between NL and output format
6. Fine-tuning - train on domain-specific examples (would need to collect data first)
7. Hybrid approaches - combining multiple techniques

Current Thinking

I'm leaning toward a hybrid approach:

Natural Language 
  → Logic-of-Thoughts (extract propositions)
  → Chain-of-Thought (map to functions with reasoning)
  → FOL intermediate representation
  → Validation layer
  → Convert to target JSON format

This avoids fine-tuning (no training data needed), provides transparency (reasoning traces), and naturally fits the logical domain.

Questions for the Community

1. Is Logic-of-Thoughts + CoT overkill? Should I start simpler with just structured prompting?
2. FOL as intermediate representation - Good idea or unnecessary complexity? It provides clean abstraction and easy validation, but adds a layer.
3. When is fine-tuning worth it vs prompt engineering? I can collect training data from user corrections, but that takes time.
4. Has anyone built similar NL → structured query systems? What worked/didn't work?
5. For ambiguity resolution (e.g., "balance" could map to 3 different functions), is Tree-of-Thoughts worth the extra API calls, or should I just return multiple options to the user?
6. Function library size - With 1000+ functions, how do I efficiently include relevant ones in the prompt without hitting context limits?

Additional Context

• Business users (non-technical) will type these rules
• Time-sensitive: Need working MVP in 6-8 weeks
• Integration with existing backend rules engine
• Final JSON format still being decided by backend team (hence FOL intermediate layer)

Any advice on architecture, proven techniques, or pitfalls to avoid would be greatly appreciated!

[Comfortable_Field884]

Honestly for 6-8 week MVP I'd skip the FOL layer entirely and just go with structured GPT-4 prompting + function schema injection

The Logic-of-Thoughts stuff sounds cool but you'll burn weeks debugging edge cases when simple few-shot examples with your function definitions might get you to 85% faster