Business Challenges
As organisations embrace GenAI for enterprise search and document Q&A, the challenge goes far beyond generating fluent responses. The challenge goes beyond delivering responses that are not only fluent but also factually accurate, well-structured, and backed by credible sources.
During our projects, we encountered the following challenges:
- Fluent responses with poor factual grounding
- Limited source diversity (responses from only 1 or 3 documents)
- Inconsistent citation formats causing confusion
- Balancing trade-offs between response quality and latency & cost
- Absence of standardized performance metrics to measure output reliability
Our Evaluation Framework
Our goal was to develop a scalable, measurable framework that consistently produces high-confidence answers. Here’s how we approached it:
- Chunking Strategy: Experimented with multiple chunk sizes and finalised using a smaller chunk size, as it enhances semantic retrieval accuracy, reduces context overflow, improves LLM response quality, and lowers inference cost.
- Model Selection: Adopted GPT-4.1-mini for general queries and detailed research tasks to optimise the balance between depth, performance, and cost-efficiency.
- Citations: Transitioned from basic page numbers to a structured format of Evaluation Title – Document Name – Citation Numbers, improving traceability and user trust.
- SME Feedback Loop: Actively engaged subject matter experts throughout the project. Their continuous feedback helped fine-tune tone, prompt design, and response structure (e.g., numbering, tables).
- Performance Metrics: Leveraged Azure metrics such as Relevance, Coherence, F1 and similarity to evaluate and iteratively improve AI-generated responses.
- Retrieval Strategy: Dropped summary/keyword methods in favour of semantic chunking, which improved consistency and context-rich retrieval across 5000+ documents.
The Impact We Delivered
- Standardized and clear citation formats
- Increased diversity in source documents retrieved per query
- Reduction in hallucinations and irrelevant answers
- Enhanced SME satisfaction through improved retrieval accuracy and contextual relevance.
- A future-ready framework adaptable for new topics and evolving GenAI models
Evaluating GenAI responses is not a nice-to-have, but it is the foundation for building reliable and trustworthy assistants. Our experience has shown that the right combination of chunking, retrieval strategies, evaluation metrics, and expert input leads to accurate, structured, and dependable responses.
Looking to build Intelligent Assistants that scale with confidence and gain user trust?
Let’s connect to collaborate and take your GenAI journey to the next level.