Artificial Intelligence
RAG or fine-tuning for enterprise knowledge? Start with the failure.
An assistant that cannot find a current policy has a different problem from one that finds it but returns the wrong format. Diagnose that difference before training.
- User question
- Authorized sources
- Grounded response
Suppose an internal assistant gives an employee the wrong expense policy. Before choosing a new model, inspect the failure. Did the system retrieve an outdated document? Was the right policy missing? Did the model ignore a relevant passage? Or should that employee never have received the document?
Those failures need different fixes. Our starting recommendation for changing internal knowledge is to evaluate retrieval first. For a repeatable behavior problem, establish a prompt-based baseline and then assess whether fine-tuning adds enough value to justify the training and maintenance work. Neither approach replaces access control or testing.
Knowledge and behavior are different problems
Retrieval-augmented generation, or RAG, supplies relevant source material while an answer is being generated. AWS describes a knowledge base searching connected data for relevant information and using that information to improve the response. Retrieval provides a way to connect an application to organizational documents rather than relying only on what a model learned during training.
Fine-tuning uses training data to adapt a model for particular tasks or behavior. AWS lists supervised fine-tuning among its model customization methods. A fine-tuning evaluation might target a consistent response structure or a specialized task. It should not be treated as a live connection to an organization’s changing records.
These methods can be combined, but combining them introduces more components to evaluate. Start with the smallest intervention that addresses a measured failure.
Diagnose before choosing
The following framework is our proposed way to triage an enterprise knowledge pilot. It is not a benchmark comparing commercial models.
Scroll sideways to compare all columns.
| Observed failure | First investigation | Candidate intervention |
|---|---|---|
| The relevant document is absent | Source coverage and ingestion | Add or repair the source connection |
| The wrong passage is retrieved | Document structure and retrieval tests | Improve chunking, indexing or ranking |
| The passage is correct but the answer is unsupported | Instructions and answer evaluation | Improve prompting and require grounded answers |
| A stable output format is repeatedly wrong | Prompt baseline and representative examples | Evaluate fine-tuning if simpler changes are insufficient |
| A user receives restricted information | Identity and document permissions | Fix authorization before further model work |
Do not count an answer as correct just because it contains a citation. Check whether the cited passage actually supports the answer and whether the reader is authorized to see it.
Build the test set before the system grows
Ask the intended users for questions they genuinely need to answer. Include ordinary questions, ambiguous wording, conflicting documents and requests that cannot be answered from the available material. Keep a separate set of questions out of any training examples so the evaluation is not just a memory test.
For each question, record the acceptable evidence and the expected action when evidence is missing. A useful outcome may be a request for clarification or a referral to a document owner. Forcing the assistant to answer every question would hide uncertainty from the people relying on it.
Evaluate retrieval and generation separately. If the correct passage never reaches the model, a more fluent answer is not progress. If retrieval succeeds but the response adds unsupported details, investigate generation rather than rebuilding the document index.
Treat permission changes as a test case
For a proposed internal assistant, test with users who have different document access. Remove an entitlement and repeat the same query. Check citations, excerpts, cached answers and conversation history as well as the final response.
These are recommended acceptance checks, not a claim that every RAG service enforces permissions automatically. The application team needs to verify the behavior of its selected connectors, retrieval service and storage settings. Sensitive material should not be used in an early prototype without approval from its owner.
Decide on evidence, not a preferred technique
Compare approaches using the same questions and operational requirements. Record answer correctness, unsupported claims, permission failures, response time and the effort needed to keep sources current. Add the cost of evaluation and maintenance to the cost of model calls.
AMARK INTELLIGENCE is AMMEGATECH’s business for human-centered AI, including organizational knowledge and agents. A useful discussion starts with the task, the source documents and the failures the organization cannot accept. Those inputs are more valuable than a requirement to use RAG or fine-tuning before the problem has been examined.