Insights
Choosing between retrieval and fine-tuning
A decision framework for separating current knowledge, learned behavior, and the controls an enterprise AI system needs.
In brief
Does the failure come from missing information or unsuitable behavior?
- Retrieval supplies evidence at task time; adaptation changes learned behavior.
- Evaluate each subsystem independently before combining them.
- Choose the architecture using failure analysis, maintenance requirements, and cost per accepted result.
Identify what the system needs to learn
Retrieval and fine-tuning solve different parts of an AI system’s problem. Retrieval supplies information at the time of a request. Fine-tuning changes learned behavior through training. They can work together, and a useful architecture decision begins by identifying the failure that needs to improve.
If an assistant lacks a recently revised policy, the missing ingredient is current evidence. If it repeatedly misinterprets a specialized document pattern despite receiving the correct evidence, the gap may involve learned behavior. If it cannot access a restricted source, the problem belongs to authorization. These distinctions prevent a model-training project from absorbing problems that require different controls.
NYIC’s approach is to make the decision through controlled comparisons. First define the task and its evidence requirements. Then improve the baseline sufficiently to expose the remaining limitation. The result may favor retrieval, adaptation, a combination, or a simpler deterministic component.
Understand the information boundary
A model’s parameters contain learned patterns and knowledge acquired through training. They do not provide a convenient authoritative record of every fact or a direct mechanism for enforcing changing access rights. A retrieved document can carry a source identifier, revision date, and permission context that the application can inspect.
The original retrieval-augmented generation paper combined parametric generation with non-parametric retrieval for knowledge-intensive tasks. It provides a foundational example of bringing external evidence into generation. Enterprise systems extend this idea with source governance, freshness, permissions, and task-specific validation. Read the RAG paper.
Retrieval does not guarantee grounded answers. The retriever can miss the relevant passage, return a superseded source, or supply a near match that confuses the generator. The model can also ignore the evidence. An architecture must evaluate the complete chain from source ingestion to final claim.
Understand the behavioral boundary
Fine-tuning can teach output structures, terminology, task procedures, and response preferences. It is most compelling when examples capture a stable behavior that the base model does not perform reliably with available context. The training objective and dataset should match that behavior explicitly.
LoRA offers a way to learn adaptation updates with a relatively small set of trainable parameters. This can make experimentation more accessible, while leaving the central architecture question unchanged. The adapted model still needs to demonstrate a benefit on held-out tasks and retain required capabilities. Read the LoRA paper.
Fine-tuning also creates maintenance obligations. Training data needs review, models need versioning, and future changes need reevaluation. If the desired behavior changes every week, a prompt or rule may be easier to maintain. If it reflects a durable domain convention expressed across many examples, learning may offer more value.
A decision matrix for recurring failures
The following matrix describes starting hypotheses. It is a guide to experiments rather than a universal prescription.
| Observed problem | First intervention to test | Evidence to collect |
|---|---|---|
| Missing current facts | Retrieval from authoritative sources | Recall and freshness |
| Incorrect document access | Authorization and filtering | Permission-boundary tests |
| Repeated output-format errors | Schema constraints and clear examples | Validity and field correctness |
| Persistent specialist interpretation errors | Targeted adaptation | Held-out semantic accuracy |
| Unsupported claims despite good evidence | Grounding prompt, validators, then adaptation | Claim-level support |
| Excessive latency or cost | Context reduction, routing, smaller models | Cost per accepted result |
| Arithmetic mistakes | Deterministic calculation tools | Reconciled numeric outputs |
A team should be willing to stop before training if a simpler intervention resolves the problem. Conversely, it should recognize when repeated prompt patches have become a fragile substitute for a consistent learned skill. The comparison should consider maintenance complexity alongside immediate quality.
A worked internal-policy assistant
Consider an illustrative assistant that answers employee questions about travel policy. The policy changes periodically, and some documents apply only to particular regions or employee groups. The system must identify the controlling source, preserve exceptions, and explain when a question needs human review.
Retrieval is a natural part of the architecture because the answer depends on current, scoped documents. The source pipeline should preserve effective dates, region, document status, and access rules. The retriever should prefer controlling versions while allowing the answer to explain relevant changes when the user asks about history.
Suppose the base model receives the correct passage but repeatedly drops conditions from reimbursement rules. A targeted adaptation might teach condition preservation through examples and counterexamples. The adapted model would still use retrieval for current policy. Training would address a stable interpretation behavior rather than serve as the update mechanism for every policy revision.
The experiment should compare the prompted base model and adapted model against identical retrieved evidence. It should also run an oracle-evidence condition in which the relevant passage is supplied directly. This separates retrieval failures from interpretation failures and makes the adaptation’s contribution easier to measure.
Evaluate retrieval as its own subsystem
Retrieval quality depends on ingestion, segmentation, indexing, query construction, ranking, and filtering. A paragraph split away from its heading may lose meaning. A table converted to plain text may scramble relationships. An embedding match may be semantically related yet insufficient to answer the question.
Measure whether the retrieved set contains the evidence required for the answer. Then measure whether that evidence is placed in a form the generator can use. These are separate tests. A system can achieve high document recall while omitting the specific footnote that controls the answer.
Include negative cases where no authorized source answers the question. The system should identify the limitation rather than fill the gap from plausible background knowledge. Evaluate whether access filtering occurs before evidence reaches the model and whether cached results respect the same scope.
Evaluate adaptation under the same evidence conditions
An adaptation study should hold the evidence pipeline constant unless the experiment explicitly tests a joint change. Otherwise an improvement in retrieval can be mistaken for an improvement in the model. Version prompts, source snapshots, inference settings, and evaluation examples.
Hold out document families and time periods that reflect expected production change. Test unfamiliar phrasing, rare exceptions, incomplete records, and conflicting sources. Measure the ability to retain important conditions as well as answer correctness. A concise answer that omits an exception may look better to a superficial evaluator while being less useful.
Regression tests should cover abstention, instruction following, and general task behavior. Specialized training can produce overly confident domain language. Expert reviewers should judge against evidence and explicit rubrics rather than perceived sophistication.
Combine methods without hiding the dependencies
A combined architecture can use retrieval for current evidence, an adapted model for interpretation, and deterministic checks for output validity. It can also route difficult questions to a larger model or a human reviewer. Each layer should have a defined responsibility and an observable failure state.
The combination introduces version interactions. Updating the embedding model can change the evidence distribution. Updating document segmentation can change which conditions appear together. Updating the generator can change how citations are used. Evaluation should track these dependencies so that a release can be reproduced and rolled back.
A useful operational record captures the question, authorized source identifiers, retrieval configuration, model version, relevant validation results, and final disposition. It should minimize unnecessary sensitive content while retaining enough evidence to investigate failures.
Consider economics across the lifecycle
Retrieval incurs indexing, storage, maintenance, and request-time work. Fine-tuning incurs data preparation, training, evaluation, and model maintenance. Both can affect latency and support complexity. Token price alone cannot resolve the decision.
Estimate cost per accepted result, including retries, review, and escalation. Consider how often source information changes and how stable the task behavior is. A high-volume stable extraction task may justify adaptation. A low-volume assistant answering constantly changing questions may benefit more from a strong general model and carefully governed retrieval.
Account for organizational capacity as well. A team needs the ability to maintain the architecture it chooses. An approach with modestly higher variable cost can be preferable if it reduces operational fragility and allows updates to be verified reliably.
Let failure analysis choose the architecture
The most useful decision is usually specific. A system may need retrieval for evidence, adaptation for a recurring interpretation skill, and a tool for calculations. It may need none of these for a simple classification that a conventional model handles well.
The architecture should follow observed failure modes and measurable requirements. That creates a clearer path from experiment to deployment and avoids treating retrieval or fine-tuning as an identity choice. Both are instruments for building a system that can answer the intended questions with reliable evidence and manageable operating costs.