Adding AI features via the OpenAI API — chatbots, summarisation, content generation, semantic search — is genuinely easier than it has ever been. It is also easier than ever to ship an AI feature that hallucinates confidently, costs far more than expected, or sends sensitive data somewhere it should not go. This guide states plainly what the API can and cannot do, where the costs actually come from, what happens to your data, and when an AI feature is the wrong answer.
What the OpenAI API can actually do — and what it cannot
The API is strong at: drafting and summarising text, answering questions over documents you provide (retrieval-augmented generation), classifying and extracting structured data from unstructured text, conversational interfaces over a defined knowledge base, and code assistance. These are pattern-matching and language tasks, and current models handle them well when the scope is bounded and outputs are verified.
It cannot: guarantee factual accuracy (models hallucinate — producing plausible, confident, wrong answers), reason reliably about novel complex logic, or replace domain expertise and human judgment. Any AI feature whose failure mode is "confidently wrong answer presented as fact" needs human review, confidence thresholds, or a design that makes uncertainty visible — not just a better prompt.
RAG vs fine-tuning vs prompting: choosing the technique
| Approach | Use when | Limitation |
|---|---|---|
| Prompting | General tasks, quick prototypes, behaviour shaping | No new knowledge; brittle across model updates |
| RAG (retrieval-augmented generation) | Answering from your documents — support bots, policy Q&A, document search | Only as good as your retrieval; needs document pipeline maintenance |
| Fine-tuning | Consistent style/format at scale, domain-specific phrasing | Does not reliably teach new facts; retraining cost on each update |
Most business AI features should be RAG, not fine-tuning. A common and expensive mistake is fine-tuning a model to "know" company documents — retrieval over a well-maintained document store is cheaper, more accurate, and updatable without retraining.
Token costs: where the money actually goes
OpenAI charges per token — roughly per word-fragment — for both input and output, with different rates per model. The cost drivers that surprise teams:
- Context window bloat: stuffing entire documents or long conversation histories into every request multiplies input tokens. Smart retrieval (send only relevant chunks) and conversation summarisation keep costs linear instead of explosive.
- Model choice: the most capable models cost many times more per token than smaller ones. Many tasks — classification, extraction, simple Q&A — work fine on smaller, cheaper models. Defaulting everything to the flagship model is the fastest way to an alarming bill.
- Unbounded usage: a public-facing chatbot with no rate limits, caching, or usage caps is an open wallet. Semantic caching of common questions, per-user quotas, and spend alerts are operational necessities, not optimisations.
- Output verbosity: long generated responses cost output tokens at premium rates. Constraining response length and format is both a UX and a cost decision.
Cost depends on usage volume, model choice, and implementation discipline. Ask for an architecture that includes a cost model — projected token usage per user action, per model tier — before building, and insist on spend monitoring and caps from day one.
Data privacy: what happens to what you send
When you call the OpenAI API, your prompts and documents leave your infrastructure. The practical considerations:
- Do not send what you cannot afford to expose: customer PII, credentials, trade secrets, and regulated data should be redacted or kept out of prompts entirely. Design the data flow so sensitive fields never reach the API.
- Review the data-use terms for your tier: API data-handling commitments differ from consumer product terms — read the actual API terms for your account type and region rather than assuming.
- Consider data residency needs: if your compliance situation requires data to stay in a region, verify what the provider offers before architecting around the API.
- Log carefully: prompt and response logging is useful for debugging and dangerous for privacy. Decide what gets logged, for how long, and who can see it — before launch.
What can go wrong
- Hallucinations presented as fact: the signature failure. Mitigations: grounding answers in retrieved documents, showing sources, confidence thresholds, and human review for high-stakes outputs.
- Prompt injection: users (or documents) can manipulate the model's instructions — extracting system prompts, bypassing guardrails. Treat all model input as untrusted; validate outputs, not just inputs.
- Model updates changing behaviour: providers update models, and pinned versions get deprecated. A feature tuned on one model version can degrade silently. Version pinning plus regression testing on model changes is part of maintenance.
- The -to-production gap: a chatbot that impresses in a with curated questions fails on real user input — typos, edge cases, adversarial queries, out-of-scope requests. Production AI needs fallback handling, escalation paths to humans, and analytics on failure patterns.
When an AI feature is the wrong answer
Not every problem needs a language model. If the task is deterministic — calculations, lookups, rule-based decisions — code is cheaper, faster, and exactly correct every time. If accuracy must be 100% with no human review, current models cannot promise that. If the data cannot leave your infrastructure and no compliant deployment exists, the constraint decides. And if the "AI feature" is really a search box with extra steps, build the search box. The test worth applying: does the probabilistic nature of the model add value, or just risk?
How an OpenAI integration typically runs
- Use-case validation: define the task, the failure modes, and whether probabilistic output is acceptable — kill AI ideas that need determinism before spending anything.
- Data flow and privacy design: map exactly what data reaches the API, what gets redacted, logging policy, and the compliance review — before any code.
- Technique selection and prototyping: prompting vs RAG vs fine-tuning, chosen by evidence; prototype against real documents and real user questions, not curated demos.
- Guardrails engineering: grounding in retrieved sources, output validation, prompt-injection defences, confidence thresholds, and human-escalation paths for high-stakes outputs.
- Cost architecture: model tiering per task, retrieval discipline, caching, per-user quotas, and spend alerts with hard caps.
- Evaluation and launch: test against adversarial and edge-case inputs, instrument failure analytics, launch with monitoring on quality signals — not just uptime.
FAQs
Will the AI give wrong answers to our customers?
It can — language models hallucinate, producing plausible but wrong answers. The mitigation is architectural, not just better prompts: ground answers in your retrieved documents, show sources so users can verify, set confidence thresholds, and route high-stakes questions to human review. Any vendor who promises the AI "won't make mistakes" is either inexperienced or misleading. Design for fallibility.
What data of ours does OpenAI see?
Whatever you send in API requests — prompts, documents, conversation history. That is why the data flow must be designed before coding: redact PII and secrets before they reach the API, review the API data-use terms for your account tier, and decide logging and retention policy up front. The rule is simple: do not send what you cannot afford to expose.
Should we fine-tune a model on our company data?
Usually not as a first step. Retrieval-augmented generation — searching your documents and giving the model the relevant chunks — is cheaper, more accurate for factual Q&A, and updatable without retraining. Fine-tuning is for consistent style and format at scale, not for teaching facts. Most businesses that jump to fine-tuning are buying complexity they do not need.
How do we control AI costs?
Four levers: choose the smallest model that handles each task (not everything needs the flagship), retrieve smartly instead of stuffing context windows, cache repeated queries semantically, and enforce per-user quotas with spend alerts and hard caps. Insist on a projected cost model per user action before building — "we will monitor it later" is how bills surprise you.
What happens when OpenAI updates their models?
Model behaviour can shift with updates, and pinned versions eventually get deprecated — a tuned feature can degrade silently. The defence is version pinning plus a regression test suite run against model changes, treating model updates like dependency upgrades: tested, reviewed, and rolled out deliberately rather than absorbed blindly.
Considering AI features for your product? Talk to us for an honest assessment — including when a regular search box or rule engine serves you better than a language model. Related: AI chatbot development, AI automation development, and API integration development.