Infra & cost
API
An interface that lets developers send requests to an AI model and get responses programmatically — the way most AI tools talk to LLMs.
01 ——
In plain English
An API (Application Programming Interface) is the technical way one piece of software talks to another. AI APIs let you send a prompt to a model and get a response back over the internet — powering nearly every AI tool you use.
How it works:
- Your app sends a request (a prompt + parameters) to the model provider
- The provider runs inference on their servers
- The response comes back as JSON
- You're billed based on tokens used
Common AI API providers:
- OpenAI (GPT-4, GPT-5)
- Anthropic (Claude)
- Google (Gemini)
- Together AI, Replicate, Hugging Face (open models)
Why it matters: Most AI tools don't host their own models — they call an API. Pricing, latency, and rate limits set by the API provider directly shape what AI products can do.
02 ——
Related terms
Inference
The process of running a trained AI model to generate a response — as opposed to training the model.
Latency
The time it takes an AI model to respond to a request — from when you hit send to when the first or final word appears.
Streaming
Sending an AI model's response token-by-token as it's generated, so the user sees text appear immediately instead of waiting for the full reply.
Prompt Caching
A feature that stores parts of a prompt the model has already processed, making repeat or follow-up requests much faster and cheaper.
LLM
Large Language Model — the type of AI behind tools like ChatGPT and Claude, trained to understand and generate text.
Back to glossaryLast reviewed May 2026