An AI agent is an LLM calling tools in a loop to reach a goal.

A large language model (LLM) like GPT-4.1 is the brain of an AI agent. It reads input, reasons about what to do, and decides what actions to take. But on its own, a model can only generate text — it can't look up data, call APIs, or take actions in the real world. In reality, the LLM is either reasoning about text to generate JSON, or generate a final response.

That's where the rest of the agent architecture comes in. Everything around the LLM can be though of as a harness that helps steer the LLM toward a high quality output.

The model also comes with a finite context window — a limit on how much text it can process at once. Everything the agent needs to know must fit within this window: instructions, tool definitions and responses, conversation history, and data. Context engineering (which closely relates to how the harness is set up around the LLM) is one of the most important skills an AI engineer can have.

The system prompt is where you define who the agent is and what rules it must follow. For our example, we're building a customer support agent for an online electronics retailer.

The system prompt is the first message placed into the context window. It consumes tokens, so it needs to be concise but complete.

1You are a customer support agent for TechGadgets, an online electronics retailer.
2
3Your role is to help customers with order inquiries and process returns.
4
5RULES:
6- Only access order data for the customer you are currently helping
7- Never share one customer's data with another customer
8- You may only process returns for orders delivered within the last 30 days
9- Always confirm the action with the customer before executing a return
10- Be concise, friendly, and professional

Tools are functions that the agent can call to retrieve data or take actions. The model doesn't execute the tools itself — it tells your code which tool to call and with what arguments. Your code executes the function and returns the result to the model.

Tool definitions also go into the context window — the model needs to see the available tools to decide which one to call. Our customer support agent has two tools:

1{
2  "type": "function",
3  "function": {
4    "name": "get_order_data",
5    "description": "Retrieves order details for a customer by their order ID. Returns order status, items, delivery date, and total.",
6    "parameters": {
7      "type": "object",
8      "properties": {
9        "order_id": {
10          "type": "string",
11          "description": "The unique order identifier (e.g., ORD-12345)"
12        }
13      },
14      "required": [
15        "order_id"
16      ]
17    }
18  }
19}

1{
2  "type": "function",
3  "function": {
4    "name": "create_return",
5    "description": "Initiates a return request for a specific order. Creates a return shipping label and schedules a refund.",
6    "parameters": {
7      "type": "object",
8      "properties": {
9        "order_id": {
10          "type": "string",
11          "description": "The order ID to process the return for"
12        },
13        "reason": {
14          "type": "string",
15          "description": "The reason for the return",
16          "enum": [
17            "defective",
18            "wrong_item",
19            "not_as_described",
20            "changed_mind"
21          ]
22        }
23      },
24      "required": [
25        "order_id",
26        "reason"
27      ]
28    }
29  }
30}