OpenAI turns the model into a 150-millisecond decision-maker

OpenAI Is Pulling AI Out of the Chat Box

OpenAI will launch the Decisions API at its Developer Day on September 30. It is neither another chat interface nor an attempt to stuff a full-scale model into business systems. Instead, it narrows model capabilities to a more specific problem: quickly making a single structured decision from a limited set of options.

According to publicly available information, the Decisions API is currently based on OpenAI's small Luna model, with a target response time of approximately 150 milliseconds. Compared with calling Luna through the regular API, this interface is roughly 10 times faster. For customer-service triage, content moderation, ticket routing, and Agent workflows, 150 milliseconds is already close to the point where "the business logic can simply wait for the result."

This also reflects a clear shift in OpenAI's recent product direction: models are no longer treated only as general-purpose entry points that can chat and write code. They are beginning to be broken down into specialized judgment components that can be embedded into business processes.

The Problem Is Not "Is the Model Smart Enough?" but "Can It Make a Decision in Time?"

Traditional large-model APIs are suited to open-ended tasks. Developers give the model a block of context and ask it to generate text, call tools, or complete a chain of complex reasoning. But many production systems do not need these capabilities.

For example, when a customer-service ticket arrives, the system may really need only three fields:

• Issue type: payment, login, logistics, or other;
• Priority: normal, urgent, or major outage;
• Next action: transfer to a human agent, call a refund tool, or search the knowledge base.

When using a general-purpose model, the system often has to handle prompts, output formatting, retries, JSON parsing, and exception fallbacks. Even if the model ultimately needs to return only an enum value, it may first generate an explanation, or incur additional latency because of complex context.

The idea behind the Decisions API is to fix the answer space before the call is made. Developers provide context such as text or images, define a question and a limited set of possible answers, and then receive the selected option, confidence, and structured information for use by downstream logic.

In other words, it is more like a model-driven "soft-rules decision engine" than a scaled-down chatbot.

This design sacrifices some openness, but delivers three things that production systems care about more: response time, output stability, and controllability. For a task that only requires choosing one of five options, allowing the model to freely generate an answer is inherently wasteful.

What Does 150 Milliseconds Mean?

In a demonstration environment, 150 milliseconds sounds like nothing more than an attractive number. In a real business, it can change the system architecture.

A typical Agent workflow may include intent recognition, authorization checks, tool selection, parameter completion, and tool execution. In the past, developers would typically have a general-purpose model handle several of these steps. This is flexible, but every additional model call increases latency, cost, and the probability of failure.

The Decisions API can be inserted at the points in these workflows that are best suited to "making a choice":

1. Determine which tool to call next based on the user's input;
2. Decide whether to continue, retry, or transfer to a human based on the tool's result;
3. Choose a lightweight model, a deep-reasoning model, or human review based on task priority;
4. Choose whether to allow, rate-limit, or block content based on risk labels.

If a workflow contains three model-based decisions, and each one drops from several seconds to approximately 150 milliseconds, the overall interactive experience will feel noticeably different. This is especially true in customer service, search, recommendation, and real-time risk control, where users generally do not distinguish between "the model is slow" and "the backend system is slow." They simply conclude that the product is sluggish.

However, it is important to note that 150 milliseconds is more likely to be an interface target than an end-to-end latency guarantee for every request, region, or context length. Network round trips, input size, image processing, concurrency, rate-limiting policies, and queuing on the business side will all affect the final result. When evaluating performance, developers should examine P50, P95, and P99 rather than focusing only on the fastest response in a single demonstration.

Four Scenarios Are the Best Places to Start

Customer Service and Ticket Routing

Customer-service systems are among the easiest scenarios in which to deploy this technology. User descriptions are often insufficiently standardized, and relying solely on keyword rules creates many edge cases. But allowing a general-purpose large model to answer freely can introduce instability into routing.

The Decisions API can constrain the question to "Which team should this ticket be assigned to?" and limit the answers to options such as payments, accounts, security, logistics, and human review. The model handles natural-language understanding, while the ticketing system handles assignment. The boundary between the two is clear, making issues easier to trace when something goes wrong.

The system can go one step further by using confidence as a routing condition: high-confidence requests are automatically sent to the corresponding queue, medium-confidence requests are passed to a second-level classifier, and low-confidence requests go directly to human review. In this arrangement, the model does not replace the ticketing system. It becomes a semantic routing layer in front of it.

Content Moderation and Policy Selection

Moderation systems often involve more than simply deciding whether content is "allowed" or "not allowed." The same piece of content may need to be approved for publication, downranked in recommendations, restricted in distribution, delayed for publication, or sent for human review.

The key in these tasks is not to have the model write an analytical report. It is to have the model select one of several predefined policies and provide sufficient grounds and confidence for that judgment. The Decisions API is suitable for this layer of decision-making, while the rules engine behind it can continue to handle deterministic conditions such as regional policies, age restrictions, account levels, and historical behavior.

This combination is important: the model handles ambiguous semantics, while the rules engine handles hard constraints. Enterprises should not hand final compliance responsibility to a generative model that cannot be fully explained.

Decision Nodes in Agent Orchestration

One of the easiest problems to encounter in Agent systems is that "being able to call tools" does not mean "knowing when to call them." If every step is left to a general-purpose model, the workflow may perform repeated searches, make ineffective calls, choose the wrong tool, or execute in loops.

The Decisions API can serve as a lightweight decision node responsible only for answering questions such as:

• Should the current task call a search tool, a database, or a code-execution tool?
• Does this request need to be escalated to a more powerful reasoning model?
• After receiving the tool's result, should the system continue, ask the user a follow-up question, or hand the task to a human?

This architecture is similar to breaking a large program into an explicit state machine. The general-purpose model handles complex problems, while the Decisions API quickly switches between a limited number of branches. For high-concurrency Agent platforms, this is easier to control in terms of cost than having a single large model handle all orchestration responsibilities.

Ambiguous Areas of Business Rules

In banking, insurance, e-commerce, and enterprise software, rules engines remain the backbone. The genuinely difficult parts are often the information contained in natural language, images, or unstructured materials.

For example, a rules engine can clearly determine amounts, regions, account status, and time windows, but it has difficulty directly understanding an uploaded repair photo, a contract clause, or a complaint description. The Decisions API can first provide a limited result such as "recommend approval," "recommend requesting additional materials," or "recommend human review." The rules engine can then combine that result with hard conditions to make the final determination.

This is more reasonable than having the model directly return "approve or reject." The model provides an auditable intermediate judgment, while the final decision remains within the enterprise's existing authorization, risk-control, and compliance systems.

Compared with General-Purpose Model APIs, the Tradeoffs Are Clear

The Decisions API's advantages are built on restrictions. Developers must define the question and options in advance, which means it is not suitable for open-ended question answering, long-form writing, complex code generation, or exploratory tasks where the answer set changes frequently.

It is closer to a classifier for natural-language input, but more capable than a traditional classifier at handling semantics, context, and image information. The challenge is that enterprises must first break their business decisions down clearly before they can determine which judgments are suitable for the API.

If a product team has not even defined "what counts as correct," calling the interface will not automatically resolve process confusion. A model can help make judgments, but it cannot replace business rules, responsibility boundaries, or exception-handling design.

Another practical issue is confidence. The confidence returned by the model should not be interpreted as an absolutely reliable probability in the statistical sense. It is better used as a routing signal than as a standalone safety threshold. Production environments still need to determine thresholds using offline evaluations, manual sampling, error distributions, and online monitoring.

Developers should pay attention to at least the following metrics:

• Accuracy and recall for different options;
• The proportion of low-confidence samples;
• P50, P95, and P99 latency;
• Performance differences across text, image, and multilingual scenarios;
• The business cost of misclassification;
• The outcome when the rules engine and model judgment conflict.

TypeSafe's Jev Will Be a Direct Competitor

In terms of product positioning, TypeSafe's Jev is currently one of the closest competing products. Both are attempting to compress model capabilities into low-latency, structured, and controllable decision-making tasks rather than continuing to compete primarily around the chat experience.

The Decisions API's advantage is that it can directly leverage OpenAI's existing models, APIs, and developer ecosystem. Teams that already use Luna to process business text should, in theory, find it easier to migrate some requests to a dedicated decision interface. OpenAI Hub also supports unified access to multiple mainstream models. For teams that need to compare OpenAI, Claude, Gemini, or DeepSeek solutions simultaneously, these decision nodes can be placed into the same OpenAI-compatible calling and monitoring system, with the specific model or service path selected according to latency, price, and accuracy.

But ecosystem advantages do not guarantee that a product will win. If Jev is more flexible in end-to-end latency, observability, decision version management, or private deployment, it can still secure a position in enterprise scenarios. For developers, the real comparison should not be the advertised "10 times faster," but effective throughput and error costs under the same inputs, options, and concurrency conditions.

OpenAI Is Repartitioning Its Model Product Line

Luna itself is positioned as a lighter and lower-cost model. OpenAI is now providing a more specialized Decisions API on top of it, indicating that the company is dividing model capabilities by task: flagship models handle complex reasoning, small models handle large-scale routine processing, and specialized interfaces turn these capabilities into stable business components.

This direction has significant implications for API developers. In the past, model selection often came down to "Which model is the smartest?" Now it is closer to "Which model and interface are suitable for this step?" A complete system may use a powerful reasoning model for a small number of complex requests, Luna for large-scale classification, and the Decisions API for high-frequency routing.

This will also shift evaluation away from a single model leaderboard toward comprehensive metrics within real workflows: how many calls a task requires, how many tokens each call produces, how retries are handled after failures, how long completion takes, and whether the model can consistently output within the specified options.

Conclusion: Not the End Point of AI Decision-Making, but a More Practical Middle Layer

The greatest value of the Decisions API is not that it reduces a model's response time from several seconds to 150 milliseconds. It is that it acknowledges that many enterprise tasks do not need "chat" at all. What they need is a decision module that can understand unstructured input, respond consistently within a limited set of options, and connect to business workflows quickly enough.

This type of interface will not replace rules engines, nor will it replace a complete Agent. It is more like a layer between the two: using a model to process linguistic and visual information that rules cannot easily cover, then returning the result to a deterministic system for execution.

As of September 29, external public information indicates that the Decisions API will be formally announced at the September 30 Developer Day. Its specific availability regions, pricing, quotas, latency definitions, and returned fields should still be confirmed against OpenAI's subsequent documentation. For developers, the first questions worth validating are not "Can the model make decisions?" but three more practical ones: Is it accurate enough on their own data? Does its P95 latency actually meet business requirements? And can its confidence signals and human fallbacks be integrated into existing systems?

If all three questions can be answered affirmatively, the Decisions API may have a chance to become more than a product launch. It could become infrastructure for Agents and real-time business systems.