AI FinOps③：AI 网关集中管理用量与成本控制

가

The body content of the article will be changed to this letter size.

스크롤 이동 상태바

Infrastructure optimization is considered a core policy in artificial intelligence (AI) FinOps. It prioritizes cost-effective models and calls high-performance models only when necessary to minimize token consumption. The FinOps Foundation also recommends rightsizing AI environments to manage costs.

Routing, which distributes requests appropriately, is a key service offered by AI gateway providers. AI gateways serve as intermediaries between applications and multiple large language models (LLMs), making them an ideal point for real-time cost tracking, control and optimization. OpenRouter, Portkey and LLMRouter are among the companies gaining prominence in the global market.

AI gateway services manage external commercial APIs and open models hosted on companies' own GPUs through a single entry point and cost ledger. Commercial API providers charge based on token usage, while self-hosted models require custom cost mapping, which assigns virtual per-token prices through gateway settings.

AI gateways can then measure input and output tokens in real time, including traffic to and from models running on internal GPUs, and estimate total costs. This enables providers to offer real-time cost tracking, departmental showback and chargeback, and spending controls such as rate limiting and budget caps.

Most AI gateway services offer cost management, or AI FinOps, as one of their core features. (Source: Portkey)

In South Korea, AI managed service providers (MSPs), including Bespin Global and MegazoneCloud, are introducing similar AI FinOps services. Although AI FinOps is an additional offering, it helps sustain their core services. Uncontrolled token costs and GPU usage can increase customer dissatisfaction and make contract retention difficult, turning cost optimization capabilities into a competitive advantage.

MegazoneCloud has promoted AI gateway capabilities since introducing its AI operating system, AIR Studio v2, and further developed those features with the launch of v3 earlier this month. Lablup, which specializes in GPU resource optimization and orchestration, also launched AI gateway and AI FinOps solutions late last month. AI Gateway X has entered the market with AI FinOps as a central part of its AI gateway offering.

Lablup provides Backend.AI, a platform for optimizing GPU and AI accelerator resources. At its annual lab | up conference last month, the company unveiled Continuum Router, an LLM gateway, and Continuum Hub, a solution for directly managing and controlling AI token costs, marking its entry into the AI gateway and AI FinOps markets.

Continuum Router acts as a single entry point for all LLM requests within an organization, recording the model used, input and output token counts, and latency for each request. Continuum Hub aggregates these records by organization, team, project and API key to display costs in real time.

According to Lablup, Continuum Hub provides three levels of cost control. First, organizations can set monthly spending and token limits by organization, tier and API key, automatically blocking only the affected key and issuing an alert when a limit is reached. Second, separate budgets can be assigned to individual models, with requests automatically redirected to a designated lower-cost model when a budget is exhausted rather than rejected. Third, costs can be allocated to departments for showback and chargeback billing.

Lablup's Continuum Router AI gateway (top) and Continuum Hub AI FinOps solution.

Lablup identifies three differentiating features of its AI FinOps service: separation of management and request processing, metadata-based data protection, and support for on-premises and air-gapped environments connected to customers' own GPU infrastructure.

Continuum Hub handles policy management and aggregation, while Continuum Router enforces spending limits along the LLM request path. This architecture allows policies established in Continuum Hub, including tier limits, budgets and guardrails, to be distributed simultaneously to multiple Continuum Router instances. It also supports phased deployment, allowing policies to be tested on selected routers before broader rollout or rollback.

Even if Continuum Hub becomes unavailable, Continuum Router continues processing requests and enforcing limits. Policy synchronization resumes within 30 seconds after Continuum Hub recovers. According to Lablup, routers retain usage records during disconnections and transmit them without duplication afterward, preventing gaps in cost accounting.

The system also prevents uncontrolled spending by automatically switching to lower-cost models when budgets are exhausted and blocking only the affected API keys when limits are reached, while issuing alerts. Because Continuum Router enforces these controls directly in the request path, enforcement is immediate and remains operational even during central server outages.

Lablup is targeting both enterprises and service providers, including MSPs and cloud service providers (CSPs), with Continuum Hub and Continuum Router. For enterprise customers, the company emphasizes ease of deployment in on-premises environments and data protection.

How Lablup's Continuum Hub and Continuum Router operate. (Source: Lablup)

Continuum Hub does not contain fields for storing prompt or response content. Instead, it collects only the numerical metadata required for cost management, making the system suitable for organizations sensitive to data transfers, including financial institutions and public-sector entities.

Continuum Router can also start with a memory allocation of just 64 MiB, facilitating deployment in on-premises, air-gapped environments. Its architecture supports rate limiting and concurrency control without requiring an external database.

For enterprise customers, Lablup particularly emphasizes the ability to quantify savings from switching from external API models to internally fine-tuned, open-weight models, providing measurable evidence of return on investment (ROI). "Models served directly through Lablup's Backend.AI infrastructure platform and external commercial APIs can be compared in the same cost ledger, providing a basis for determining which workloads would cost less if moved to in-house GPUs," a Lablup official said.

Lablup said its solutions are already installed and operating at a financial-sector customer and that it is conducting proof-of-concept (PoC) projects with several companies, including financial institutions.

MegazoneCloud provides AI governance and cost management through its AIR Studio AI operating system and solutions from US-based Portal26. AIR Studio was upgraded to v3 earlier this month, while MegazoneCloud also signed a domestic reseller agreement for Portal26's solutions this month.

Both solutions provide AI governance capabilities that make enterprise AI usage visible and manage costs, security, permissions and policies. However, they differ in the scope of governance and their operating frameworks.

According to MegazoneCloud, AIR Studio v3 focuses on enterprise AI governance covering AI assets and operations across an organization. Centered on an AI gateway, it integrates management of model and agent usage and costs, permissions and budgets by user, department and project, API access policies, and audit logs.

Companies can also adopt governance capabilities independently while retaining their existing AI development and operating environments. This allows them to establish AI control frameworks without migrating to a separate platform and subsequently expand into AI agent operations and management.

Portal26, by contrast, focuses on monitoring the use of generative AI services within organizations and managing related policies. AIR Studio v3 provides integrated governance and scalability across enterprise AI operational assets, including models, agents and APIs, while Portal26 emphasizes visibility into employees' generative AI usage and the management of usage policies, security and costs.

Integrated dashboard of the AIR Studio v3 Governance module. (Source: MegazoneCloud)

MegazoneCloud highlights selective module adoption, its proprietary AI gateway and granular control policies as key features of its AI cost management solutions.

Customers can selectively deploy the Data, Agent, Ops and Governance modules of AIR Studio v3 according to their needs. The modular architecture allows organizations to implement permission management, token limits, cost controls and audit logs without changing their existing AI development and operating environments.

According to MegazoneCloud, companies seeking to prepare existing data for AI applications can begin with the Data module, while those requiring business-specific AI agents can first adopt the Agent module. Organizations already using multiple agents can establish integrated operations and monitoring through the Ops module or manage AI-related security and policies through the Governance module.

AI gateway capabilities first appeared in AIR Studio v2, announced in March. The service resembles offerings from dedicated AI gateway providers by centrally managing AI requests, monitoring usage and costs, and providing personal data protection.

In AIR Studio v3, unveiled earlier this month, these capabilities were expanded to include virtual keys that prevent exposure of original authentication keys, audit log tracking, blocking of API calls that bypass the gateway, and token limits and budget controls by organization and user. MegazoneCloud said these features help reduce security blind spots that could grow as connections to external Model Context Protocol (MCP) services and tools increase.

Another feature of MegazoneCloud's AI FinOps offering is the integration of token optimization functions, including model routing, prompt caching and batch inference, to deliver actual cost savings. The company said combining these capabilities with its TokenOps consulting and engineering services allows customers to continuously optimize costs as AI usage grows, going beyond basic cost visibility and control.

"Companies using multiple clouds and models, including AWS, Microsoft Azure and Google Cloud, can see where and why costs arise on a single screen," MegazoneCloud said. "Our consulting team can diagnose the causes and directly improve prompt, retrieval, caching and model selection structures."

Hyun-Seon, Park hspark@thelec.kr

My comment