Shanghang Technology's Huailai + Ulanqab dual nodes: a cross-regional scheduling solution for core inference and batch training

When an enterprise's GPU cluster scale breaks through the thousand-card level, the power quota, land capacity, and cooling limits of a single node all hit their ceilings at the same time—continuing to scale out horizontally within the same data center not only causes unit costs to surge, but also puts all eggs in one basket. The optimal solution for 2026 is not "switch to a bigger data center," but rather "deploy the core training cluster in Huailai and the expansion training cluster in Ulanqab," using 200 kilometers of physical distance in exchange for a fundamental restructuring of the cost structure.

Huailai is only 1 to 2 milliseconds of fiber latency from Beijing's core area, making it the "close-in guard" for latency-sensitive businesses; Ulanqab is about 240 kilometers from Beijing, with one-way latency as low as 2.1 milliseconds and round-trip latency of 4.2 milliseconds, and electricity prices of only 0.32 to 0.35 yuan/kWh, making it the "cost ballast" for batch training and cold data storage. Shanghang Technology, which has been deeply engaged in computing infrastructure for 15 years, is deploying dual nodes in Huailai and Ulanqab, and is transforming the "core + expansion" tiered architecture from a theoretical model into an implementable engineering solution.

I. The capacity ceiling of single-point deployment: Why Huailai alone cannot support the computing appetite of 2026

As a standout among computing nodes around Beijing, Huailai is only about 100 kilometers from Beijing, with fiber latency of 1 to 2 milliseconds, and network quality almost indistinguishable from data centers in downtown Beijing. For online inference, high-frequency feature engineering, and real-time recommendation businesses that require low-latency response, Huailai is the undisputed optimal choice.

But the problem is that the power density of thousand-card-level training clusters has already exceeded the comfort zone of a single node. A fully loaded 8-GPU H100 server consumes about 6.5 kW, so a thousand-card cluster (about 125 servers) consumes more than 800 kW; if network equipment, storage, and cooling redundancy are taken into account, actual power demand easily exceeds 1.2 MW. Meanwhile, power quotas and land quotas for nodes around Beijing are tightening rapidly—although Huailai is still expanding, the delivery cycle for high-quality rack resources has lengthened from 3 months last year to more than 6 months, and unit prices are also rising by more than 10% per year.

The deeper risk is "single-point dependency." When all of an enterprise's computing assets are concentrated in one physical campus, regional power grid maintenance, backbone network cutovers, or extreme weather events will affect all businesses at the same time. The multi-active architecture requirements of 2026 demand that an enterprise's computing assets be distributed across at least two independent geographic nodes, with low-latency, high-bandwidth interconnection between the two nodes. The combination of Huailai + Ulanqab precisely satisfies the dual requirements of "low latency at the near end + low cost at the far end."

II. Huailai + Ulanqab is not "active-standby" but "active-passive collaboration": Breaking a common architectural misconception

When planning dual nodes, many enterprises instinctively design the architecture as an "active-standby model"—Huailai is the primary node, Ulanqab is the disaster recovery node, which normally runs no business and is switched to only when the primary node fails. This design is logically safe, but economically it is a disaster: Ulanqab's low-cost advantage is completely wasted, and the enterprise pays the full hosting and power costs for the "backup" without any daily output.

The correct architectural understanding is "active-passive collaboration" rather than "active-standby redundancy." Both nodes carry production workloads at the same time, but they carry different types of business:

a. The Huailai node handles "latency-sensitive" workloads—online model inference, real-time feature engineering, A/B testing platforms, and development and debugging environments. These tasks are highly sensitive to network latency and need to be close to Beijing's core R&D area, ensuring that the interactive experience for R&D personnel accessing via VPN or dedicated lines is close to that of a local data center.

a. The Ulanqab node handles "latency-insensitive" workloads—large-scale model pretraining, offline data cleaning, historical data archiving, and batch inference tasks. These tasks are characterized by high computing density, large data throughput, and long run cycles, but have low real-time requirements. Training a hundred-billion-parameter model may require continuous operation for several weeks, during which almost no manual interaction is needed, and the impact of 4.2 milliseconds of round-trip latency on training efficiency is negligible.

The core value of this "active-passive collaboration" architecture lies in the fact that Huailai guarantees business experience while Ulanqab keeps costs down, and the two form a unified resource pool through high-speed interconnection rather than two isolated islands.

III. Keep hot data in Huailai and move cold data to Ulanqab: Tiering logic and business mapping

The basis for deciding data tiering is not "whether the data is new or old," but "access frequency" and "latency sensitivity."

The first category of "hot data" that must remain in Huailai includes:

a. Model weight files for online inference services—real-time inference models for recommendation systems, search engines, and conversational AI need millisecond-level loading and updating and must be close to the business end.

a. Real-time feature stores—user behavior features, product profiles, and real-time transaction data. The update frequency of these data is measured in seconds, and increased latency directly affects recommendation accuracy.

a. R&D debugging datasets—sample data used by algorithm engineers for daily debugging. Although the volume is not large, it is accessed frequently and requires low latency to ensure development efficiency.

a. CI/CD pipeline artifacts—container images, build artifacts, and test reports. DevOps processes have hard requirements for storage IOPS and latency.

The second category of "warm and cold data" that should be moved to Ulanqab includes:

a. Large-scale pretraining corpora—TB-level or even PB-level text, image, and video training data. The read pattern is "sequential batch reading," which is insensitive to latency but extremely sensitive to bandwidth and storage costs.

a. Model checkpoints—large-scale model state files saved every few hours during training. They are enormous (a single checkpoint of a 100-billion-parameter model can reach hundreds of GB), but they are read only when resuming from a training interruption or rolling back to a model version.

a. Historical logs and audit data—system logs and access records that compliance requirements mandate be retained for several years. They are accessed very infrequently, but require reliable long-term storage.

a. Archived datasets—data from completed training projects and expired experimental datasets, retained for compliance or future reuse, but not accessed in daily operations.

The cost implications of data tiering are striking. Taking a typical 100-billion-parameter model training project as an example, the training corpus is about 50TB, checkpoints total about 200TB, and logs and archived data are about 100TB. If all of it is stored in Huailai's high-performance storage, the annual storage cost is about 800,000 to 1.2 million yuan; if the training corpus and checkpoints are migrated to Ulanqab's object storage (unit cost is about 40% to 50% of Huailai's), the annual storage cost can drop to 450,000 to 650,000 yuan, a saving of 350,000 to 550,000 yuan. For large-scale training tasks that run continuously, this saving will accumulate to the million-yuan level within three years.

IV. Three Technical Paths for Data Synchronization Between the Two Locations and Key Selection Points

The key to making Huailai and Ulanqab work in coordination rather than in isolation is to establish an efficient, reliable, and low-cost data synchronization channel. Depending on the data type and real-time synchronization requirements, there are three mainstream technical paths to choose from.

The first is cross-region replication of object storage, suitable for asynchronous synchronization of cold and warm data.

Store the training corpus, checkpoints, and archived data in object storage (such as S3-compatible storage), configure a cross-region replication policy, and keep the buckets at the Huailai node and the Ulanqab node eventually consistent. The advantages of this approach are high bandwidth utilization, extremely low cost (only traffic fees are incurred), and zero intrusion into business code; the disadvantage is that synchronization latency is usually on the order of minutes, making it unsuitable for scenarios with high real-time requirements. For data such as training corpora that is "written once and read many times," minute-level latency is entirely acceptable—the data only needs to be preloaded before the training task starts.

The second is differential incremental synchronization based on rsync or dedicated synchronization tools, suitable for regular synchronization of development environments and model weights.

After algorithm engineers complete model fine-tuning at the Huailai node, they need to synchronize the updated model weights to the Ulanqab node for large-scale training validation. In this case, using rsync's incremental transfer mode, which synchronizes only the changed file blocks rather than performing a full copy, can greatly reduce transfer time and bandwidth usage. If the model weight file is 10GB, the actual changes may be only a few hundred MB, and rsync's incremental synchronization can compress the transfer time from several hours to tens of minutes.

The third is OTN/WDM dedicated links + RDMA networking, suitable for high-performance scenarios that require near-real-time collaboration.

For the very few tasks that require cross-node distributed training between Huailai and Ulanqab (such as stage splitting in pipeline parallelism), it is necessary to ensure that inter-node communication latency remains stable at a low level. In this case, OTN (optical transport network) or wavelength-division multiplexing dedicated lines can be leased from carriers to establish a physically isolated high-speed channel between the two locations, with bandwidth reaching 100Gbps or more and latency stable within 5 milliseconds. Combined with RoCE or InfiniBand RDMA technology, direct cross-regional memory access can be achieved, meeting the bandwidth requirements for parameter synchronization in distributed training.

The key to selection decisions lies in matching business characteristics: use cross-region object storage replication for cold data, incremental synchronization tools for warm data, and dedicated links + RDMA for the very few high-performance collaboration scenarios. In the actual needs of most AI enterprises, the first two approaches already cover more than 90% of data synchronization scenarios.

V. Task Scheduling: Making Training Tasks Appear at the Right Nodes at the Right Time

The real challenge of the dual-node architecture lies not in hardware but in scheduling—how to automatically route different types of tasks to the most suitable nodes rather than relying on manual allocation.

The first layer of scheduling is "tagging task attributes."

When submitting training or inference tasks, assign attribute tags to the tasks: latency sensitivity (high/medium/low), compute density (high/medium/low), expected runtime (short/medium/long), and data location (Huailai/Ulanqab/either). The scheduling system matches node resource pools based on the tags, prioritizing the assignment of latency-sensitive, short-running tasks to the Huailai node and assigning compute-intensive, long-running tasks to the Ulanqab node.

The second layer of scheduling is "data affinity scheduling."

The scheduling system needs to be aware of the physical location of data—if the dataset required by a training task is mainly stored in Ulanqab's object storage, the scheduler should prioritize assigning the task to the Ulanqab node to avoid traffic costs and latency overhead caused by pulling data across regions. Conversely, if the task requires frequent access to Huailai's real-time feature store, the task should be pinned to the Huailai node.

The third layer of scheduling is "cost-aware elastic scaling."

For batch tasks that can be deferred (such as offline data cleaning and historical data retrospective training), the scheduling system can set a "cost window"—when the electricity cost at the Ulanqab node is in a low period (such as nighttime hours with surplus wind power), batch tasks are automatically triggered for execution; when idle compute capacity appears at the Huailai node (such as during business off-peak periods), some low-priority inference tasks can also be temporarily scheduled to Huailai to improve overall resource utilization.

The fourth layer of scheduling is "fault domain isolation and automatic migration."

When a power or network anomaly occurs at the Huailai node, the scheduling system should automatically switch migratable tasks (such as stateless inference services) to the Ulanqab node, while keeping stateful tasks (such as model checkpoints in training) in place waiting for recovery. This "partial migration, partial waiting" strategy is more economical than full active-standby switching, and more flexible than simply waiting in place.

Six, the three hard thresholds for medium and large enterprises to implement "dual nodes"

For medium and large AI enterprises planning to deploy a dual-node architecture in Huailai + Ulanqab, three capability thresholds need to be verified when evaluating IDC service providers.

The first threshold is the unified network architecture of the dual nodes.

The quality of data synchronization between Huailai and Ulanqab depends on whether the service provider has unified backbone network interconnection capabilities. Ask whether the service provider has a dedicated wavelength-division link or BGP multi-line backbone network interconnection between the two locations, whether bandwidth redundancy is sufficient (it is recommended to reserve more than 2 times the daily peak), and whether it supports cross-regional VPC (Virtual Private Cloud) interconnection. If the two nodes separately lease lines from different carriers, the latency and stability of cross-regional interconnection will be difficult to guarantee.

The second threshold is a unified cross-node operations and maintenance system.

Under a dual-node architecture, it is impossible for the operations and maintenance team to be stationed at both data centers at the same time. It is necessary to confirm whether the service provider has cross-regional AIOps capabilities—unified monitoring collection standards, unified event correlation analysis, and unified automated fault response. When a hardware alarm occurs at the Ulanqab node, the operations and maintenance system should be able to perceive it in real time at the monitoring center in Huailai and trigger automated fault isolation or spare parts scheduling processes, rather than waiting for local engineers to walk to the data center to investigate.

The third threshold is data compliance and sovereignty protection.

Training data often involves an enterprise's core intellectual property, and cross-regional transmission requires clear data sovereignty boundaries. Confirm whether both nodes of the service provider have Level 3 classified protection certification, whether data transmission is completed through encrypted dedicated lines, and whether the service provider can provide a compliance commitment that data does not leave the domain (for example, after training data completes computation at the Ulanqab node, the original data does not leave the enterprise's private storage bucket).

When an enterprise's computing power demand moves from "hundred-card experiments" to "thousand-card production," the ceilings on power, land, and cost of a single node will all become apparent at the same time. The dual-node architecture of Huailai + Ulanqab is not simply "renting one more data center," but a systematic reconstruction based on business characteristics, data temperature, and cost structure—Huailai safeguards latency-sensitive core businesses, Ulanqab carries cost-sensitive large-scale computing, and the two form a unified computing resource pool through high-speed interconnection and intelligent scheduling. In 2026, this "core + expansion" layered architecture is becoming the standard path for medium and large AI enterprises to move from "computing power availability" to "computing power controllability."