AI Middle Platform: Deployment and Validation of a Neutral Integration Layer in the Production Environment

Opening: Purpose of the Work

Thank you to the community for recommending this for the homepage. The purpose of deploying and verifying the neutral integration layer in the production environment is to bring the six functional component configurations—connection, extraction, translation, write-back, control, and ODS—that have been verified in the test environment online in the production environment, confirm that they are usable under real data volumes, that write-back is correct, and that alerting and auditing are in place, after which they will be handed over to IT for transition to daily operations.

This requires confirming: that the configurations remain usable after the production addresses, accounts, and parameters are replaced, and that extraction does not affect the operation of business systems; that write-back is correct on real business data, and that duplicate delivery, bidirectional conflicts, and partial failures are all within control; and that alerts actually trigger and reach the recipients, and that audit trails and log desensitization are compliant.

I. Preparation of the Deployment Plan

The data architect is responsible for preparing the "Deployment and Verification Plan for the Neutral Integration Layer in the Production Environment" and submitting it to the IT head for approval. The content includes:

(1) Scope and discrepancy rules

1. Go-live scope. The plan shall state the go-live scope, the field scope, and the systems not included.

2. Comparison basis. The plan shall state the comparison basis against the finalized version of the test environment, listing the comparison items one by one: addresses, accounts, credentials, timeouts and concurrency, capacity, and read scope.

3. Discrepancy classification and handling. Addresses, accounts, credentials, and capacity are environmental differences and shall be registered for reference; timeouts and concurrency, and read scope shall be executed according to the finalized version, and any inconsistency with the finalized version shall be explained, with configuration defects required to be cleared to zero. Missing mappings, version inconsistencies, and uncovered access scope are configuration defects and must be cleared to zero before go-live; those that cannot be cleared shall be recorded as outstanding items with the impact and completion time noted.

(2) Window and approval path

1. The deployment and cutover window shall avoid month-end closing, quarter-end closing, and database backups;

2. The plan shall state the window start and end, the scope of business notification, and the affected positions.

3. Approval is two-tiered according to risk: read-only go-live and routine configuration changes are approved under routine change approval, by the IT head;

4. Enabling write-back, trial write-back, and write-back to financial fields are submitted to the next higher level of approval, by the leader in charge, and write-back to financial fields additionally requires countersignature by the business and finance departments. Suspension and rollback are decided on site by the approver, with the approval form completed afterward. Approval form numbers shall follow the enterprise's existing change order numbering rules.

(3) Gray release and suspension conditions

1. Deployment window. Deployment and cutover shall avoid month-end closing, quarter-end closing, and database backup periods. The plan shall state the window start and end times, the scope of business notification, and the affected positions.

2. Approval authority. Approval is divided into two tiers according to risk level: read-only go-live and other routine configuration changes are approved under the enterprise's routine change process, by the IT head; enabling write-back, trial write-back, and write-back to financial fields are submitted to the next higher level of approval, by the leader in charge. Among these, write-back to financial fields additionally requires countersignature by the business department and the finance department.

3. Suspension and rollback. Suspension and rollback are decided on site by the leader in charge or an authorized person, with approval formalities completed afterward.

4. Approval documents. Approval form numbers shall follow the enterprise's existing change order numbering rules, without separate numbering.

(4) Trial write-back data rules

1. Identification rules. Trial write-back in the production environment writes directly to production data; before trial write-back, data identification rules shall be determined: use dedicated test identifiers or test documents to ensure that fields are identifiable and documents can be voided.

2. Cleanup and reversal. The plan shall state the method of cleaning up or reversing data after trial write-back, as well as the executor and the confirmer.

3. Prohibitions. Real customers, real orders, or real financial fields shall not be used as trial write-back data.

4. Impact confirmation. Even after test documents are voided, they will still enter business statistics, reconciliation, and audit trails. Their impact on the enterprise's reports and reconciliation shall be assessed and confirmed by the business department in the plan; only after it is confirmed acceptable may this method be used for trial write-back.

(5) Monitoring requirements

1. The plan shall state the four elements: monitoring item categories, thresholds, response time limits, and alert content.

2. Alert content shall include four items: system name, batch number or data scope, error code, and impact scope.

3. Alert recipients shall be designated separately by alert level; P1 alerts shall have one primary and one backup recipient each, and P1 alerts in this business domain must ensure that both the primary and backup recipients can be reached.

4. The plan shall specify two requirements: check before go-live whether the monitoring configuration is in place, and verify after go-live whether alerts can actually be triggered.

(6) Rollback target and decision rules

The plan shall state which configuration version and which data state to roll back to, who decides, how long it takes to complete, and whom to notify. It shall state that a drill must be conducted once before go-live, as well as the record requirements for the drill.

II. Deployment Preparation

The following eight items shall be executed one by one according to the plan, and each item shall proceed to the next only after verification passes; those that do not pass shall stop at that item.

(1) Version alignment.

Check five types of versions: the respective versions of the six types of configurations (six), the deployment plan version, the "Production Environment Parameter Table" version, the test environment verification record version, and the data extraction and desensitization script version. The six configuration versions shall be registered in separate columns in the parameter table. In case of inconsistency, first identify the cause and then align; do not go live with inconsistent versions. The data architect checks, and IT registers.

(2) Preparation and confirmation of the discrepancy list.

Compare system by system and configuration item by configuration item according to the comparison basis set in the plan, and compile a discrepancy list between production and the finalized version of the test environment, divided into two columns: environmental differences and configuration defects, with handling executed according to the rules set in the plan. The list is prepared by the data architect, IT provides the system and interface list, and the business department confirms the field scope. The list shall be submitted to the approver along with the go-live approval.

(3) Account application and authorization.

Production accounts shall be applied for independently and shall not reuse accounts from the test environment. They are divided into four categories by purpose: a production database read-only account, granted only the read permission on the database tables required for extraction; a separate write-back write account, granted only the write interface permission of the target system, authorized by table and by field; an agent ODS read-only account, with read-only access to wide tables and views; and a database creation management account, used only for creating the ODS database, creating tables and configuring permissions, recovered immediately upon completion, and not used for daily extraction or operations and maintenance. Credentials shall be stored in key management, and configuration files shall contain only references. Holders, rotation cycles and recovery records shall be registered in the Credential Ledger. IT applies for, authorizes and configures them, and the data architect reviews the permission scope.

(4) Production database setup and ODS layering.

Use the database creation management account to create the ODS database and layer it: the source-aligned layer, the integration mapping layer, the entity consolidation wide table, the monitoring table, and the read-only masked views. Tablespaces, partitions and indexes shall be set according to the actual data volume, converting the measured daily increment of a single system in the test environment into the production scale, with an additional 30% margin reserved. The character set and time zone shall be aligned with the source system. The backup strategy and retention period shall be determined according to compliance requirements and completed before the database is opened. After the database is created, use the read-only account to test the connection and confirm that each layer and view is readable.

(5) Monitoring configuration and P1 connection test.

Configure all monitoring items, thresholds and recipients specified in the plan: the four types of metrics—extraction tasks, queue backlog, write-back failure rate, and source database usage—shall be configured in place, and alarm content shall contain the four elements. The primary and backup recipients for P1 shall each be connected once, and the connection time and the person who answered shall be recorded. The configuration results and connection records shall be entered in the Pre-Go-Live Verification Record.

(6) Backup and recovery drill and rollback drill.

Perform a full backup once and an actual recovery once, and check the data volume and the row counts of key tables. Conduct a drill once according to the rollback target specified in the plan, rolling back to the specified configuration version and data state, and record the version number, recovery duration and decision-maker. Go-live is permitted only after both records are complete. IT executes, and the data architect confirms the version and data state.

(7) Implementation of test-write data identification.

According to the rules specified in the plan, set the identification on fields and documents, clarify the method of cleanup or reversal, the executor and the confirmer, and adopt it after confirmation by the business department. The identification rules and cleanup method shall be retained as an attachment to the plan, and the execution results shall be submitted with the Write-Back Verification Record.

(8) Pre-go-live verification.

Six items shall be actually executed one by one:

1. Use the production database read-only account to attempt insert, update, delete and DDL respectively; all four attempts should be rejected;

2. Spot-check the permissions of the write-back account and check for excess write permissions;

3. The direct connection between the agent and the business system has been removed, and the write-back interface returns a rejection to it;

The results shall be entered in the Pre-Go-Live Verification Record and signed off by the data architect and the internal security post of IT; if verification fails, go-live shall not proceed. This section is implemented by IT, verified and signed item by item by the data architect, and the business department confirms the field scope and the impact of test writes.

III. Deployment

Production go-live is divided into two waves: read-only go-live and write-back gray release. The next wave proceeds only after the previous wave is deployed and passes verification. The six types of configuration shall be rebuilt according to the finalized version of the test environment, replacing production addresses, production accounts and production parameters, and test configuration files shall not be copied and modified.

(1) First wave: read-only go-live

Configure the five types of configuration one by one and register them in the Production Environment Parameter Table. After each type is configured, verify it, and after it passes, configure the next type.

Connection. Connect using the production database read-only account. Timeouts and concurrency shall take the measured values from the test environment. Verify connectivity system by system and pull metadata; register connected systems in the System Access Ledger and include them in monitoring. Network policies and interface whitelists shall be opened by IT, and connectivity shall be verified immediately after opening; those that fail shall be recorded and the responsible party noted.

Extraction. Scheduling shall avoid business peak hours, month-end closing and backup windows. Concurrency and rate limiting shall take the measured values from testing. Configure resumable transfer and dead-letter recipients, and enable row-count discrepancy alarms. Scheduling tasks shall be included in monitoring, and recipients shall be notified when tasks fail.

Translation. Configure protocols, messages and code value mapping tables according to the finalized version. Newly added enumerations in source systems shall first be passed through and listed in the pending verification list, and shall not be counted in dashboard metrics. Verify mappings in the test environment before source system upgrades.

Control. Configure the permission matrix by position, and the correspondence between positions and roles shall be confirmed by the business department. Set primary and backup recipients for P1. Rule changes shall go through approval. Audit trails and log masking shall be configured at the same time. The agent shall be granted only the ODS read-only channel, and the write-back interface shall be blocked for it.

ODS integration mapping. The three types of integration mappings and wide tables shall be released by version. Capacity and refresh frequency shall be set according to the actual data volume. Abnormal data shall be written structurally into the ODS monitoring table and made visible in BI.

Closure. After the five types of configuration other than write-back are deployed, freeze the configuration and register the version, and close the write-back entry. After closure, complete read-only verification and monitoring security verification; after both pass and approval to enable write-back is obtained, proceed to the second wave. The data architect signs off at the freeze point, and IT registers it.

(2) Second wave: write-back gray release

Configuration. The write-back channel, idempotency key composition, anti-ping-pong for the three types of fields, per-target transaction boundaries and independent write account configuration shall be completed and registered in the Write-Back Channel Registration Table, with the entry not opened.

Opening. Open after the three prerequisites are met: read-only verification passes; monitoring and security verification pass; approval to enable write-back is obtained. Obtain approval for test writes before small-batch test writes, and after test writes pass, confirm by batch. After opening, according to the permission matrix, only non-financial fields shall be opened, and the first batch field list shall be consistent with the plan.

Batches. Expand batch by batch according to the batches listed in the plan. After each batch is written, check the receipts and the target database, and proceed to the next batch after confirmation by the business department. Financial fields shall be opened only after separate approval. In the event of batch failures, duplicate writes or business complaints, pause and roll back according to the plan.

This section is carried out by IT, and the data architect checks and registers each item one by one.

IV. Verification

Production verification consists of sample-based checks and verification under real conditions; it does not repeat the rule checks of the test environment in full. The three steps proceed in waves: first read-only; immediately after read-only go-live, verify monitoring and security; then verify write-back. For each step, write four items: basis, operations, criteria and records. The criteria and sampling scope take the values set in the plan, and the sample size shall be no fewer than 20 records.

(1) Read-only verification

Basis. The Production Environment Parameter Table, the System Access Ledger, the Interface Adaptation Registration Form, the Extraction Rule Registration, and the test environment verification records. The test environment conclusions serve here as expected values for comparing deviations.

Operations. Verify connectivity system by system and compare metadata; perform one full extraction and reconcile it against the source system's row count for the day; perform one extraction according to the incremental rules and check whether all changed data is captured; sample no fewer than 20 records for field-by-field comparison, with samples allocated by system and field, covering amounts, dates, statuses and association keys; check key enumerations item by item; record source database CPU and IO usage during peak hours; check whether the scheduling window overlaps with business peak hours and the backup window; interrupt once and check breakpoint resumption; create a batch of failures and check dead letters, notifications and reruns.

Criteria. Row count differences are within the threshold set in the plan; sampled records are consistent field by field; source database usage is within the approved upper limit, and the business system's use on that day is unaffected; tasks complete within the approved window; dead letters have a recipient and can be rerun from the failed batch.

Records. Results are recorded in the Read-only Verification Record, writing the measured values, number of sample records, comparison conclusions and peak source database usage item by item. Connectivity results are written back to the System Access Ledger, and extraction rule anomalies are written back to the Extraction Rule Registration.

(2) Monitoring and security verification

Basis. The four elements of alerts and response time limits set in the plan, the permission matrix, the six audit requirements, the masking rules, and the requirements for structured storage of exceptions and BI visibility.

Operations. Check the primary/backup connection records from the pre-check; inject one real failure and check alert triggering, delivery to primary and backup recipients, and response within the time limit; check the four elements of alert content; spot-check audit logs and check the six items—operating subject, time, object, type, result, request_id—and the retention period; check masking of sensitive fields in logs and alerts; call the write-back interface as an agent identity and check that it returns a rejection; create one anomalous data record and check its storage in the ODS monitoring table and its visibility in BI.

Criteria. Alerts actually trigger and are delivered to primary and backup recipients, with responses within the time limit; all six audit items are present and the retention period meets the standard; no plaintext appears in logs or alerts; the agent's call to the write-back interface is rejected; anomalous data is queryable in BI.

Records. Results are recorded in the Monitoring and Security Verification Record, with attached alert serial numbers or screenshots, the number of audit log samples and screenshots of the retention period configuration, and the connection records from the pre-check.

(3) Write-back verification

Basis. The One-way/Two-way Write-back Field List, the consistency handling list, the Write-back Channel Registration Form, and the test-write data identification rules and cleanup or reversal methods set in the plan.

Operations. Perform small-batch test writes of non-financial fields, with data bearing a dedicated test identifier; submit the same request repeatedly and check whether a second record is created; for two-way fields, construct simultaneous modifications on both sides and check whether values are taken according to the arbitration rules; create a situation where some targets succeed and some fail, and check whether status is recorded per target and whether only failed items are retried; check the write scope of each role according to the permission matrix; after test writes, clean up or reverse according to the plan and check the source system for residue; repeat the above checks as batches expand.

Criteria. Repeated submission does not create a second record; conflicts are arbitrated according to the rules; partial success does not resubmit already successful targets; there is no unauthorized writing; test-write data is cleaned up completely with no residue in the source system.

Records. Results are recorded in the Write-back Verification Record, with attached test-write data lists, number of records written, repeated submission results, conflict arbitration results, permission matrix verification results and cleanup or reversal results; test-write results and reconciliation differences are confirmed and signed by the business department.

All three steps' records are marked with the configuration version verified, corresponding one-to-one with the version number in the Production Environment Parameter Table. Records are signed by the data architect, IT and the business department supervisor. For items that fail, state which version to roll back to, who decides, and when to retry; whether to roll back is according to the plan. The business complaint channel and the trigger conditions for pausing write-back are communicated to the business department before verification begins.

This section is chaired by the data architect, executed by IT with IT participating in signing off on security verification, and the business department confirms the test-write results and reconciliation differences.

V. On-duty and handover

Observation period. It starts after write-back gray release has expanded to the agreed batches and alerts are running stably. The period from read-only go-live to write-back opening is not counted in the observation period; anomalies during that period are handled as alerts. Track four items: extraction success rate, queue backlog, write-back failure rate, and alert response time; thresholds take the values set in the plan. Summarize daily, review weekly, and handle issues by severity. If the observation period is not stable, extend the observation period or roll back, as decided by the approver. IT is on duty and issues daily reports, the data architect reviews the daily reports, and the business department provides feedback on issues.

Handover. Hand over after stability. Items to hand over: configuration version list, parameter table, monitoring list, rollback plan, system access ledger, channel debt ledger, credential ledger, interface adaptation registration form, alert recipient list, write-back channel registration form, account recovery records, audit read-only permissions. Hand over and sign off item by item. Temporary accounts used for deployment are recovered, and permissions are re-granted according to operations positions. After handover, IT maintains it long-term.

Archiving and changes. Configuration versions, verification records, rollback drills and observation period records are archived as the baseline for comparison in subsequent changes. Subsequent changes follow change management; do not modify configurations directly in production; emergency changes are documented retroactively.

This section is organized by the data architect for handover, received and signed by IT, and the business department participates in acceptance.

Conclusion

This section delivers: the deployment plan, the Production Environment Parameter Table, the discrepancy list, the Pre-launch Verification Record, the six types of configuration registrations, the read-only verification record, the monitoring and security verification record, the write-back verification record, the monitoring and alert list, the rollback plan and drill records, the go-live approval, the write-back opening approval, the test-write approval, the financial field approval, the various ledgers, and the handover and archiving records. The above records correspond one-to-one with the parameter table version numbers; check against this at delivery.

Conditions for transfer to daily operation: read-only verification passes; monitoring alerts actually trigger; write-back gray release verification passes; audit, masking and permission matrix verification pass; ledgers are complete; signatures are completed; the observation period is stable. If any item is not met, extend the observation period or roll back.