A three-day restoration effort brought a long-dormant Huawei SANEX9510 signaling-traffic analysis platform back online without restarting servers or the Sybase IQ database. The system, running version V200R003C10SPC110, had been out of service for ten months after certificates expired in October 2025. What appeared to be a single certificate problem turned out to be a cascade of seven distinct failures spanning four independent TLS-protected circuits.
System Architecture and Affected Components
The platform consists of two main servers (cnoss1 at 10.0.0.14 and cnoss2 at 10.0.0.15) plus four DSS collectors. Data flows across four separate encrypted paths: WAS–DAS over Thrift (port 29301), DAS–DSS over FTPS, WAS–DSS over SGDSS (port 6688), and Store–Sybase IQ (port 2988). Each circuit maintains its own keystore and truststore, making diagnosis especially difficult when generic errors such as “Connection Interrupted” or “PKIX path building failed” appear.
Layer-by-Layer Recovery
Layer 1 – DAS identity: The first expired certificate was replaced with a new serverkeystore_das.p12 containing a leaf valid until 2030. DAS started successfully, creating a false sense that the remaining issues would be equally straightforward.
Layer 2 – WAS zero-byte keystore: Investigation revealed that the WAS identity PKCS#12 file was exactly 0 bytes. The file was rebuilt from server_final.pem and SI.cer, confirming a valid PrivateKeyEntry for CN=mae.huawei.com.
Layer 3 – Missing ‘@’ in encrypted password: The TrustStorePass value in ThriftClient.properties had lost its leading “@” during a PTY copy operation. Restoring the character allowed the encrypted secret to decrypt correctly and the truststore to open.
Layer 4 – Separate FTPS keystore: DAS uses serverkeystore_das.p12 for Thrift but serverkeystore.p12 for FTPS, configured in ftpsCer.properties. A dependency map of all six property files was created before the FTPS identity was updated, preventing accidental breakage of the newly restored WAS–DAS link.
Layer 5 – DSS private-key format incompatibility: New certificates had been loaded on the server side months earlier, yet DSS collectors still referenced the old files. After loading Certificate_CA.pem, Certificate_Pub.cer and Certificate_Priv.pem via RMV CERT / LOD CERT, the FTP service refused to start. The private key had been generated with modern PBKDF2 parameters (HMAC-SHA512, 1676 iterations, AES-256-CBC) that the product’s legacy OpenSSL (0.9.8 / 1.0.x) could not handle. The key was re-wrapped on a current Ubuntu system using matching PKCS#8 parameters, after which “FTP service started” was logged.
Layer 6 – WAS–DSS circuit on port 6688: Even after FTPS worked, the GUI still showed “Connection Interrupted Between WAS and Device”. The SgdssSocketSSL.properties file pointed to a separate keystore. Manual OpenSSL and Java TLS tests confirmed successful mTLS negotiation; the remaining issue was stale application sessions that recovered after the network links were briefly reset.
Layer 7 – Store connection leak to Sybase IQ: Data files arrived but were not written to the database. Instead of altering database passwords, technicians examined connection counts. Store had accumulated 204 connections against a -gm 300 limit. Restarting only the Store process reduced connections to 113, restored authentication, and allowed procedure calls for MAP, ISUP, SCCP and TCAP to succeed.
Additional Finding
An unrelated alarm “Failed to import DAS data to database” (Alarm ID 20) had been present since 30 July 2026 because the PB_DAS_DIRSIZE table was missing. The table was created manually from the installer DDL without running any upgrade scripts, after which the import cycle completed successfully.
Actions Deliberately Avoided
- No server reboots of cnoss1 or cnoss2
- No Sybase IQ restart
- No password resets or ALTER USER statements
- No execution of replace_encryption.sh
- No crypto work-key changes
The restoration demonstrates that deep dependency mapping and careful verification of every TLS circuit can revive complex legacy systems even when official support channels have been unable to do so for nearly a year.
