After a period of monitoring, we can confirm that all services were restored and have been stable. Enhanced Network Infrastructure subscriber traffic priority has returned to steady state out of DC1.
Posted May 13, 2025 - 06:00 CDT
Update
After an 8 hour monitoring period, we are deeming the CCM and IPsec VPN Service Operational as all production nodes are up and operational. Enhanced Network Infrastructure and data traffic is stable and operational, however it continues to run out of DC2 for the time being. We will continue to monitor and provide updates as available.
Posted May 08, 2025 - 21:57 CDT
Monitoring
Mitigation measures have been applied to restore Multi-Network SIM subscriber traffic.
We will continue to monitor for stability.
Posted May 08, 2025 - 13:25 CDT
Update
AT&T Enhanced and Verizon Enhanced has been rerouted successfully through the Secondary DC (Miami). Multi-Network SIM work is still underway between Support and Carrier engineering teams.
Posted May 08, 2025 - 11:18 CDT
Update
Fiber has been repaired and services are being failed back over, however subscriber traffic is exhibiting communication issues. Engineering team is fully engaged on a fix.
Posted May 08, 2025 - 10:12 CDT
Update
Support and engineering teams continue to work on a new resilient link in an effort to mitigate network communication issues.
The CCM platform is available for customers use.
Single sited Austin IPsec VPNs are still impacted.
Multi-Network SIM traffic is still being served from our Miami DC. Any customers who are still experiencing issues with MN SIM connectivity and do not have an IPsec VPN to Austin should attempt power cycles/reboots if possible.
The original issue with the DC provider is still ongoing and the fiber provider is attempting to restore service but no expected time to resolution is known.
We will provide further updates as they become available.
At this time some customers located in Austin Texas area may be experiencing a loss of network connectivity. The Cogent NOC is investigating. There is no ETR at this time. The ticket number for this outage is HD304990015.
Posted May 08, 2025 - 06:51 CDT
Update
Engineering teams have accelerated a planned upgrade and have failed traffic to a new resilient link.
This has restored service to the CCM platform.
Teams are working on the cross-site communications between Austin and Miami, so this may still be impacting Single sited IPsec VPNs as Multi-Network SIM traffic is currently in Miami.
The original issue with our DC provider is still ongoing but we are working on further mitigating the problem.
We apologize for any inconvenience, and we will provide further updates as they become available.
Posted May 08, 2025 - 04:33 CDT
Identified
Austin DC outbound internet is currently impacted. Support has engaged our DC provider who is currently investigating the issue on their end.
Multi-Network SIM traffic has failed over and is being served by our Miami DC.
Single sited Austin IPsec VPNs & the CCM platform will still be impacted.
We are continuing to work with our DC provider on resolving this issue.
Posted May 07, 2025 - 22:24 CDT
Update
The DC provider has dispatched a field engineer to site. The DC provider is continuing to investigate the issue.
We will provide further updates as soon as more information is available.
Posted May 07, 2025 - 22:14 CDT
Investigating
Support teams are currently investigating issues with the Austin DC which may have an impact on your service. We have identified from monitoring that we are seeing connection issues/outbound internet issue with the Austin DC. This will be impacting the CCM platform, single sited Austin IPsec VPN and Multi-Network SIM connectivity. VZW Enhanced and ATT Enhanced Connectivity has failed over through the secondary data center.
Update: Support teams are currently investigating the issues.
We will provide an update shortly and apologize for any inconvenience.