VMware Cloud on AWS and VMware Cloud on PKS: Availability Issue
Incident Report for VMware Cloud Services
Resolved
VMware Cloud PKS and VMware Cloud on AWS availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1 stands resolved and fully recovered. Engineering Team has restored propagation for Route 53 Private DNS changes. This is the final notification for this incident.

Start Time: May 10, 2019, 8:50 PM UTC
End Time: May 11, 2019, 9:46 AM UTC

Impact: None
Posted 16 days ago. May 11, 2019 - 10:27 UTC
Update
VMware Cloud PKS and VMware Cloud on AWS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team has restored propagation for Route 53 Private DNS changes. Teams continue to work towards full recovery. Queries to existing DNS records are not affected by this issue. Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 09:26 UTC
Update
VMware Cloud PKS and VMware Cloud on AWS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team continues to make progress towards restoring propagation for Private DNS. A majority of hosts have processed all updates, and the team is working to accelerate recovery of the remaining hosts. Queries to existing DNS records are not affected by this issue. Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A


Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 07:04 UTC
Update
VMware Cloud PKS and VMware Cloud on AWS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team processed the majority of the backlog of queued changes for Public DNS. Private DNS changes are still experiencing propagation delays. We continue to work towards full recovery. Queries to existing DNS records are not affected by this issue. Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 05:22 UTC
Update
VMware Cloud PKS and VMware Cloud on AWS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team processed the majority of the backlog of queued changes for Public DNS. Private DNS changes are still experiencing propagation delays. We continue to work towards full recovery. Queries to existing DNS records are not affected by this issue. Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 03:55 UTC
Update
VMware Cloud PKS and VMware Cloud on AWS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team is observing steady recovery on the propagation times of DNS edits to public and private hosted zones, with the oldest changes propagating first. Teams continuing to work through the backlog of queued changes. Queries to existing DNS records are not affected by this issue.Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 02:48 UTC
Update
VMware Cloud PKS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering Team is working on the problem with the increased propagation times of DNS edits to public and private hosted zones, and increased failures for Delete Hosted Zone, Associate Private Hosted Zone and Disassociate Private Hosted Zone API and Console operations. Queries to existing DNS records are not affected by this issue. Next update in 1 hour.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters. Queries to existing DNS records are not affected by this issue.
Posted 16 days ago. May 11, 2019 - 01:13 UTC
Update
VMware Cloud PKS continues to experience availability issues in US-WEST-2, EU-WEST-1, and US-EAST-1. Engineering teams have identified a possible root cause and are progressing toward a solution to restore normal service.

Start Time: May 10, 2019 8:50 PM UTC
End Time: N/A

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters.
Posted 16 days ago. May 10, 2019 - 23:42 UTC
Identified
We are continuing to investigate the situation and will provide more updates as we have more information.

Impact: Cluster creation and deletion operations may fail. Customers can continue to use their existing clusters.
Posted 16 days ago. May 10, 2019 - 22:47 UTC
Investigating
VMware Cloud PKS is experiencing an availability issue in US-WEST-2, EU-WEST-1, US-EAST1. Engineering teams are engaged to resolve the issue and restore service.
Posted 16 days ago. May 10, 2019 - 22:19 UTC
This incident affected: VMware Cloud on AWS (VMware Cloud on AWS) and VMware Cloud PKS.