VMware Cloud on AWS Service Intermittent Availability Issue
Incident Report for VMware Cloud Services
Resolved
VMware Cloud on AWS team has successfully pushed the fix to the production environment and have completed validation of services. Scheduled SDDC updates have restarted.

Impact: None

Start Time: July 30, 2018, 07:25 AM UTC
End Time: July 31, 2018, 01:20 PM UTC
Posted Jul 31, 2018 - 14:29 UTC
Update
VMware Cloud on AWS team has successfully pushed the fix to the production environment. The team continues to validate services and full functionality in the production environment. Scheduled SDDC updates have restarted. We will provide updates as they become available.

Start Time: July 30, 2018, 07:25 AM UTC
End Time: N/A

Impact: During validation, we expect no impact to users although there may be instances where users are not able to perform some operations on VMware Cloud on AWS.

Scheduled SDDCs updates may be impacted but have restarted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 31, 2018 - 11:19 UTC
Update
VMware Cloud on AWS team has successfully pushed a fix to the production environment. The team continues to validate services in the production environment. We will provide updates as they become available.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud on AWS.
Scheduled SDDCs updates may be impacted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 31, 2018 - 10:39 UTC
Update
VMware Cloud on AWS team has successfully pushed fix to production environment. The team continue to validate services in the production environment. We will provided updates as they become available.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud HCX Services running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.

Scheduled SDDCs updates are impacted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 31, 2018 - 07:12 UTC
Update
VMware Cloud on AWS team has successfully pushed fix to production environment. They are monitoring the environment. We will provided updates as they become available.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud HCX Services running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.

Scheduled SDDCs updates are impacted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 31, 2018 - 01:26 UTC
Update
VMware Cloud team is performing emergency push on production environment. All tasks will be paused for 15 – 20 minutes. Customers will be able to trigger new tasks. The VMC UI may appear to be unresponsive (some operations will result in spinners) while the upgrade is in progress.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud HCX Services running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.

Scheduled SDDCs updates are impacted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 31, 2018 - 00:07 UTC
Update
VMware Cloud HCX Migration Services and Add/Remove Cluster/Host services continue to experience intermittent availability issues for a subset of customers. The VMware Cloud on AWS Engineering team is currently rolling out a fix to address the HCX migration issue and a fix is being developed for the Add/Remove Cluster/Host issue. Additional updates will be provided as they become available. We apologize for any inconvenience this issue may cause.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud HCX Services running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.

Scheduled SDDCs updates are impacted. If your SDDC update is impacted you will be notified directly with a revised schedule.
Posted Jul 30, 2018 - 21:03 UTC
Update
VMware Cloud HCX Migration Services and Add/Remove Cluster/Host services continue to experience intermittent availability issues for a subset of customers. Issues with the Compute and Management Gateways have been resolved.The VMware Cloud on AWS Engineering team is currently rolling out a fix to address the HCX migration issue and a fix is being developed for the Add/Remove Cluster/Host issue.
Additional updates will be provided as they become available. We apologize for any inconvenience this issue may cause.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud HCX Services running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.
Posted Jul 30, 2018 - 15:44 UTC
Update
VMware Cloud backend services continue to experience intermittent availability issues.Users will encounter issues if they attempt for HCX migrations or attempt to reconfigure their Compute and Management Gateways. VMware Engineering teams have identified root cause, runbooks have been created and impacted customers are being engaged to remediate the issues. If you are encountering issues then please engage VMware Support immediately via Chat quoting internal VMware Reference CSSD-26859.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.
Posted Jul 30, 2018 - 13:04 UTC
Identified
VMware Cloud backend services continue to experience intermittent availability issues.Users will encounter issues if they attempt for HCX migrations or attempt to reconfigure their Compute and Management Gateways. VMware Engineering teams have identified root cause and are developing runbooks to resolve these issues. If you are encountering issues then please engage VMware Support immediately via Chat quoting internal VMware Reference CSSD-26859.

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.
Posted Jul 30, 2018 - 11:32 UTC
Update
VMware Cloud backend services continue to experience intermittent availability issues. Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments. VMware Engineering teams are continuing the investigation on how to resolve the issue. If you feel you are experiencing issues then please engage VMware Support via Chat quoting internal VMware Reference CSSD-26859

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.
Posted Jul 30, 2018 - 10:28 UTC
Investigating
VMware Cloud backend services are experiencing Intermittent Availability issue. Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.VMware Engineering teams are actively engaged on resolving the issue.If you feel you are experiencing issues then please engage VMware Support via Chat quoting internal VMware Reference CSSD-26859

Start Time: July 30, 2018 07:25 AM UTC
End Time: N/A

Impact: Users may not be able to perform some operations on VMware Cloud on AWS or service (including HCX and NSX) running on VMware Cloud on AWS. This issue does not impact new SDDC deployments.
Posted Jul 30, 2018 - 08:55 UTC
This incident affected: VMware Cloud on AWS (VMware Cloud on AWS).