Wherobots - Notice history

This incident has been resolved. The identity-service degradation at our cloud provider ended at approximately 3:20 PM PDT (22:20 UTC), and API error rates, latency, and workload launches have been normal for over an hour since. Any SQL sessions, notebooks, or jobs that failed to start between 2:20 and 3:20 PM PDT can be retried normally. We apologize for the disruption and will follow up with the changes we're making to reduce our sensitivity to this class of provider issue.

Our cloud provider's identity service has recovered. API error rates, latency, and workload launches have been normal since approximately 3:20 PM PDT (22:20 UTC), and all API instances are healthy. We are continuing to monitor before marking this incident resolved. Any workload launch that failed during the incident can be retried now.

We have confirmed the issue fully mitigated.

A third-party service provider experienced an outage that intermittently prevented some Wherobots services from retrieving their live configuration. During this period, customers may have experienced:

1. Sessions, notebooks, and job runs in the AWS US East (Ohio / us-east-2) region failing to start, or the region appearing unavailable.

2. Slower startup for sessions, notebooks, and job runs.

Impact was intermittent and did not affect already running workloads. Service has been fully restored, and we have removed the affected startup paths' dependency on the third-party provider and added monitoring to detect this condition in the future.

All services have been fully restored.

We implemented a fix. The us-east-2 compute region is stably back to operational now. The workload launch time is recovering, we are closely monitoring.

We have moved to the fallback registry and the image pulling failures are mitigated.