Wherobots - Intermittent failures starting compute due to a third-party outage – Incident details

All systems operational

Intermittent failures starting compute due to a third-party outage

Resolved
Partial outage 20 %
Started 20 days agoLasted about 4 hours

Affected

Wherobots Notebooks

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

Wherobots Jobs

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

Wherobots SQL API

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

Updates
  • Postmortem
    UTC
    Postmortem

    A third-party service provider experienced an outage that intermittently prevented some Wherobots services from retrieving their live configuration. During this period, customers may have experienced:

    1. Sessions, notebooks, and job runs in the AWS US East (Ohio / us-east-2) region failing to start, or the region appearing unavailable.

    2. Slower startup for sessions, notebooks, and job runs.

    Impact was intermittent and did not affect already running workloads. Service has been fully restored, and we have removed the affected startup paths' dependency on the third-party provider and added monitoring to detect this condition in the future.

  • Resolved
    UTC
    Resolved

    All services have been fully restored.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix. The us-east-2 compute region is stably back to operational now. The workload launch time is recovering, we are closely monitoring.

  • Identified
    UTC
    Identified

    We're experiencing service outage in our us-east-2 region and service degradations in all region. We have identified the root cause and are working on a fix.