Wherobots - Intermittent failures starting compute due to a third-party outage – Incident details

All systems operational

Show current status

Intermittent failures starting compute due to a third-party outage

ResolvedPartial outage 20%3 hours 38 minutes

Affected

Wherobots Notebooks

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

Wherobots Jobs

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

Wherobots SQL API

Partial outage from 6:30 PM to 7:53 PM, Operational from 7:53 PM to 10:08 PM

  • PostmortemUTC

    A third-party service provider experienced an outage that intermittently prevented some Wherobots services from retrieving their live configuration. During this period, customers may have experienced:

    1. Sessions, notebooks, and job runs in the AWS US East (Ohio / us-east-2) region failing to start, or the region appearing unavailable.

    2. Slower startup for sessions, notebooks, and job runs.

    Impact was intermittent and did not affect already running workloads. Service has been fully restored, and we have removed the affected startup paths' dependency on the third-party provider and added monitoring to detect this condition in the future.

  • ResolvedUTC

    All services have been fully restored.

  • MonitoringUTC

    We implemented a fix. The us-east-2 compute region is stably back to operational now. The workload launch time is recovering, we are closely monitoring.

  • IdentifiedUTC

    We're experiencing service outage in our us-east-2 region and service degradations in all region. We have identified the root cause and are working on a fix.