EnderDevelopment - Ongoing Service Disruption – Incident details

Experiencing partial outage

Ongoing Service Disruption

Resolved
Partial outage
Started 4 months agoLasted about 7 hours

Affected

EnderDevelopment - AI Platform

Partial outage from 4:00 PM to 4:18 PM, Operational from 4:18 PM to 5:38 PM, Partial outage from 5:38 PM to 10:42 PM

Dashboard

Partial outage from 4:00 PM to 4:18 PM, Operational from 4:18 PM to 5:38 PM, Partial outage from 5:38 PM to 10:42 PM

EndAI - Main Backend

Partial outage from 4:00 PM to 4:18 PM, Operational from 4:18 PM to 5:38 PM, Partial outage from 5:38 PM to 10:42 PM

Updates
  • Resolved
    UTC
    Resolved

    The issue has been fully resolved.

    Root cause was identified as blocking synchronous operations on remote storage, combined with locking mechanisms that caused backend stalls under load.

    We have addressed these bottlenecks and implemented fixes to ensure smoother, non-blocking behavior.

    The system is now stable and operating normally.

    Thank you for your patience

  • Monitoring
    UTC
    Monitoring

    The issue now appears to be resolved again, and the platform is returning to normal operation.

    We are currently optimizing and fine-tuning the system to improve overall stability and prevent further occurrences.

    We will continue monitoring closely.

  • Identified
    UTC
    Identified

    The issue has unfortunately reoccurred and is again affecting the platform.

    Our team is actively investigating the root cause and working toward a stable resolution.

    We apologize for the continued inconvenience and will provide further updates shortly.

  • Monitoring
    UTC
    Monitoring

    We’ve deployed a potentially definitive fix to address the recurring backend issue.

    The system is now stable, and we are actively monitoring performance and behavior in real-time to ensure everything remains fully operational.

    While early signals are positive, we’ll keep a close watch to confirm long-term stability.

    Thank you again for your patience

  • Investigating
    UTC
    Investigating

    We are currently experiencing a recurring incident where the issue has reappeared, causing backend instability and temporary system blocks.

    This may result in failed requests, slow performance, or temporary unavailability of certain features.

    Our team is actively investigating the root cause and working on a permanent fix. Additional backend restarts and adjustments may occur during this process.

    We sincerely apologize for the continued disruption and appreciate your patience while we resolve this once and for all.