EnderDevelopment - Large Scale Issue – Incident details

Large Scale Issue

Resolved
Major outage
Started 8 months agoLasted about 8 hours

Affected

EnderDevelopment - Official Site

Major outage from 4:59 PM to 12:08 AM, Operational from 12:08 AM to 1:11 AM

Official Site

Major outage from 4:59 PM to 12:08 AM, Operational from 12:08 AM to 1:11 AM

Network - Server

Major outage from 4:59 PM to 12:08 AM, Operational from 12:08 AM to 1:11 AM

Server AMS-1 [Main]

Major outage from 4:59 PM to 12:08 AM, Operational from 12:08 AM to 1:11 AM

Updates
  • Resolved
    Resolved

    We are pleased to announce that all critical systems have now been fully restored and are operating normally. The majority of services, including website access, backend operations, and support platforms, are confirmed stable and performing as expected.

    However, the Customer Area remains temporarily offline.

    We understand this may cause some inconvenience, but this proactive step is essential to ensure a more robust, scalable, and secure experience moving forward.

    Our team will provide a follow-up update with an estimated time of completion for the Customer Area restoration.

    CEO of EnderDevelopment

  • Monitoring
    Monitoring

    We are pleased to report that we have successfully restored approximately 90% of our systems, and all restored components appear to be functioning as expected. Our team is actively monitoring the systems to ensure ongoing stability and performance.

    However, we are still experiencing significant issues with the customer area, which remain unresolved. Our technical team is prioritizing these problems and working diligently to address them as quickly as possible.

    We will continue to provide updates as we make further progress. Thank you for your patience and understanding during this time.

  • Update
    Update

    We have started the manual restoration from yesterday's backup on the rebuilt Main AMS-1 server. Our team is working at full capacity to minimize downtime and ensure a stable recovery.

    Services are expected to be fully restored within the next hour. We sincerely apologize for the inconvenience and thank you for your patience.

  • Identified
    Identified

    After thorough investigation, we have identified severe infrastructure-level corruption and irreparable data inconsistencies affecting the Main AMS-1 server. Despite recovery attempts, the integrity of critical system components and databases cannot be restored reliably.

    As a result, we have made the decision to perform a full reinstallation of the server to ensure long-term stability and data consistency.

    This process will require:

    - Complete OS and service reconfiguration

    - Manual restoration from a verified backup dated yesterday

    - Extended downtime, expected to last several more hours

    We sincerely apologize for the prolonged disruption. This action is necessary to prevent further instability and to rebuild the system on a fully healthy foundation.

    Our team is working around the clock to complete the restoration and bring services back online as quickly and safely as possible. We will provide another update as soon as the system is operational.

    Thank you for your patience and understanding.

  • Update
    Update

    We are currently investigating an issue causing instability in our services. Currently, the symptoms appear to be related to multiple factors, including:

    - Network system anomalies

    - Infrastructure stability fluctuations

    - Intermittent configuration issues and corrupt databases

    The technical team is fully engaged and analyzing all affected components to identify the root cause and restore stable operation as quickly as possible.

  • Investigating
    Investigating
    We are currently investigating this incident.