At 4PM on a Friday before a holiday weekend, one of Logicworks’ healthcare clients’ own engineer accidentally deployed a faulty software package to their entire system. Major parts of their system went down.
Logicworks Cloud Reliability Platform immediately detected the issue and alerted the incident response team. Logicworks senior engineers began to investigating the issue within seconds. They were able to identify the issue and work collaboratively with the healthcare company to remediate the issue without impact to Healthcare company’s business.
Things break. Engineers make mistakes. IT talent is difficult to hire and retain. Cybersecurity threats are on the rise. Get the continuity of care with 24×7 managed cloud support backed by certified cloud experts and platform support.
Challenges
- Software package deployed by the customer made accidental configuration changes to the entire system
- Configuration changes were deployed to nearly 200 servers, across production, stage, dev/test, and Disaster Recovery
- Since Disaster Recovery environment was impacted, failover was not an option
- Timing meant that internal teams were offline, on holiday
Solutions
- Logicworks engineers worked a combined 170 hours over three days to fix the issue
- Due to our in-depth understanding of this client’s systems, Logicworks was able to come up with a unique and efficient solution to a problem that their own engineers could not solve
- Created an automation script to programmatically fix error across environments
Results
After 170+ hours of remediation, the Logicworks team successfully brought the affected instances back to service with zero data loss, preventing a costly rebuild and potentially damaging customer relations. The incident not only highlighted the benefit of having a deep bench of outsourced cloud engineering talent, but also the value of outsourcing to a team of engineers with infrastructure-as-code platform experience. Logicworks’ unique approach to cloud management, which includes reliability and security automation, is customized for every client and empowers internal client teams to deploy faster with less risk.
Logicworks is a leading provider of platform driven cloud operations for AWS and Azure. Contact us today to learn how we can help you onboard to the cloud more efficiently, operate reliably with elevated security, and optimize as you scale.
