Initial Investigation in the Bambu Cloud Temporary Outage

Latest Updates (2023/08/18): Update for Cloud Downtime

On August 15, 2023, between 10:03 and 11:10, and again from 12:11 to 12:23 UTC, we experienced an unexpected cloud outage. This interruption caused some disconcerting issues that we are keenly aware of and deeply concerned about. We believe in transparency, so we'd like to update you with our initial findings while our investigation continues.

The Outage's Impact on Users

During the downtime, our customers were unable to send print jobs from Bambu Studio. Additionally, some printers that were on during the affected period unexpectedly started a print job without user intervention. Since we're still in the process of analyzing the incident and have only looked into a portion of the cases, our findings might not yet tell the complete story. But rest assured, we will share additional information and insights as soon as we gather them.

What We Believe Happened

The initial conclusion—or rather, assumption—is that the incident was caused by a phenomenon we're calling "Job jamming." When the cloud outage occurred, our service became unstable. Jobs were sent to printers, but our system failed to receive confirmation of receipt. This led to repeat attempts to send the job, including some manual resends by confused users, which have been confirmed in a few well-documented instances. Automatic retry by the cloud service may also have contributed, though this is still under investigation.
The result was that the print job was successfully completed on the printer, but our cloud service believed it had not been done. When service resumed, the once-jammed job was resent, leading to the unexpected printing of an already finished job.
Simply explained, the print job sent to the printer before was trapped on the cloud and had a delayed start.

Our Apology and Commitment

We wish to extend our sincerest apologies for this incident. We understand the frustration and inconvenience it may have caused, and we accept full responsibility. It is difficult to have a cloud service 100% reliable all the time, but we should at least have designed the system more carefully to avoid such embarrassing consequences.
Our R&D and Cloud Networking teams are working tirelessly to identify the root cause and ensure that this never happens again. We are planning to implement changes, such as additional verifications before a print is started, to prevent future issues.
We are committed to further developing LAN Mode for our printers, which is a way that customers can use to send prints remotely to the printer if such Cloud Outage occurs.
Besides LAN Mode, SD card printing is also available as an offline method of printing.

We're Here for You if You Need Help

If you were affected by this cloud outage, and your printer has suffered any damage, please don't hesitate to contact our Support Team at Bambu Support. We're here to assist and make things right.
We greatly appreciate your patience and trust as we work through this challenge. Your satisfaction is our priority, and we're committed to keeping you informed as we learn more.
Thank you for your understanding.