Update for Cloud Downtime
What happened?
After further investigation, we are able to provide more information about the Cloud Outage we suffered on August 15, 2023.
The root cause of the issue was related to the MQTT SDK used in the cloud connection. One of the two MQTT SDK clients was disconnected due to a timeout. Usually, the service will automatically restore the connection, but in this case, the service returned a successful connection report when, in reality, the connection was not re-established. During this time, a number of print start messages accumulated.
A second connection failure was detected and the service managed to reconnect successfully this time, but the result was the system sending the accumulated print start messages to a number of printers that received the information and started printing if the printer is idle.
There was another unfortunate issue that contributed to this problem. The large number of API access requests done at the same time have impacted the API service, which did not allow it to respond in a timely manner. Bambu Studio has a logic implemented to reinitiate a print request immediately after accessing the cloud and this has increased the accumulation of API access calls which exceeded the API service capacity.
How do we plan to solve it?
Bambu Lab printers equipped with a LIDAR (X1 Series) will receive a new firmware feature that will allow it to check if the printer has printed models that have not been removed from the plate. The printer will perform this verification before every print, and if it detects a problem, the user will be prompted with a message on the screen, Bambu Studio, and Bambu Handy to request user intervention and confirmation.
Printers that are not equipped with a LIDAR (P1 Series) will display a reminder about the need of cleaning the plate before starting a print, asking for the customer to interact with the message and confirm the operation has been done. This message will be displayed on the screen, in Bambu Studio, and Bambu Handy.
Both printers will have the option to enable or disable this feature in the printer settings, but the default option will be for the function to be enabled.
Another improvement to increase the security and safety of the printer, which will be applied on all 3D printers, is to continuously monitor the hotend and heatbed temperature. If a fault is detected, an error message will be prompted on the printer screen, Bambu Studio, and Bambu Handy, to alert the user and avoid any potential hazards. Of course, the heaters will be turned off to further minimize any potential risk.
We will also make significant improvements to the Cloud Printing logic to ensure that every time print is initiated, the printer will check the timestamp and automatically discard any outdated print which does not follow our strict configuration. This has already been implemented on the servers, but it will require additional features added in the firmware too.
Significant LAN Mode functionality will also be implemented in future firmware updates to further increase security and the capabilities of the system. File Management and Media Download functionality will be implemented, while User Certificate authentication for connectivity will be available to ensure secure connections in shared network environments.
When will this happen?
As you might know, to ensure the quality and reliability of software updates and to also be able to bring you other new features we are working on, we will need some time to do all the changes we mentioned in this update.
Some of these updates were already on our list of features to work on, but security and safety concerns discovered moved these features to the top of our list of priorities. Our team will do their best to deliver these new functions in the shortest amount of time possible.
On the Cloud Service side, some changes have already been implemented to mitigate any potential issues. The SDK service logic has been updated and we have increased the database connection sizes for better throughput.
Service Commitment
All of our customers who have suffered hardware failures due to our Cloud Outage will receive assistance to repair the printers if required, and bring them to the initial state in the shortest amount of time possible.
Spare parts will be provided to replace the damaged ones and spools of filament will be provided to compensate for the wasted filament generated by the printer.
To do this, please don't hesitate to contact our Support Team at Bambu Support and provide us with the logs of the printer to ensure everything works within the expected parameters, and we cover any potential issues.
We will also provide our customers whose printer got damaged with two spools of randomly selected PLA filament as compensation for the trouble this has caused.
