Brief Crocodoc service interruption – Friday June 16th
Canvas Incident Report
On 16 June 2016 the Crocodoc document conversion and preview service was non-functional within Canvas between 1:54 PM MDT (19:54 UTC) and 3:43 PM MDT (21:43 UTC). This was due to a configuration error within the application, which temporarily disabled the Crocodoc integration.
Incident Summary
At 1:54 PM Mountain Time (19:54 UTC) on 16 June 2016, a global setting for the Crocodoc service in Canvas was inadvertently disabled for all Canvas accounts. Users began seeing the message “Crocodoc is still processing this document” for assignment submission files they were attempting to view. However, they were still able to see a preview of those files, as previews appeared in the Box previewer service after failing to load in Crocodoc.
Support was alerted to the errors in Crocodoc at 2:14 PM Mountain Time and reached out to engineering to assist in finding and addressing the root cause. Our initial investigation seemed to indicate that the errors we were seeing were coming from the Crocodoc service itself. We contacted Box, who quickly escalated a ticket to assist us in the matter. When Box replied again, they let us know that their servers were no longer receiving requests from Canvas, and that the service was functioning properly on their end. This indicated that it was an issue with the application in Canvas, and troubleshooting continued for our DevOps and other engineering teams.
At 3:27 PM (21:27 UTC), we noticed that the Crocodoc service had been disabled for one of the accounts for which users had reported the problem. Services such as Crocodoc in Canvas are configured as plugins, to either inherit from a global setting for all accounts, or which can also be enabled or disabled for individual accounts. Engineering found that the global setting for this service had been set to a null value for the Crocodoc plugin. This setting was enabled once again, and Crocodoc file previews returned to normal functionality by 3:43 PM (21:43 UTC).
