We apologize for a continuing Azure service outage that is impacting the following Wildbooks:
African Carnivore Wildbook
Flukebook
GiraffeSpotter
Wildbook for Iberian Lynx
SeadragonSearch
Sharkbook
Root cause:
Wild Me has received several generous rounds of Azure credits from Microsoft to help keep the costs of our mission down. As each round of credits has been used up (called a “sponsorship”), our account has reverted from a sponsorship to a paying account, and Wild Me’s payment card on file began to be charged again.
For unknown reasons, as this latest round of credits was used up, Wild Me’s sponsorship account was not converted into a paying acount but rather suspended with no description of follow up action. Even more confusingly, this suspension only applied to servers in the West US region of Azure. Servers in other regions were stopped temporarily but then were brough back online in the evning of 11/24.
Wild Me is working today (11/25) to unblock the final batch of US West servers and will report back here with progress.
Thank you and apologies fo the temporary interruption. All data and platforms are intact. We’re working to resolve issues on the Azure administration side that are out of our control.
Status Update: AFter two full days of working with Microsoft Support we may have identified the issue, and Microsoft is working on the fix. We have given them until Noon Pacific time (+4 hours) to resolve the issue. If they have not, we will proceed to migrate the data to a different region and rebuild these servers manually. We did try to redeploy them in unaffected regions, but unfortunately the error blocked their easy migration through the standard Azure tools.
Status: All Wildbooks are back up and should be at 100%. Please let us know if your Wildbook remains affected in any strange ways. We observed no loss of data. We are separately tracking and working to resolve an unrelated issue with the machine learning server for the African Carnivore Wildbook.
Ultimately, the Azure Support team was not able to assist in the issue, and Wild Me staff found workarounds in Azure to bring our infrastructure online. We are now better prepared to handle unplanned outages in Azure, but we are also now considering other cloud providers after a poor showing by their support team.