Xmarks is Back

Xmarks is back. I found this on their twitter feed:

We experienced unscheduled downtime, we apologize for the inconvenience. Please try a “repair” for ongoing errors: http://bit.ly/MlqXKf

Which leads you to this ‘perfect storm’ explanation:

Xmarks bookmark sync has experienced unscheduled downtime over the last 20 hours. This morning the decision was made to disable syncing to facilitate recovery.

Xmarks has gone to backups to restore the service for impacted Xmarks bookmark sync users. If you use Xmarks bookmark sync please double check any bookmarks you’ve made over the previous 48 hours from 7/1/2012.

At this stage all users should be back in working order from the server, if you’re having issues we’d recommend trying Xmarks Settings -> Advanced -> Repair first. You may want to consider simply using Upload instead to push your local set up to the server if you notice inconsistencies.

If you use Firefox you can reference the bookmarks backups that Firefox automatically creates: http://kb.mozillazine.org/Backing_up_and_restoring_bookmarks_-_Firefox

A number of issues came together causing Xmarks to experience this problem:

– While our datacenters were not impacted, our staff was impacted by the storms that hit the Washington DC area – leaving many of our employees without power, without Internet, and without working phones.
– Our offices are also without power impacted by the storms so using them was not a possibility either.
– Nearly all of our servers were impacted by the bug detailed by Mozilla here: https://bugzilla.mozilla.org/show_bug.cgi?id=769972
– We found that rebooting machines fixed the issue before we found out the true cause (and the above bug report). Rebooting worked but a number of machines failed to shutdown gracefully causing issues bringing back up the cluster cleanly.

We apologize for this issue and thank you for your patience. We will be looking into ways we can further mitigate our risks against threats like these in the future.

That Firefox bug, by the way is ‘Java is choking on leap second‘. That plus a major power outage is very very bad luck indeed. The leap second bug had some nasty effets around the world — grounding Qantas flights and crashing various internet services.

Previously: Xmarks is Down.