Our operations team has deployed a fix globally and this incident has been resolved. At this time all traffic has been restored, and services are back to full operational capacity.
Posted over 1 year ago. Jan 01, 2017 - 05:28 UTC
Starting at 00:00 UTC on January 1, 2017, CNAME resolutions on some machines stopped working due to a bug triggered by the universal addition of one leap second, which affected both some authoritative DNS and origin DNS lookups, causing 5xx errors. Authoritative DNS was less affected since resolvers were served SERVFAIL and possibly retried until they hit a healthy instance. We deployed a fix at 01:30 UTC and it is rolling out to most affected sites. We will update this once the deployment is finished.