Cloudflare Dashboard and Cloudflare API service issues
Resolved
This incident has been resolved.
Posted Nov 06, 2023 - 19:40 UTC
Update
Restoration of our Control Plane API and Product Services to our primary datacenters in Portland has completed. Continuing to monitor.
Posted Nov 06, 2023 - 19:31 UTC
Update
Starting 11/06/2023 18:30 UTC, Cloudflare is initiating a restoration of Pages and Workers Control Plane API and Product Services to our primary datacenters in Portland. The restoration operation should last for about ~2 hours. During this window, Cloudflare API and Dashboard availability may degrade. We are working very diligently to minimize impact.
Posted Nov 06, 2023 - 18:22 UTC
Update
We're continuing to monitor stability and edge cases.
Material changes since last update: Logs are currently not being populated on the zero trust dashboard, dashboard functionality recovery continues
Starting 11/06/2023 17:00 UTC, Cloudflare is initiating a restoration of Control Plane API and Product Services to primary HA datacenters in Portland. The restoration operation should last for about ~2 hours. During this window, Cloudflare API and Dashboard availability may degrade.
Posted Nov 06, 2023 - 17:00 UTC
Update
We're continuing to monitor stability and edge cases.
Material changes since last update: Lists are still operational via Dashboard but may return 429 errors via API
Posted Nov 06, 2023 - 12:33 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Posted Nov 06, 2023 - 12:01 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Material changes since last update: Cloudflare Radar has been restored to operational status
Posted Nov 06, 2023 - 11:08 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Posted Nov 05, 2023 - 17:14 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Material changes since last update: Analytics has now been moved to operational status.
Posted Nov 05, 2023 - 00:31 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Material changes since last update: Lists have now have been moved to restored status. Lists can only be modified through the Dashboard currently.
Posted Nov 04, 2023 - 18:32 UTC
Update
We are continuing to monitor for any further issues.
Posted Nov 04, 2023 - 17:08 UTC
Update
We are continuing to monitor for any further issues.
Posted Nov 04, 2023 - 17:06 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Material changes since last update: CASB and DEX has now have been moved to restored status
Posted Nov 04, 2023 - 16:36 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Posted Nov 04, 2023 - 14:28 UTC
Update
All services have been restored to operational status. We're continuing to monitor stability and edge cases.
Posted Nov 04, 2023 - 07:58 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Material changes since last update: Email Notifications restored, Healthcheck analytics restored,
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Restored
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Dashboard availability restored
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Restored
* Stream: Restored
* Healthchecks: Restored
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): CASB API is online. Scanning engine is actively recovering, and is expected to have no data loss. Scanning engine recovery is expected to continue throughout the weekend.
* DEX (Digital Experience Monitoring): DEX API is online. Fleet status, HTTP, and Traceroute Test Result data are actively being restored with an expected recovery around Saturday Nov 4 15:00 UTC.
* DNS Updates: Restored
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Access API: Restored
* Images API: Restored
* Magic Transit: Restored
* Magic WAN: Restored
* SOC Proactive Alerts: Restored
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Restored
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Restored
* Direct Upload Deployments: Restored
Posted Nov 04, 2023 - 06:39 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Material changes since last update: Access API restored, Stream restored, SOC Alerts restored, Cloudflare Pages restored (create/delete projects and direct upload deployments).
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Dashboard availability restored
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Restored
* Stream: Restored
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): CASB API is online. Scanning engine is actively recovering, and is expected to have no data loss. Scanning engine recovery is expected to continue throughout the weekend.
* DEX (Digital Experience Monitoring): DEX API is online. Fleet status, HTTP, and Traceroute Test Result data are actively being restored with an expected recovery around Saturday Nov 4 15:00 UTC.
* DNS Updates: Restored
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Access API: Restored
* Images API: Restored
* Magic Transit: Restored
* Magic WAN: Restored
* SOC Proactive Alerts: Restored
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Restored
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Restored
* Direct Upload Deployments: Restored
Posted Nov 04, 2023 - 06:10 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Material changes since last update: CASB recovery continues, no data loss expected. Magic Transit Network Analytics restored. Magic WAN restored, Web Analytics restored.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Dashboard availability restored
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Restored
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): CASB API is online. Scanning engine is actively recovering, and is expected to have no data loss. Scanning engine recovery is expected to continue throughout the weekend.
* DEX (Digital Experience Monitoring): DEX API is online. Fleet status, HTTP, and Traceroute Test Result data are actively being restored with an expected recovery around Saturday Nov 4 15:00 UTC.
* DNS Updates: Restored
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Access API: Degraded
* Images API: Restored
* Magic Transit: Functional, and configurations possible from dashboard. Network Analytics is functional
* Magic WAN: Restored
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Restored
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
* Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 05:12 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Material changes since last update: Magic Transit dashboard configuration restored, Cloudflare Dashboard availability restored, DEX Test Result data recovery expected around 15:00 UTC Nov 4th.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Dashboard availability restored
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Restored
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is online. Fleet status, HTTP, and Traceroute Test Result data are actively being restored with an expected recovery around Saturday Nov 4 15:00 UTC.
* DNS Updates: Restored
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Access API: Degraded
* Images API: Restored
* Magic Transit: Functional, and configurations possible from dashboard. Network Analytics may be unavailable for Magic Transit traffic.
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Restored
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
* Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 04:06 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Material changes since last update: DNS updates restored, SSL restored, API restored.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Restored
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Restored
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Restored
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 03:09 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC. Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support Services: Portal, Phone, Chat are all operational
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
*Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 02:19 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: A large number of Logpush jobs have now been restored, and we expect the large majority to be restored by 07:00 UTC Most logs were lost during the incident; we will provide a full assessment once the incident is resolved
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Most analytics available in the dashboard and API have been restored. Some analytics datasets remain unavailable and will be restored in the coming hours. * Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
*Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 01:01 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: Logpush jobs are now recovering. Some logs may have been lost during the incident. Because the incident is still ongoing, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Some dashboard and API analytics have been restored. Analytics is still degraded and may be unavailable for some customers.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
*Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 04, 2023 - 00:20 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: Logpush jobs are now recovering. Some logs may have been lost during the incident. Because the incident is still ongoing, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Some dashboard and API analytics have been restored. Analytics is still degraded and may be unavailable for some customers.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
*Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 03, 2023 - 23:39 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Degraded, while processing is restored all services may not be generating email notifications
* Logpush: Logpush jobs are now recovering. Some logs may have been lost during the incident. Because the incident is still ongoing, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Some dashboard and API analytics have been restored. Analytics is still degraded and may be unavailable for some customers.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Restored without analytics
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
*Area 1 Email Security: All impacted services restored
* Cloudflare Pages: Service is restored. Creating/deleting projects degraded
*Direct Upload Deployments: Degraded
Posted Nov 03, 2023 - 22:52 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Logpush jobs are now recovering. Some logs may have been lost during the incident. Because the incident is still ongoing, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Some dashboard and API analytics have been restored. Analytics is still degraded and may be unavailable for some customers.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Degraded performance; Existing CASB Findings are available to view, but scanning is offline.
* DEX (Digital Experience Monitoring): DEX API is back online, fleet status data is being restored, test results are still offline.
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Restored
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
* Cloudflare Pages: Service is restored with the exception of creating/deleting projects being degraded
*Direct Upload Deployments: Degraded
Posted Nov 03, 2023 - 22:13 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Many customers are not receiving logs via Logpush. Some logs may have been lost during the incident. During the incident, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Analytics data is unavailable for many customers via the dashboard and API.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Restored
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Restored
* ZT Gateway: Restored
Posted Nov 03, 2023 - 21:14 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Many customers are not receiving logs via Logpush. Some logs may have been lost during the incident. During the incident, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Analytics data is unavailable for many customers via the dashboard and API.
* Workers Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Custom hostname validation delays
* ZT Gateway: Restored
Posted Nov 03, 2023 - 20:25 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Many customers are not receiving logs via Logpush. Some logs may have been lost during the incident. During the incident, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Analytics data is unavailable for many customers via the dashboard and API.
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Restored
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Custom hostname validation delays
* ZT Gateway: Restored
Posted Nov 03, 2023 - 19:35 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Many customers are not receiving logs via Logpush. Some logs may have been lost during the incident. During the incident, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Analytics data is unavailable for many customers via the dashboard and API.
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Running in degraded state and no logs or analytics at this time
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
* SSL for SaaS: Custom hostname validation delays
* ZT Gateway: Restored
Posted Nov 03, 2023 - 19:05 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: Many customers are not receiving logs via Logpush. Some logs may have been lost during the incident. During the incident, Cloudflare is not able to assess the full impact but will provide updates as appropriate.
* Cloudflare dashboard: Degraded availability but accessible
* Analytics and GraphQL API: Analytics data is unavailable for many customers via the dashboard and API.
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Running in degraded state and no logs or analytics at this time
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
Posted Nov 03, 2023 - 18:15 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates are being processed without delay
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Running in degraded state and no logs or analytics at this time
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
* DNS Updates: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact. All logs and analytics pipelines will be operational by Saturday 4th November 2023 at 7AM UTC.
Posted Nov 03, 2023 - 17:29 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Noticatifications: Emails updates from are being processed without delay
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Running in degraded state and no logs or analytics at this time
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
* SSL: Provisioning delays
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact. All logs and analytics pipelines will be operational by Saturday 4th November 2023 at 7AM UTC.
Posted Nov 03, 2023 - 16:55 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Noticatifications: Emails updates from are being processed without delay
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Running in degraded state and no logs or analytics
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: Updates to list items are temporarily suspended
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact. All logs and analytics pipelines will be operational by Saturday 4th November 2023 at 7AM UTC.
Posted Nov 03, 2023 - 16:22 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates from Cloudflare are being delayed or not sent
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact. All logs and analytics pipelines will be operational by Saturday 4th November 2023 at 7AM UTC
Posted Nov 03, 2023 - 15:44 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates from Cloudflare are being delayed or not sent
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream: Degraded performance but available.
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact. All logs and analytics pipelines will be operational by Saturday 4th November 2023 at 7AM UTC
Posted Nov 03, 2023 - 15:07 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Email Notifications: Emails updates from Cloudflare are being delayed or not sent
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream API: Not available
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Only read-only operations can be performed
* Registrar API: Only read-only operations can be performed
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 14:22 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: More jobs are being restored
* Cloudflare dashboard: Degraded availability but accessible
* GraphQL API: Degraded
* Analytics Engine: Not available
* API: Degraded availability but accessible
* Stream API: Not available
* Healthchecks: Not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: Potentially impacted
* Access API: Degraded
* Images API: Degraded performance
* Magic Transit: Configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: Configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: Not available
* Argo Smart Routing: Configuration now working, smart updates and analytics are offline
* Billing API: Degraded
* Email Noticatifications: Emails updates from Cloudflare are being delayed or not sent
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 13:23 UTC
Update
Cloudflare is working to restore impacted services. Below is a non-exhaustive list of the services currently impacted:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare dashboard: degraded availability but accessible
* API: Degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: potentially impacted
* Access API: Degraded
* Magic Transit: configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: not available
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
* Email Noticatifications: Emails updates from Cloudflare are being delayed or not sent
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 13:04 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare dashboard: degraded availability but accessible
* API: Degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
* Support portal: Not available, Enterprise customers are still able to raise tickets via Enterprise support email address
* Email Routing API: Not available
* CASB (Cloud Access Security Broker): Not available
* DEX (Digital Experience Monitoring): Not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: potentially impacted
* Access API: Degraded
* Magic Transit: configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: not available
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
* Email Noticatifications: Emails updates from Cloudflare are being delayed or not sent
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 12:11 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare dashboard: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: potentially impacted
* Access API: Degraded
* Magic Transit: configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: not available
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
* Email Noticatifications: Emails updates from Cloudflare are being delayed or not sent
* Lists: List configurations are working, but there may be delays in list updates propagating throughout the cloudflare network
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 10:59 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare dashboard: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN API: potentially impacted
* Magic Transit: configuration plane is working internally (contact your account team to make any urgent changes)
* Magic WAN: configuration plane is working internally (contact your account team to make any urgent changes)
* SOC Proactive Alerts: not available
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: degraded
* Custom Error Pages: not available
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 09:39 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare Dash: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* CDN-API: Potentially Impacted
* Magic Transit: configuration plane is working internally. Contact your account team to make any urgent changes.
* Magic WAN: configuration plane is working internally, Contact your account team to make any urgent changes.
* SOC Proactive Alerts: not sending updates
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 07:58 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare Dash: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Magic Transit: configuration plane is working internally. Contact your account team to make any urgent changes.
* Magic WAN: configuration plane is working internally, Contact your account team to make any urgent changes.
* SOC Proactive Alerts: not sending updates
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 07:36 UTC
Update
Cloudflare is working to restore impacted services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare Dash: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* SOC Proactive Alerts: not sending updates
* Magic Transit: not available to make changes
* Magic WAN: not available to make changes
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 07:30 UTC
Update
Cloudflare is seeing a gradual improvement to affected services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* Cloudflare Dash: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* ZT Gateway: restored
* Cloudflare API: restored
* Waiting Room: restored
* DNS Secondary: restored
* Magic Transit: not available to make changes
* Magic WAN: not available to make changes
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Billing API: Degraded
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 06:43 UTC
Update
We are continuing to work on a fix for this issue.
Posted Nov 03, 2023 - 06:42 UTC
Update
We are continuing to work on restoring service for the impacted applications.
Posted Nov 03, 2023 - 04:29 UTC
Update
We are continuing to work on a fix for this issue.
Posted Nov 03, 2023 - 03:05 UTC
Update
Here is the latest update:
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Cloudflare Dash: degraded availability but accessible
* Stream API: not available
* Healthchecks: not available
* Radar: Degraded
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Cloudflare Pages: restored
* Magic Transit: not available to make changes
* Magic WAN: not available to make changes
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Waiting Room: configuration updates are propagating with delay
* ZT Gateway: degraded but online
* DNS Secondary: not available to make changes
* Billing API: Degraded
* Cloudflare API: degraded but accessible
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 02:40 UTC
Update
We are continuing to work on a fix for this issue.
Posted Nov 03, 2023 - 01:13 UTC
Update
Cloudflare seeing gradual improvement to affected services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
* Logpush: more jobs are being restored
* WARP/ZT device posture: restored
* WARP/ZT Virtual Networks: restored
* Cloudflare Dash: degraded availability but accessible
* Cloudflare API: restored
* Stream API: not available
* Workers API: restored
* Alert Notification System: restored
* Healthchecks: not available
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
* Magic Transit: not available
* Magic WAN: not available
* Argo Smart Routing: configuration now working, smart updates and analytics are offline
* Workers KV: restored
* WAF: restored
* Rate Limiting: restored
* Rules: restored
* WARP/ZT Registration: restored
* Cloudflare Tunnel: restored
* Waiting Room: configuration updates are propagating with delay
* Cloudflare Pages: degraded but online
* ZT Gateway: degraded but online
* DNS Authoritative: restored
* DNS Secondary: not available
Logs
Some Logpush logs and analytics data were lost during the incident. Logs created towards the end of the incident may be recoverable; as the incident is ongoing, we do not yet know the full scope of impact.
Posted Nov 03, 2023 - 00:54 UTC
Update
Cloudflare seeing gradual improvement to affected services.
Data plane impact (i.e. full product functionality is either fully impacted or partially impacted):
Logpush: some jobs are being restored
WARP/ZT device posture + Select Virtual networks: ZT customers will have limited to no availability of private resources depending on how an organization is configured
Cloudflare Dash: degraded availability but accessible
Cloudflare API: degraded availability but accessible
Workers API: degraded availability
Control plane impact (i.e. product functionality is online however changes to configuration is impacted):
Argo Smart Routing: configuration now working, smart updates and analytics are offline
DNS Authoritative: online
WARP/ZT Registration: Devices should be able to connect and generally reach the internet
Waiting Room: configuration updates are propagating with delay
Posted Nov 02, 2023 - 22:56 UTC
Update
We are still seeing gradual improvements and working to restore full functionality.
Posted Nov 02, 2023 - 21:29 UTC
Update
We are still seeing gradual improvements and working to restore full functionality.
Posted Nov 02, 2023 - 19:55 UTC
Update
Cloudflare seeing gradual improvement to affected services.
Posted Nov 02, 2023 - 17:56 UTC
Update
Power to Cloudflare’s core North America data center has been partially restored. Cloudflare has failed over some core services to a backup data center, which has partially remediated impact. Cloudflare is currently working to restore the remaining affected services and bring the core North America data center back online.
Posted Nov 02, 2023 - 17:08 UTC
Update
We are continuing to work on a fix for this issue.
Posted Nov 02, 2023 - 16:38 UTC
Update
We are continuing to work on a fix for this issue.
The following products are currently impacted at the data plane / edge level, meaning that the full product functionality is either partially or fully affected: Logpush, WARP / Zero Trust device posture, Cloudflare dashboard, Cloudflare API, Stream API, Workers API, Alert Notification System.
These products are impacted at the control plane / core level, meaning that only the changes to the existing configuration are affected, but the product is functioning at the edge: Magic Transit, Argo Smart Routing, Workers KV, WAF, Rate Limiting, Rules, WARP / Zero Trust Registration, Waiting Room, Load Balancing and Healthchecks, Cloudflare Pages, Zero Trust Gateway, DNS Authoritative and Secondary, Cloudflare Tunnel, Workers KV namespace operations, Magic WAN, Cloudflare Images.
Posted Nov 02, 2023 - 15:31 UTC
Update
Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. We are working to resolve this problem.
The following products are currently impacted at the data plane / edge level, meaning that the full product functionality is either partially or fully affected: Logpush, WARP / Zero Trust device posture, Cloudflare dashboard, Cloudflare API, Stream API, Workers API, Alert Notification System.
These products are impacted at the control plane / core level, meaning that only the changes to the existing configuration are affected, but the product is functioning at the edge: Magic Transit, Argo Smart Routing, Workers KV, WAF, Rate Limiting, Rules, WARP / Zero Trust Registration, Waiting Room, Load Balancing and Healthchecks, Cloudflare Pages, Zero Trust Gateway, DNS Authoritative and Secondary, Cloudflare Tunnel, Workers KV namespace operations, Magic WAN, Cloudflare Images.
We will keep providing regular updates until the issue is resolved.
Posted Nov 02, 2023 - 14:49 UTC
Update
We are continuing to work on a fix for this issue.
Posted Nov 02, 2023 - 13:58 UTC
Identified
Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.
We will keep providing regular updates until the issue is resolved, thank you for your patience as we work on mitigating the problem.
Posted Nov 02, 2023 - 13:40 UTC
Update
We are actively investigating this problem and working on a solution.
The following products are currently impacted at the data plane / edge level, meaning that the full product functionality is either partially or fully affected: Logpush, WARP / Zero Trust device posture, Cloudflare dashboard, Cloudflare API, Stream API, Workers API, Alert Notification System.
These products are impacted at the control plane / core level, meaning that only the changes to the existing configuration are affected, but the product is functioning at the edge: Magic Transit, Argo Smart Routing, Workers KV, WAF, Rate Limiting, Rules, WARP / Zero Trust Registration, Waiting Room, Load Balancing and Healthchecks, Cloudflare Pages, Zero Trust Gateway, DNS Authoritative and Secondary, Cloudflare Tunnel, Workers KV namespace operations, Magic WAN, Cloudflare Images.
We will provide more information shortly.
Posted Nov 02, 2023 - 13:18 UTC
Update
We are continuing to investigate this issue.
Posted Nov 02, 2023 - 12:33 UTC
Update
Cloudflare is still investigating issues with Cloudflare Dashboard and related APIs.
This issue is impacting all services that rely on our API infrastructure including Alerts, Dashboard functionality, Zero Trust, WARP, Cloudflared, Waiting Room, Gateway, Stream, Magic WAN, API Shield, Pages, Workers.
Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.
Posted Nov 02, 2023 - 12:07 UTC
Investigating
Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge.
Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.
Posted Nov 02, 2023 - 11:54 UTC
This incident affected: Cloudflare Sites and Services (Access, Analytics, API, API Shield, Argo Smart Routing, Authoritative DNS, Billing, Cloud Access Security Broker (CASB), Dashboard, Digital Experience Monitoring (DEX), DNS Updates, Email Routing, Firewall, Gateway, Images, Load Balancing and Monitoring, Lists, Logs, Magic Transit, Magic WAN, Pages, Radar, Registrar, Stream, SSL for SaaS Provisioning, Support Site, Tunnel, Waiting Room, WARP, Web Analytics, Workers, Workers KV, Zero Trust, Zero Trust Dashboard).