Earlier today we experienced an issue that caused your GitHub project access to be revoked. This was caused by a regression in GitHub’s API that resulted in the loss of access to your GitHub projects on CircleCI.
We apologize for the confusion and downtime this may have caused for you and your team.
What happened:
Around 5:00PM UTC, GitHub pushed a change to their API that incorrectly omitted the permissions field of certain endpoints. This caused us to revoke permissions for many users.
Around 7:00PM UTC, GitHub deployed a fix of their API. This resolved issues for most customers. However, as an effect of the false-negatives on permissions, some users remained without access to a subset of projects. This subsequently caused some builds to appear to revert to CircleCI 1.0 configurations.
At 8:15PM UTC, we started deploying a fix which ensured users who lost access and now properly had access would automatically resume following projects on CircleCI. This fixed further issues regarding visibility and access to certain projects.
At 8:45PM UTC, we communicated through StatusPage that if our fix had not deployed to you yet, that a manual workaround involving resetting your GitHub webhooks existed.
What we’re doing to prevent this in the future: We are currently setting up systems to help us more closely monitor for changes to GitHub’s API that can cause these types of issues. Revoking access for users where we no longer see permissions present is the correct action for security purposes. Moving forward, we will maintain our focus on the security of your projects while also looking for ways to be more robust in the face of upstream changes.
Steps you can take: If your project continues to not build as expected, have your organization’s GitHub administrator take the following actions:
- Remove all CircleCI webhooks and services from your GitHub repo settings.
- Tell the project to stop building on CircleCI through “Project settings.”
- Add the project back on CircleCI.
Self-hosted CircleCI customers using github.com were also impacted and can use the steps described above to remove and re-add projects for builds to resume.
If you have any additional questions, please don’t hesitate to reach out. You can reach our support team here.
Sincerely,
The CircleCI Team