Late last year I decided to see how many misconfigured CI/CD (continuous integration and deployment) installations I could find on the internet. I decided to focus my research on one of the most popular CI/CD applications – Jenkins. This article isn’t an attack on Jenkins in any way shape or form – any piece of software and/or hardware can be configured incorrectly… it just so happens to be that CI/CD servers often host some very sensitive configurations – some of which I will come on to in a bit.
What is Jenkins?
Jenkins is a continuous integration/deployment web application built in Java. It is designed to run ‘something’ (usually tests to ensure the new code works and doesn’t break production) every time a developer commits code to a repository (Github being the most common) then delivering that tested/working code to an environment such as a production AWS account.
Given the nature of the beast, Jenkins typically requires credentials to the code repository and access to an environment in which to deploy the code to – in my research this was mainly AWS and Azure, in that order.
Given its position in the workflow, Jenkins is a very attractive target for attackers.
How did you find misconfigured Jenkins boxes?
I used my good friend Shodan to do the heavy lifting for me. I used a few search queries to narrow down Jenkins servers in regions and hosting providers I was personally interested in looking at.
What did you do?
As of this post there are approximately 25,000 Jenkins hosts exposed on the internet. I looked at about half of them and that alone took several weeks of my time – most of which was spent manually validating issues as well as reporting said issues to organisations where possible.
I ran a few searches on Shodan and downloaded the reports in .csv format. I created a list of URLs to crawl – specifically I was interested in Jenkins instances which required no authentication, instances with the self-registration link present and lastly, instances that had a SSO/SAML integration.
If I found that self-registration was enabled, I would register a test account on the instance and see if I was granted any permissions.
If SSO functionality/plugins were present then I would attempt to log-in to the application using a test account (i.e., a test Github/Bitbucket account).
If I gained access to a Jenkins instance (either by default or by registering/logging in with a test account) then I immediately attempted to find out which organisation it belonged to by checking the ‘system’ and ‘credentials’ pages.
I want to make it absolutely clear that I did not exploit any vulnerabilities to gain access to Jenkins servers – I simply walked through the front door which was visible to the world, then told the owners to close said front door.
What issues did you find?
- I will confidently say that between 10-20% of Jenkins servers I looked at were misconfigured. Misconfigured in this context means any one of the following:
- Wide open to the internet with either guest or administrative permissions by default – guest can be just as catastrophic and damaging as having admin rights
- The web application was behind a log-in prompt but allowed ‘self-registration’ which granted guest or admin rights
- The web application was behind a SAML/OAuth log-in linked to Github or Bitbucket but was misconfigured to allow any Github/Bitbucket account to log-in to Jenkins rather than being locked down to the organisation’s user pool
- Almost all of the misconfigured instances also leaked one or more of the following sensitive information:
- Credentials to private source repositories – this includes usernames, passwords and private keys
- Credentials to deployment environments – this was typically in the form of usernames, passwords, private keys and AWS access/secret tokens stored in plain text
- Job log files which did not obscure private/sensitive information – it was not uncommon for me to find credentials, tokens and other sensitive information in completed job logs.
What kind of servers did you find? What responses did you receive?
In no particular order I will list a small number of incidents I reported as well as responses received:
- London’s government funded transport body Transport for London had a Jenkins instance which exposed credentials to internally hosted private repositories. After messaging a number of contacts on LinkedIn, I finally got through to someone. I got a ‘thanks’ from them.
- Sainsbury’s and Tesco (two of the UK’s largest supermarkets) had misconfigured Jenkins servers. Sainsbury’s was particularly bad – they had exposed sensitive financial reports, service account credentials, Excel spreadsheets with hundreds of usernames and passwords, JWS signing keys, e-mail account credentials and much more. Of all misconfigured instances I found this was probably the worst. They valued this security disclosure with a £20 in-store voucher.
- Google had exposed sensitive tokens on their Jenkins instance. I reported the issue via their bug bounty programme where it was quickly triaged and fixed.
- A company who manufacturers toys for children had an exposed instance which granted administrator privileges and exposed sensitive tokens and credentials. The person I dealt with was awesome and offered to send some toys for my kids 🙂
- Clearscore is a service used by millions of people in the UK. It allows customers to request free copies of their credit scores, reports and much more. As you probably already know – these guys hold a lot of sensitive data about customers so you’d think security would be a top priority (just like it was for Equifax, right guys?).
I found an open Jenkins server that exposed artifacts, logs, credentials, tokens and more.
After hunting around for a security contact on their website (hint: there were none) I messaged a number of employees on LinkedIn. None replied. As a last resort (the only reason I persisted is because of the sensitivity of the data they hold on unsuspecting individuals using their ‘free’ service) I e-mailed a few generic e-mail addresses and finally got a response from someone who didn’t even work in the security team. I got a ‘thanks’ from them.
- Pearson is the largest education company and the largest book publisher in the world. They have no responsible disclosure process. Again I had to hunt for a contact on LinkedIn. Fortunately I found one. However, Pearson were the only company out of hundreds I contacted to respond in a negative way. It was implied that I was unprofessional for asking if they had a bug bounty programme.
I had found two Jenkins instances which exposed SSH private keys, credentials and tokens. The last interaction I had from Pearson was on the 20th September (after I had disclosed the issues). I was supposed to hear back from them but they went quiet after my disclosure.
- News UK is a large newspaper publisher in the UK. It’s ironic that for an organisation whose sole purpose is to have their voices heard by millions of people around the country, it is impossible to have your voice heard in order to report a serious incident. Again, no security e-mails or responsible disclosure process.
I reached out to a number of contacts on LinkedIn – one of which accepted my request but didn’t bother actually replying to my message!
News UK are leaking private tokens and credentials to both their WordPress instance and their AWS accounts.Of all the examples given above, News UK were the only ones to completely ignore my requests and e-mails. Unfortunately, sometimes naming and shaming organisations is the only way you can get them to notice and pay attention to security (or lack thereof).
It’s 2018 and most organisations don’t have the most basic of responsible disclosure processes in place. Surprisingly (or not) big names fall foul of this problem too.
I ended up sending e-mails to generic [email protected], [email protected] and [email protected] addresses as well as attempting to track down security contacts on LinkedIn; even then most didn’t bother replying.
If you work in InfoSec or are responsible for the security of your infrastructure, now’s a good time to methodically crawl through your infrastructure to ensure you’re not unknowingly exposing sensitive interfaces to the internet. It only takes one misconfigured instance to destroy your business.
Whilst you’re at it, create a quick security page on your company website with one or two contact methods so that security researchers can contact you easily; a security researcher shouldn’t have to jump through the hoops I had to just to report a security incident.
Bug Bounties programmes can also work really well so consider starting a private programme for your business if it’s appropriate for what you do.
Here are some examples of organisations doing it right: