Orchestrating deployment of @myexploit2600's hacklab with Ansible and Vagrant [REDUX]
Deploy your hacklab using Ansible and Vagrant for fast, repeatable results. Building on the work by @myexploit2600, we're going to use Ansible and Vagrant to automate the manual steps of constructing a domain and vulnerable users.
Hey, thanks for coming to read this blog. This is an amalgamation of 5 posts I published on my blog in early 2021 that I took down cos I like breaking links on the internet! Andy has kindly let me republish it here after calling me out in his Steelcon Talk. I've tried my best to reformat correctly, remove references to being separate posts, and update links where necessary. I have not retested any of the playbooks or configs so it may be full of gremlins - play at your own risk.
You're a thoughtful and well-informed pentester/consultant/hacker, so you've obviously read the superb blog post by @myexploit2600 about building and attacking your own Active Directory lab. Now you'd like a repeatable way to set it up repeatedly with fresh virtual machines, without doing all the steps manually, right? Lucky you, here we go.
This short post will not be a technical deep dive, but rather a quick overview of the PowerShell scripts I used and the Ansible playbook I wrote to automate the deployment of my hacklab using @myexploit2600's guide.
"But Aidan, I've never heard of Ansible and I'd like to learn more before using it on my machine/network/smart fridge."
The good news is that Ansible is owned by Red Hat and has an exceptional attitude towards documentation. If you'd like something audio-visual, there is a 1h30m webinar with all the basics you need to know. Alternatively, if you prefer to read about Ansible, Red Hat's got you covered there too.
Virtual Machines (Windows server and workstation)
This blog is going to presume you've followed the short instructions on creating your server and workstation virtual machines; whether in Virtualbox, VMware, or Hyper-V. Since you can create snapshots or checkpoints in most commercial virtualisation software, rolling back to the earliest stage is a simple process that avoids having to re-deploy the virtual machine (VM). Get your VMs prepared as far as having reached this point in @myexploit2600's blog.
Once you've established this baseline installation, you can quickly roll back to here should you need or want to.
Firstly, we'll repeat the sanity check of confirming our PowerShell version before trying to execute anything. Here's a little statement taking advantage of PowerShell's comparison functions:
Preparing Ansible and inventory
You may already be aware that Ansible cannot be run on a Windows host but it can manage them. If you have Windows Subsystem for Linux (WSL), you're good to go, otherwise you will need a Linux host of some description which can reach the hacklab hosts. As mentioned before, this is not a technical deep dive or a guide to Ansible, so I'll point to the relevant documentation.
Managing Windows targets with Ansible requires the WinRM service be enabled. It is possible to use Windows OpenSSH but this remains experimental. Since we're setting up a hacklab, I'm going to take a few liberties when it comes to "best practice". We're going to use the script provided by the Ansible team to set up WinRM for all the Windows hosts in our lab. The script can be found here and can be executed with wanton disregard for safety by using the following snippet:
This will expose an HTTP listener on port 5985 and a matching HTTPS listener on port 5986 using a self-signed certificate. It also enables Basic auth although we are going to use NTLM authentication, rather than just plaintext credentials. You can set up listeners in whichever way you feel comfortable. Ansible's WinRM module supports all of the standard Windows authentication methods so if you're concerned with establishing real confidentiality, the options are there. I'm setting this up in an isolated network segment since it's going to involve configuring an intentionally vulnerable domain, therefore a self-signed certificate is no big deal.
Once the listeners are established on your hacklab hosts, we can start work on our inventory. This part is straight forward. The inventory can be declared globally using /etc/ansible/hosts, or a contextual inventory file can be defined when executing a command or playbook using -i /file/path. I prefer to use the YAML format over the older INI format for inventory, but both are valid and supported by default.
I've intentionally used an FQDN for the server and specified an IP for the workstation here to illustrate how host declarations can work. If your Ansible host cannot resolve the FQDN (no DNS, not in the hosts file, etc.), you would need to specify the ansible_host variable as exemplified for labpc1. I've used the global variable declaration because the authentication method for both is the same, but I have declared per-host variables for usernames and passwords. These will be the local credentials for the relevant hosts since there are no domain credentials yet. If you're using a different authentication method, you will need to swap out the variables as appropriate.
The next step is installing the necessary Ansible plugins to support interaction with Windows. community.windows will automatically install the ansible.windows plugins, but we'll stick it in the command for good measure.
If you'd like to test that the inventory file is sound and that WinRM has been configured correctly, you can now fire off a quick ad-hoc command using win_ping. The inventory flag and target can be omitted if you use the global inventory.
$ ansible -i my_inventory all -m win_ping
All being well, you should get console output that resembles mine..
There are a wide range of errors you could run into at this stage and it goes beyond the scope of this post to troubleshoot them all. I'm not on Twitter anymore but I suspect if you tweet about your problem and @ZephrFish, he'll get your question to me.
Assuming you have successful ping responses, we can now execute our playbook to configure the domain and then we're almost finished. Ansible, much like Python, is self-documenting code, which explains what it's doing via syntax. So in the interests of brevity, I'm going to share the complete playbook right here with some light commenting and then explain the few quirky parts.
If you want to jump ahead to executing this with reckless abandon, then you can do so with ansible-playbook playbook.yml -i inventory.
As mentioned, there are a few quirky bits in the playbook to look at more closely. Firstly, we're calling the domain hacklab.local instead of server1.hacklab.local. The reason for this is to allow the domain name to more accurately reflect a real-world domain while we use server1 as the hostname for the Windows Server/Domain Controller. Simple enough really but if you want to revert this change, then switch the values on line 33 and 34 with server1.hacklab.local and server1 respectively, and then the value on line 60 with some other name of your choosing like dc01 or keith.
Secondly, we jump down to line 76 where we're breaking out into a raw PowerShell command to add our service principal name (SPN). I was unable to find an Ansible module that specifically supported adding an SPN so this is the dirty solution. Since it's not taking in any user-supplied arguments, it's safe enough, but it's just a bit untidy.
Lastly, there is some jiggery-pokery on lines 123 and 124, where we point our workstation to the newly configured domain controller. Varying versions of Ansible support one or both of the directives used here. In the interests of pOrTaBiLiTy, we're using both because they do not conflict, and Ansible will pass the task correctly even with both present.
All the other tasks, I believe, are self-explanatory, and the name key-value of each one gives a plain idea of what they're doing. You can search for ansible $insert_unknown_phrase_here to find extensive documentation on most of it.
As mentioned, the execution of the playbook is achieved by performing the following command, omitting the -i inventory if you're using the global inventory in /etc/ansible/hosts.
$ ansible-playbook playbook.yml -i inventory
This will begin the execution of the playbook in sequence, configuring the server as the domain controller and provisioning the hacklab domain. Then it will create the victim and attacker users before attaching the workstation to the domain, ready to be used to carry out your kerberoasting attacks and more.
"You're using passwords in plaintext you monster"
I am indeed a monster because I live fast and have designs to die young. Ansible has a really neat secret management tool called Ansible Vault which you can read about and implement pretty quickly yourself. This is left as an exercise for the reader... ahem. Alternatively, you can employ a .env file, environment variables, or an Ansible vars file to separate the egregious use of plaintext passwords in the playbook. As previously alluded to, I'm running this in a Fort-Knox-slash-Fritzl-dungeon-style network segment where I'm not concerned about APT-01 robbing my noodz. If you have the desire for the most robust practices, Ansible is very much designed to enable that.
The beauty of the playbook is that you can replay it against the same hosts repeatedly without the fear of duplicating configurations, users, or other objects. Ansible is idempotent and will not try to repeat tasks unless explicitly told to do so. This is useful if you have reduced resources available and the installation of features takes longer than the timeout. Indeed, if the playbook fails due to a timeout, you can run it again with pretty high confidence that it will carry on right after the failed step.
Once the playbook is finished, you can jump on to your workstation, log in as hacklab.local\user2 and get roastin'.
Automating virtual machine deployment with Vagrant
Moving onwards, once again, we're going to be standing on the shoulders of giants and this time it's Jordan Borean (@BoreanJordan), one of the Ansible developers, and Gusztáv Varga (@gusztavvargadr), a principal software engineer at Knab bank.
For our lab, we need Windows Server 2016 and either Windows 7 or 10 (I've opted for 10). Jordan Borean who is specifically involved in the development of the Windows support in Ansible has produced a Server 2016 Vagrant box while Guztáv has produced a Windows 10 Enterprise box. Both boxes attach evaluation licenses on deployment (180 days and 90 days respectively) which ties very neatly into our desire to be able to nuke and reboot the hacklab at will. Both boxes use the credentials vagrant:vagrant and expose WinRM listeners, allowing us to immediately communicate with them using Ansible. What more could we ask for?
As a side note, having looked at the well-documented repositories used to construct both boxes, I may look at spinning forked copies which are prepared specifically for the hacklab to save some time at the Ansible stage.
"This is written for Hyper-V?! You said you'd do it in Virtualbox. You sit upon a throne of lies!"
Correct. My lab environment runs on a Hyper-V server so this is my working, tested Vagrantfile that I've used to deploy to my own infrastructure and that I know works. Fear not, however, as I will show you, with the magic of ~*~aNoThEr CoDe SnIpPeT~*~ how to configure it for Virtualbox. If you are using Hyper-V like me, you will note that the Virtualbox Vagrantfile specifies a network setting absent from the Hyper-V Vagrantfile. This is because Vagrant cannot (yet) manage Hyper-V network adapters. As such, when you launch the Vagrantfile, you will get an interactive prompt to select which interface you'd like to supply to the machine.
This Vagrantfile will configure the boxes similarly to the Hyper-V variant above. You can adjust the memory and CPU assignments to whatever fits best for your available resources. The workstation will run fairly with 2GB RAM and 1 CPU but the server will start to creak a little with less than 2 CPUs while 2GB memory will make it a little sluggish.
The Virtualbox configuration sets up both boxes on the host-only adapter but you can change these as you see fit. Take a look at HashiCorp's documentation for more guidance.
Invoking the Ansible playbook with Vagrant
The key benefit to nesting the Ansible playbook into the Vagrantfile is that we don't need to define an inventory or worry about authentication. Vagrant already has access to the machines and can execute the playbook based on the hostnames defined. We need to modify the playbook a little to integrate with Vagrant and to account for some differences between the standard ISO/VHD evaluation versions from Microsoft and the Vagrant boxes we're using to replace them.
Firstly, we're force-installing the DNS service because of some funkiness I ran into during the setup stages. We need it installed anyway so there's no redundant steps here and this ensures we get the services we need. Secondly, we're targeting Vagrant's dynamic Ansible inventory to get the IP addresses for setting DNS on both hosts. Lastly, we're disabling IPv6 because it is the devil's work and we don't abide by hexadecimal layer 3 addressing round these parts - we worship the dot-decimal notation and so should you.
As mentioned, we no longer need to define an inventory because Vagrant can execute the connection for us. We do, however, have to override its default connection type to make it use NTLM. Fortunately, we can do that in the same code block where we tell it to use our playbook. You need to add the following snippet inside the parent code block of the Vagrantfile from above (i.e. after the Vagrant.configure... line and before the last end).
If you insert the code block as-is, it will generally execute once per machine meaning that things will get double provisioned - which is absolutely fine. However, if we use a little conditional magic, we can make sure to invoke it only once. Stick the following code block in instead, as the last thing to be executed inside the config.vm.define opts["name"] do |machine| loop, and it will ensure the playbook is only executed after the workstation comes up, which will only be provisioned after the server is up.
You need to put the Vagrantfile and playbook.yml in the same directory or change the location directive (ansible.playbook = "playbook.yml").
You will need, obviously, to have installed Vagrant and Ansible on the host you're using to build the lab. The build can then be started by simply running vagrant up in the directory. If you have to re-run it due to any timeouts or non-blocking failures, you'll need to run the same command but include the --provision flag to ensure Ansible is executed again too - this also works if you want to apply changes you've made to the Ansible playbook post-deployment. If all goes well, you will end up with a fully configured hacklab ready to go. If you want to power the lab down, you can use vagrant halt, and if you want to scrub the lot, vagrant destroy.
Add some more users!
"I want more users to target, please."
Sure thing. We're going to employ Ansible again to add 4 more users to the domain. As usual, let's get the playbook and then explore the important parts.
The first user we add to the domain is configured to use AES-256 for Kerberos ticket encryption. However, it still uses a weak password. Simply changing the encryption type in use does not mitigate the risk of kerberoasting.
The second new user has the domain pre-authentication requirement disabled. This means that the user is susceptible to an attack termed AS-REP roasting. In short, you can kerberoast this user without domain credentials. Get yourself onto a logical network location where you can simply communicate with a domain controller and you can now recover kerberos tickets for any user with pre-authentication disabled.
The third user is a group managed service account, meaning that its password is managed automagically. The password set is 240 bytes (120 characters) and is generated in a cryptographically secure fashion. If you crack the password on this account, please let me borrow your access to the NSA's cracking farm.
Important: In order to add the Key Delegation Server root key, we need to be a Domain Administrator, instead of a built-in Administrator on the DC. We can achieve this in one of two ways. We can promote the vagrant user we've been using for Ansible (which is a built-in admin) or we can change our entry for the DC in our inventory to use user1 instead as it is already a Domain Admin. It's up to you which way you'd like to do this.
The fourth and final user added has a complex, 64-character, alphanumeric and symbol laden password. Short of adding this password directly to your cracking dictionary, this would be resistant to cracking attempts. Despite the fact the user is still using RC4 encryption for Kerberos tickets, it is exceedingly unlikely that even a brute-force guessing attack would recover the plaintext.
The idea for the accounts added here was inspired by @ZephrFish's Paving the way to DA series, which I highly recommend if you'd like a delicious wine pairing to go with this blog. In the first part of the series, Andy demonstrates an attack targeting an older vulnerability in Windows domains affecting Group Policy Preferences. As he discusses, this issue only affects Server 2008 R2 or earlier, since Microsoft rolled out a patch in 2014. We can't, therefore, target the Server 2016 instance in our lab and we'll need to spin up a new server.
There are a few ways we could approach this but I'm going to create an isolated domain for this scenario because, frankly, managing Windows Server 2k8 from Ansible is a bit of a ballache and it was hard enough getting it working in its own domain. Additionally, we're going to have to do some fakery because, as of yet, I have been unable to find a way to programmatically interact with the necessary Group Policy Preference. Not to worry though, the scenario will behave familiarly enough to a real world environment to be useful practice.
Provisioning the lab
Firstly, we have to provision our Windows 2k8R2 server. As per usual, lets get the playbook out and have a look at what we've got.
There is a lot going on here. If you've been following along the past few posts, it will look vaguely familiar but you will likely note the increase in use of raw shell commands. This is primarily because the Ansible modules rely on newer PowerShell modules and updated functionality of modern Windows server versions. I've endeavoured to make these as idempotent and error-resistant as possible but I make no guarantees.
Once executed, the playbook will put the server in much the same state as we have our 2016 server. However, before we can use the playbook, we need to spin up the virtual machine. Let's look at our Vagrantfile.
Once again, we're relying on the hard work of Jordan Borean. This is my Hyper-V version but I've run up a quick Virtualbox version too.
Important: When you first run this Vagrantfile, it is likely going to run into an error - the Windows 2008 Server may not be licensed. The solution is quick and easy. Connect to the VM via the GUI and when the dialog loads, choose the Ask me later option. After this, you can run vagrant up --provision and the provisioning will proceed as normal. I will try to solve this over the next few days and get rid of this friction - for the time being, sorry, but this is the best option.
I've also provisioned a Windows 10 workstation specifically for this scenario but this is not explicitly necessary. You can repurpose the existing Windows 10 host. Either way, you'll want to target it with a playbook similar to this. The only modification I've made is to target the gpp.hacklab.local domain defined in the earlier playbook. You will need to swap out the comments on line 19 and 20.
Once you have provisioned the server and either spun up a new workstation or redirected the other, we can create our SYSVOL/GPP scenario from @ZephrFish's post.
Creating the attack scenario
This was an enormous pain in the arse. Newer versions of PowerShell and associated modules introduced the ability to target more of the Group Policy Management (GPM) options. Namely, Get-GPRegistryValue and Set-GPPrefRegistryValue, which I expected to allow me to access the necessary entries to simulate the weakness detailed in Zephr's blog. Nope. That was foolish of me to assume they did what they were supposed to do. The only way to create a Group Policy Preference (GPP) defining a built-in Administrator password, as far as I have been able to divine, is through the GPM GUI. This, as you would expect, does not go very well with the idea of automating the lab.
However! Since we know how this weakness occurs, we can manually simulate the conditions in a way that lets us practice the attack in a realistic manner. When the GPP is created, it writes a directory to SYSVOL which contains all the components needed for us to perform the actual attack. So if we produce it using the GUI and then create a copy of the directory and contents, we can, in a sense, replay it by pushing the copy to the correct location. This is not ideal because it is not how most systems administrators would handle such administration (although, it is not entirely farfetched that some might do it this way). So, lets take a look at what we're going to push into SYSVOL. You do not need to repeat the following steps yourself because I'm going to provide the zipped GPO for you. However, if you want to produce your own, here's how.
In the subdirectory path Machine\Preferences\Groups, we find Groups.xml.
Here we can now see the cpassword field where the eNcRyPtEdpassword resides. The next step is to stick this whole directory tree in a zip archive so we can pull it to our host system and use it to inject the policy later. First though, let's quickly follow one step of Zephr's guide to make sure the policy is targetable.
That's a bingo. We can see the path to the Groups.xml that the script is pulling the cpassword value out of and very kindly decrypting for us too! Now then, let's delete the policy via the GUI and then use Ansible to push the zipped policy right back into SYSVOL. Deleting the GPO is as easy as right-clicking on it from the GUI and its associated subdirectory will then dissapear from the \\hacklab.local\SYSVOL\hacklab.local\Policies directory.
Injecting the vulnerable GPO
The first step in Ansible is to copy the GPO archive on to the server and then we can simply unzip the archive. Thankfully, this is a really simple operation.
As I said above, I've pulled the GPO from my server and zipped it up for you to use if you'd like to save time. If you want to use mine, you can download it here:
Once you've downloaded it or created your own, you can stick it in the directory with your Ansible playbook and Vagrantfile. The snippet above, if added to your playbook or run separately (you'll need to add a hosts and tasks declaration), will push the archive onto the target and then unpack the archive into the correct subdirectory of the SYSVOL share.
And if we jump back on to our workstation and run Get-GPPPassword again, we can once again collect the decrypted cpassword value.
That was a lot of work to pull a password out of a file, right? However, we now have a viable Server 2008 (R2) lab. We could explore some legacy attack paths and techniques. I can say confidently that I still encounter 2008 servers in the wild - hell, 2003 still hasn't fully died a death yet - so there's a lot of value to be found in learning how to break legacy environments.
Domain trust exploitation
Compromising a Domain Admin user during a pentest is a great result. Leveraging that user to establish a foothold into adjacent forests by exploiting domain trusts could lead to much more serious implications for your client, their suppliers, and their customers. I'm not going to belabour an existing attack by explaining how it works when others have done a better job than me already so please go and have a quick read of @spotheplanet's iRed.team blog about escalating from Domain Admin to Enterprise Admin.
We're going to deploy three Windows domain controllers and two domain forests, and a child domain for one of the forests. Once deployed, the lab will resemble the setup right before the Back to Empire: From DA to EA section in @spotheplanet's blog.
I'll share the playbook and Vagrant file as usual but you might notice some changes in the way things are done earlier. The biggest change is the switch to using OpenSSH to communicate with the VMs. WinRM was proving to be grossly unreliable. Any amount of network volatility could result in a failed provision and a borked DC. The change to OpenSSH has massively increased the reliability of both the Vagrant and Ansible stages of provisioning.
The second big change is the prevalent use of win_shell to invoke PowerShell commands directly, instead of relying on Ansible modules. This is mostly due to an absence of the right functionality to get the domains behaving exactly as I needed them to. As much as possible, I've written in conditional logic to prevent replying commands if they'll throw an error due to existing or duplicate objects in the AD.
Lastly, I'm not going to supply the Virtualbox version of the Vagrantfile because it should be easy enough now to make the changes by referring to previous config versions.
Looking through the Vagrantfile, we're provisioning three Windows 2016 servers, named Westminster, Holyrood, and Senedd - can you see where this is going yet?
The scenario we're going to construct is one of a independent DC which has broken away from the shackles of an oppressive forest to set up its own, nicer forest, with continental friends. Residents of one of the remaining child domains of the original forest, seeing this grand leap for independence, are inspired to make their own push for freedom by.. exploiting a domain trust. The Ansible playbook is going to establish two forests; one on the Holyrood DC and another on the Westminster DC. The Senedd DC will then be attached to the Westminster domain before being nested into a child domain. After that, we'll establish a unidirectional trust between the two forests, creating an avenue by which an ambitious user in the Senedd child domain might achieve privileged access to the Holyrood domain.
The playbook is self-explanatory for the most part. We're preparing all the hosts with the necessary tools and modules to build the domains. Then, we're building the Westminster forest and Senedd child domain, after which we build the Holyrood forest. Lastly, we establish the inbound domain trust from the Westminster forest to the Holyrood forest.
With the hosts deployed and provisioned, we can step through the stages of compromise outlined in @spotheplanet's post. You may notice that their post targets an additional user that isn't present in the environment provisioned by our playbook. For this post, I've focused on deploying the domains and trust. The next blog, coming very soon, will introduce a second playbook to add some targets to the estate. In fact, we're going to be adding targets in a randomised and somewhat obfuscated fashion to introduce a bit of challenge to the practice, so stay tuned! Wouldn't that be nice... this bit was written when I had a lot more time and motivation to write about orchestration. If there's enough interest, I'll do this.
Troubleshooting, problem solving, bug squishing, addendums, and errata
If you're having any kind of issues with the playbooks, Ansible set-up, Vagrant configs, or anything else, try turning it off and on again. I'm happy to give you a hand but I'm not on Twitter and I'm generally allergic to social media! Try pinging @ZephrFish and tell him I'm ruining your life. If you get on absolutely problem free, I would love to hear about that too.
And a huge shout out, of course, to @myexploit2600 for writing a brilliant set-up guide for the AD hacklab and for continuing to share his expertise so freely. And a massive thank you to @ZephrFish for his continued work on Paving The Way To DA and for letting me post this perishingly long out of date blog on his website.