Complete failure in CCDC
Posted on 2014-02-03
So I competed in CCDC (Colligate Cyber Defense Competition) qualifier yesterday and let me just say that it was quite a few steps above anything I've competed in before. However, that is still no excuse for my team's inability to advance past the qualifiers.
The team was fundamentally flawed, and the competition itself could use some overall improvement as well. Lets get started with explaining how everything was setup.
The scenario we were given is that we are a new IT Team filling in for a company who just fired it's previous IT team. Our job was to secure the machines as best we could while also dealing with two rather problematic factors.
1. After about 20 minutes, our network began being attacked by a team of professional penetration testers.
I only know two of the people who were on this team (The Red team, for future reference). Ironically enough I have worked with both of them in a previous job of mine, and I even suspected that one of them would be part of the red team as he has been previously. More on this later.
2. Occasionally we would receive "injects" from the white team (In this scenario: They were playing as the people running the company we worked for). These injects could be anything from install a software to setting up new services for the network.
Each CCDC Team is allowed 8 people to compete. We have the entire internet and any printed reference material at our disposal.
Our virtual network consisted of 3 different subnets. The DMZ, which contained a Centos E-commerce machine, and an Ubuntu DNS Server. On the LAN subnet, there was a Debian e-mail server, a windows server 2003 machine running FTP, a windows 2008 server acting as a DFS host, and a windows server 2008 R2 acting as our Active directory server / internal DNS server. (ADDNS).
Connecting these two subnets was a PFSense firewall, (PFSense being a specialized distribution of Linux, made specifically to act as a firewall) and finally the third subnet being an outside employee machine, running windows 7.
All of these machines, inclusive of the PFSense firewall were said to have no guarantee of security. This means we have 8 team members with 8 machines to manage. Not too bad. It was evenly split between 4 windows and 4 Linux machines as well.
Also to be nice, each CCDC team had been provided practice times with the exact same machines and network topology several times before the qualifiers occurred. This means each team should have had plenty of practice time with the network and been well prepared for what was going to happen. The only difference in the practice was that there were no injects, and there was no red team. Fantastic right? Well...
The first issue is team work. Our team doesn't hardly know each other. We pretty much decided who was going to be on the team by whoever showed up to the very last practice session before the qualifier.
As a team, this means we don't know each other's strengths and weaknesses and we really didn't have that much of a choice on who was going to do what. It was pretty much a game of "pick something and do it". However, because I had shown the ability to change passwords on Linux at the command line, I was immediately labeled the Linux guy, and I was stuck with the Ubuntu DNS Server. (Come to find out later, it was running Ubuntu 8.04, which is several years old.)
2. Lack of practice
But we had several weeks of practice and lots of allotted practice time, much more than other teams in fact. How could that translate to a lack of practice?
Simply put, no one cared about the practice. During the practice sessions everyone was kind of poking around on the machines doing their own thing, not really caring about what anyone else was doing, or if they were on the same machine. Rather than it being a practice of "lets make sure we know what's going on with this machine and how to secure it", it became a practice of "Lets see if i can figure out how to change the password from command line".
The practices were completely unorganized, and pretty much no one showed up to them anyways. This was a huge waste of a truly invaluable resource that could have been used to our great advantage and it saddened me to see everyone wasting this time.
It also didn't help things from my end that all of the practices were either 6 pm - 10 pm, or 8 am - 12 pm. 6pm-10pm after I have been dealing with classes all day isn't the greatest (not to mention the 30 minute drive to the school for practice), and from 9-11, I have classes. That left myself with no practice as well. You can say "excuses excuses, you should have been better prepared" and you would be absolutely right, but I had no clue what to expect from this competition.
3. Competition setup
Our team's slot was 3pm - 9pm. We were also supposed to be able to enter the scoring engine for the team up to an hour before the start of the competition. This would also be the system we received our initial login passwords from. Well due to some technical difficulties, at 3 pm we were unable to login to the scoring engine. It seems no team was able to, actually. So they delayed the game by 10 minutes... then by a whole hour.
Fine, 4 pm, and we lose an hour of competition time. Who cares. That is, until 4 pm game, we were all set to go -- and just as we had done about 10 minutes of work on the network, we were all kicked out and our machines reverted to their previous state.
so 4:30 pm we finally had a stable system to run on, and we were able to begin the competition, only now with an hour and a half less time than other competing teams.
So now we move on to a huge issue for our team. It seemed that everyone wanted to be the big leader of the whole group who knew everything and called the shots. Unfortunately, no one was actually confident in their decisions, and they were all pushing out different misinformation's to the whole team. This leads to everyone just kind of doing their own thing in silence while competing, thus leaving the entire network in disarray to begin with.
The next issue is that everyone seemed to think that during our 20 minute window of time in which the red team would hopefully not actively attack us, everyone on the team seemed to have the notion that changing the default passwords right away (and never changing them for the duration of the competition) would be a great idea, and it should absolutely be the first thing we do on every machine.
The issue with this is that these machines have no guarantee of security, and that the red team is allowed to perform passive actions on our machines during this time. What does this mean?
This means that any keyloggers or passive hash dumping mechanisms that were already installed on all of the machines (and I can almost guarantee there was at least keyloggers on every machine) would pick up our passwords and feed them to the red team as we updated them. This makes the fact that we changed the passwords a completely moot point, a tactless countermeasure, and a waste of the passwords that we only ever changed that one time.
5. Lack of knowledge and experience. (And also lack of action)
Each person on our team seemed to think that they were the biggest badass on windows that could secure it so hard that no one could ever break it. That would be great -- if they knew what the hell they were talking about. When you are struggling to remember how to change the password from command line in windows, this should be an indication that you're going to have issues.
I don't think the team performed even a single check for malware on any of the windows machines. They were all so focused on their policy updates that they figured their machines were inherently secure due to their previous password change. This was a huge mistake and is just part of the reason that we had our asses handed to us in a pretty Chinese rice box.
This is the first time most of these people have competed in CCDC, or any cyber defense competition for that matter. This leaves a hole of experience on what to expect at all from dealing with an active red team, or a white team that continues to push forward injects that they refused to answer due to their inexperience. (More on that in the next section.)
Finally, as I mentioned before I had already informed the team that I have worked with one of the red team members, and I know his tool of choice. Metasploit. I told them if they didn't want to be completely owned, they needed to defend against metasploit. Period. I thought this was going well when a member of the team brought in a bit.ly link with a program that checks every 30 seconds or so for the metasploit shell running in memory, and kills it. Awesome right?
Hah, no! Because they didn't run it. They didn't use the tool that they brought. They were too busy updating their machines, and as a result they got completely owned by metasploit. What do I mean exactly by completely owned? This picture should explain that better than I can.
If you're still not sure what you're looking at, let me explain it a bit more. You're looking at a screenshot from aforementioned red team member who loves metasploit. In this screenshot, he has used metasploit to install Linux onto our domain controller system, and overwritten the MBR (Master Boot Record) of the computer. Having overwritten the MBR, all he did after that was reboot our machine, and it booted into Linux instead of windows server 2008. Surprise!
We received several injects throughout the game. This included:
- Create a centralized logging server and configure all of your systems to log to it. Send screenshots as proof.
- We have discovered some machines are using Telnet, please remove it and upgrade to SSH.
- Please create a disaster recovery plan
- Please create a report telling us how we can avoid ending up like Target, and having all of our customer's information stolen.
- Create an NTP server on the network and configure all systems to sync with it.
- Create a DHCP server and configure the network to use it.
There were a few others, but these are what I remember off the top of my head. The ones having to deal with policy and plans, we had two people get to work on it right away. However, during the time that they were creating the requested documents they left their machines completely unattended. This means the red team was free to do what they wanted, and not to mention they never got to fixing any security holes that existed on the system.
All of the injects having to do with "create a ... on your network", NONE of them ever got completed. No one had experience setting up these services and each inject had a time limit of 15 minutes to get them done. This left the team scrambling trying to figure out who was going to install the server on their machine, and then it became finger pointing because no one knew how to install the servers or configure them, let alone set all the computers on our network to point to these services. This of course led to none of them being completed, despite them being 35-50% of our overall score.
The girl handling our PFSense firewall had no idea what she was doing in the first place, but because she knew how to add rules via the web interface, she got the task of setting up the firewall. unfortunately, she was forgetting a few things.
1. This is still a Linux distro. This means you have to secure is just as much as we have to secure any of our other machines, it is not invulnerable to attacks or secure. It is part of the game, and was probably heavily compromised by the red team due to her lack of securing the machine.
2. She had no clue how firewall rules work in Linux. She was having to receive assistance from one of our other team mates on how to properly configure the firewall so that our outward facing services wouldn't be blocked.
This means she was pretty much a waste of space to the team, as she didn't accomplish anything except for hindering the scoring engine from reaching our services, which hurt our scores.
The good stuff
So through all this bad, was there actually any good that was done by our team? Not a whole lot, but I can brag on the team of 3 of us working on Linux machines for the entirety of the competition (Not including pfsense here). We communicated.. ok, not the greatest, but ok. We all had knowledge on how firewalls work and we were all running iptables with very strict rules. We all had some kind of knowledge on locking out the bogus accounts on the machines and overall I feel our Linux machines were a much harder target for the red team to hit than our windows machines.
Take this as a guide of what NOT to do in CCDC. We failed to move on for a variety of reasons, and we need a lot of work if we want to get even close to being able to advance to any reasonable place in the competition. Hopefully we'll get some new people in the next year, and be able to compete and hold our selves up better. It would also be wildly beneficial to learn from our mistakes from this year, but we'll see how that goes.