Stepping through CodeMash CTF 2019

This past year CodeMash held their annual Capture the Flag competition. With the event wrapping up, I thought it would be a great learning opportunity for everyone to describe each of the challenges and explain the steps I took in order to solve each one.

There are fifteen in total so without further ado, lets begin!

  1. ASCII Art
  2. Busted File
  3. Esoteric Stuff
  4. Espresso
  5. Krafty Kat
  6. Dot Nutcracker
  7. Capture the Falg
  8. Ghost Text
  9. The Parrot
  10. Stacked Up
  11. Unicorn
  12. Danger Zone
  13. Mrs Robot
  14. Cars
  15. On Site Challenge


The first challenge that was unlocked was some ASCII art that spelled out "CODE MASH". It also included this sentence: I love ASCII art. What about you?

 ##99##  #109##  ##49### ####57#      ####  ####  #####  ####### ##   ##
#45  ## ###  97# ##  115 99#          ########## ### ### ###  ## ##   ##
105     45    ## 67   ## ###79##      ##  ##  ## ####### #####   #######
68      ##    69 ##   45 #97####      ##  ##  ## #######   ##### #######
###  49 110  ### ##  116 45#          ##  ##  ## ##   ## ##   ## ##   ##
 #72###  ##52##  #82#### ####68#      ##  ##  ## ##   ## ####### ##   ##

Though if you notice, the word CODE has some numbers scattered throughout.

The problem seems to be hinting pretty hard that this problem is relating to ASCII and ASCII codes. So I tried taking each number in the word CODE, assumed it was an ASCII number, and tried to translate it to a character.


Well, that looks like a flag to me!

Busted File

We are given a zip file ( that contains a single image (image.png). Though when I tried to extract the image, I got an error saying that the archive is corrupt.

Upon trying to repair the archive using WinRAR, I was given an error about a Corrupt header.

I had no idea what a zip header should look like, so I created my own zip file and opened it up into a hex editor.

You can see here that the newly created (and valid) header contains:

50 4B 03 04 14 00 02 00 08

Opening the corrupt file, you'll notice that the header contains:

DE AD BE EF 14 00 00 00 08

Using the hex editor to replace the corrupted header with the expected header allowed me to successfully unzip the file.

The contents of the zip file contained the following image:

This approach of hiding text within an image is called steganography. There are a couple ways that this could have been done.

  1. The flag could be rendered into the image itself. Typical methods of extraction could be to invert the colors, adjust the hue and saturation, etc. Play with the layers and colors to see if the original author put the text into the image itself.
  2. Another approach, and the actual solution to this challenge, is to embed the text into the actual file. If you open up this image with Notepad or a hex editor, towards the beginning of the file you'll see the flag in plain text.

Esoteric Stuff

The third challenge contained a large block of periods, exclamation points, and question marks.


It also included this sentence: It's getting a bit esoteric now. Ookay?!

Note the bolded text.

If you're unfamiliar with the term esoteric it means that something is only understood by a small set of individuals. In the world of programming, it refers to esoteric programming languages. These languages are typically just meant for fun and really aren't intended for production use. My goto example of an esoteric language is LOLCODE.

I also noticed that the Ook in Ookay was bolded. I made the conclusion that Ook must be another esoteric programming language and the given text is source code written in Ook.

I attempted to use an Ook compiler to retrieve the flag, and.. success!


This challenge only provided a single file, Espresso.class

Luckily, class files are something that you can decompile pretty easily, so the first step I took was to take the class file and put it through a Java decompiler which resulted in the following source:

Looking at the source of the class file, I saw that there is a conditional that checks to see if the user's input was equal to localStringBuffer. If it matches, the program will print out: Flag correct, well done!

That seems like a pretty good starting point, right?

locationStringBuffer is a private variable that is defined and then built up in a for loop. After the for loop completes, the variable doesn't change and should theoretically contain the flag.

To get what localStringBuffer would be after running the for loop, I just went ahead and removed all of the extra code and just focused on getting localStringBuffer to be built.

Running this slimmed down version of Espresso.class yields the flag.

Krafty Kat

RSA encryption.. my dear old friend. This was the first challenge with a difficulty of "hard", and honestly took a decent bit longer than the other ones to crack.

This challenge has a flag.enc file and a key.pem file. The flag is obviously in the flag.enc file, and is encrypted. I knew I needed to use the provided public key in order to gain access to the encrypted file. The public key looked like this:

-----END PUBLIC KEY-----

Now, how RSA encryption works could be a post all in itself, so I'll just go over each step and describe whats happening.

The above public key is a base64 encoded string that contains what is called the Modulus in RSA. We can use openssl to extract this result like so:

Here we can see the Modulus and the Exponent (65537). Now, the Modulus is currently in hex, but I need to get this number as an integer.

The first step to do this is to strip away all of the colons


The next step is to convert this into an integer. I used python to cast it, but the method doesn't really matter.


When working with RSA, a really important formula is n = p * q, and the number above (the really big one), is my n value. n is derived from the product of p and q (both of which are prime numbers) resulting in the equation n = p * q. If you know p and q, you can derive the private key.

However, these prime numbers are kept secret and should not be known. It is possible to figure out what they are though if the RSA was implemented poorly (e.g. bad primes were chosen).

So if n is the product of p and q, and both p and q must be prime. How do we figure out what they could possibly be? Brute force is one option, but would take.. a long time. Too much time.

The alternative is to see if any factors of n already exist. I did this through a website called factordb. I plugged in the n value I calculated above and noticed that there are no factors.l

Well.. if n = p * q and p and q must be prime.. but n has no factors.. p must be equal to n and then q must be equal to 1.

At this point we know p, q, e, and n. The last value we need is d or the decryption key. Honestly at this point, d can just be calculated from all of our other values so I just wrote a script in Python to do just that.  

from Crypto.Util.number import inverse
from Crypto.PublicKey import RSA
import gmpy2

#open the provided key file
pub = open("key.pem", "r").read()
pub = RSA.importKey(pub)

#set the values of n and e from the key file
n = int(pub.n)
e = int(pub.e)

#set p and q from factordb
p = 258536048570605626988915097172641057803353415992887757589736645808726151727194085610744190012322229640908553929414833022731988789391928353742297595957524280335918484224592037842886381777837922164216258913020883767452162759055614423529478656392491416399318604362756647059576165428708885769576577464120568116338612287783043168328279883802253392551292712278925133675758884573430117126518137049529478694956180472120065649305066144792948879520988172974401270565515875720597321219813209954571652637983550511681618170361995093036165713586725018263017699754932627926751512891645482818654590419836934782173099306176053375927411
q = 1

#calculate d
d = int(gmpy2.invert(e,n-1))

#use d to create a private key and read the message
key = RSA.construct((n,e,d))
message = key.decrypt(open('flag.enc').read(), '')


This will print the contents of the flag.enc, which contains the flag itself!

Dot Nutcracker

In this challenge, there was an executable file that gave you three attempts in order to type in a password. If you got the password correct, the program would output what the flag was.

Now obviously, the three attempts were just fluff, as you could just start the program again. Regardless, the password (and the flag) have to be inside of the executable somewhere.

To get at the source code of the application, I used decompiler called dotPeek which allowed me to view the source of the executable. I even took it a step further and exported the source to a new project. This allowed me to make modifications to the program itself, as well as do any sort of debugging.

Reading through the source code, I came across the primary if check that compared the users input with the stored password.

      while (num > 0)
        Console.WriteLine(string.Format("You have {0}/3 attempt(s) left.", (object) num));
        Console.Write("Enter passphrase: ");
        string str1 = Console.ReadLine();
        string str2 = CryptoHelper.DecryptStringAES("EAAAAH5ZA4kASLVjLUsYmLK3h74KWmkS4BvBS61BuaD4lnyqdz3AO8/xfGO1atVdci0x1g==");
        if (str1 != null && cryptoHelper.Equals(str2, str1))
          Console.WriteLine(string.Format("Decrypted Flag is: {0}", (object) CryptoHelper.DecryptStringAES("EAAAAOlDKPcRaUj/ITV1q9IHN1bAQyUWxZqVob+G1gpmyoIJIPej1O3T4TWnRUndqp4NnA==", CryptoHelper.GetHashString(str2), str1, str1)));
          Console.WriteLine("Press enter to quit");
        Console.WriteLine("That's wrong.");

Reading the code above, it looks like str2 contains the password. I did a Console.Writeline of str2 and saw that acrackeradaykeepsthedoctoraway is printed out. Success! Found the password.

However, when attempting to use the password, the program crashes with the following message:

I went through the source code some more and noticed that the original if statement that compares the passwords is actually using CryptoHelper's Equals method and it has the following implementation:

public bool Equals(string a, string b)
  if (!b.Equals(a))
    return a.Equals(b + "y");
  return true;

This means that both acrackeradaykeepsthedoctoraway and acrackeradaykeepsthedoctorawa should both be correct passwords. Well, if the former causes the application to crash, maybe the second one wont.

It did not!

Capture the Falg

This challenge had the following text:

This falg is easy to catch, isn't it?

and gave this picture:

If you notice, the name of the challenge is Capture the Falg and not Flag. The a and l are jumbled up. The image is also jumbled up. Another interesting notice is that the word falg has 4 letters, and theres also 4 boxes for each piece of the flag.

To render the image correctly, you'd want the 1st image to stay where it's at. Followed by the image directly below it, followed by the top right image, and lastly keep the last image where it's at.

Applying this same approach the word falg, you would get the following:


To read the word flag you need to start from the top left, move down, and then go back to the top in the next column.

Now, if we were to apply this same pattern to the string that we were given (presumably the flag itself), we get the following:


Reading the text just like before will render the flag.

Ghost Text

For this challenge the description was:

Ghost text is ... invisible!

The challenge also provided a text file with this text:

​In​ ​folk​l​or​e​, ​a​ g​hos​t​ is ​the​ ​s​oul​ or​ s​p​ir​it​ ​o​f a​ dea​d​ pe​r​son​ ​or​ ani​m​al ​that ​ca​n​ a​pp​e​ar t​o t​h​e​ l​ivin​g​.​ I​n​ ​gh​o​st​l​o​re, ​gh​o​st​ d​e​s​cr​ipti​o​ns​ vary​ ​wid​el​y,​ from​ ​inv​is​i​bl​e ​pre​sence​s to l​if​el​ike​ vi​si​ons​.

Now, given the challenge description, I assumed there was something hidden inside of the text file that was given. So I opened it with my trusty HexEditor.

You'll notice that there's a lot of jumbled characters intertwined between the text itself, characters we could not see before. These characters are known as zero width characters and are a common approach to hiding messages in text.

To more easily digest how this message really looks, we can use a unicode analyzer. I've provided the link below.

Using this website, we can see every single character that is contained within the string. You'll notice that there are "ZERO WIDTH SPACE" characters throughout the text. This is important!

This approach to hiding messages in text generally involves the use of binary. Every approach to encryption is going to be different, but most messages end up as some sort of binary representation. Maybe each zero width character is a 1 and each real space is a 0. These 1's and 0's will form a binary string, which can then be translated into readable text.

It's also very common for CTF events to use the same flag structure. In this one, all of the flags (so far) have started with cm19.

With this in mind, I wanted to see if I could extract cm19 in binary from the zero width spaces.

cm19 in binary is: 01100011 01101101 00110001 00111001

If you reference the unicode analyzer page linked previously, you may see a pattern here.

The word "in" is followed by a zero width space.

The single space afterwards is followed by a zero width space.

The word "folklore" has a zero width space after "folk"

If we assume that the zero width spaces are some sort of terminator, it actually starts to fit our assumption of the cm19 binary string.

In would be 01

A single space would be 1

Folk would be 0001

.. and so on. If we combine the above we get 0110001 which is the start of our cm19 binary string!

Now I had no desire to go through this entire sentence by hand and finish out the sequence, so I wrote a quick GOLANG program to do it for me.

package main

import "strings"

func main() {
	input := "​In​ ​folk​l​or​e​, ​a​ g​hos​t​ is ​the​ ​s​oul​ or​ s​p​ir​it​ ​o​f a​ dea​d​ pe​r​son​ ​or​ ani​m​al ​that ​ca​n​ a​pp​e​ar t​o t​h​e​ l​ivin​g​.​ I​n​ ​gh​o​st​l​o​re, ​gh​o​st​ d​e​s​cr​ipti​o​ns​ vary​ ​wid​el​y,​ from​ ​inv​is​i​bl​e ​pre​sence​s to l​if​el​ike​ vi​si​ons​.​"

	result := ""

	for _, v := range input {

		if v == 8203 {
			result = strings.TrimSuffix(result, "0")
			result += "1"

		} else {
			result += "0"



Running the application gives the following binary string:


Which can then be converted to the flag itself using an ASCII to hex converter.

The Parrot

This challenge had a single pdf file called parrot.pdf. Though when attempting to open the pdf to view it's contents, I was presented with a password dialog.

This is a password protected PDF and I needed a way to get the password so that I can open the PDF in order to see what's inside.

For this, I used a tool called PDFcrack and a collection of commonly used passwords called rockyou.txt. Putting the two together, I got the following result:

You'll notice on the very last line, the program managed to find the password "cracker".

Note: I later found out that if you open the pdf in a text editor and look at the Authors, it actually spells out the password if you look closely:

<< /Authors: charlie romeo alfa charlie kilo echo romeo >>

Moving on!

When the PDF was opened, it contained the following image:

In order to get the image out of the PDF, I used another tool called pdfimages. At this point, I already knew the password, so I ran pdfimages with the password 'cracker'.

pdfimages parrot.pdf -upw cracker .

This command extracted two images in .ppm format. Opening the first image and.. tada!

Stacked Up

The challenge description stated that there was a vulnerable service running in the cloud and that it could be exploited in order to get the flag. A copy of just the binary was given freely for download.

Running the program locally, it just looked like it echo'd back whatever I typed in.

Doesn't seem like a very useful program, does it? I checked out what the hint had to say about this challenge. The hint plainly said:

HINT: What happens if you input a veeeeeeeeeery long string?

Fair enough, I tried entering in a long string.

I didn't think that was incredibly surprising. With input that large I probably caused a buffer overflow.

However, at this point, I went down a pretty large rabbit hole. I spent a lot of time focus'd on the core dump, convinced that there was going to be something valuable inside of the stack trace when the program errored. Ultimately, I did not really get anywhere going down that path.

The next thing I wanted to try, was just to decompile this binary and see if I could make any sense of what was going on inside.

After decompiling the binary using IDA, I noticed a couple really interesting things.

First, the main method had no jumps. There weren't any if statements and all the main() method did was get user input and print it out. I did not expect that at all.

Second, I noticed that there was a method called flag()

This method looked almost identical to the main() method with the exception that this method tried to open a file called flag.txt, and then print its contents.

I knew at this point the goal was going to be to call the flag() method. There must exist a flag.txt file on the remote server where the binary is running, and calling the flag method would read that file and print it out.

In order to do this, the return address of the main() method would need to be overwritten.

When a function is called (in this case main), the return address is stored on the stack and is just above the base pointer. If I overwrite the return address of the main method to point to the flag() method instead of the default return address, the program should call the flag method.

To accomplish this I needed a couple of things.

First, since the return address is right after the base pointer, I needed to know exactly how many characters were required to overflow the program before I started to spill over into the return address.

You can see here in this photo that the return address looks like

0xff021b92 0x00007fff (the last two hex values on the top row)

When I used a lot of A's in order to get the program to overflow, I noticed that the return address starts to get overwritten at the 1033rd character.

See the 41's that are bleeding over into the 3rd column?

Ok, good progress! The 1033rd character is the start of the return address, so we need to fill the buffer with 1032 characters of junk, and then tack on the return address of the flag() method.

Using IDA again, I got the address of the flag() method just by clicking on the starting line of the method (0x400676):

At this point, I have everything needed in order to call the flag() method. To create the input for the application, I just used python to generate all of the junk characters as well as append the address of the flag method to the end of the string

python -c 'print "A"*1032 + p32(0x400676)'

Connecting to the remote service, and using the generated input overwrites the return address, calls the flag() method, and prints the flag!


This challenge was relatively straight forward. A Unicorn.svg file was given, and that's about it.

Typically in CTFs, if you're given a svg, they expect you to manipulate the svg in some way in order to see the flag.

In the case of our cute little Unicorn, I just opened the file into an SVG editor and deleted the Unicorn. The flag was hidden behind it.

Danger Zone

This challenge had the following description:

Can you enter the ****, and find the flag?

Service listening on:**5553**

Note: this is **NOT** about the web site!

Like most CTFs, a lot of clues on how to solve the problem are given in the name and the description. For this challenge, it talks about and a service that is running on port 5553. The hint itself also really helpful:

HINT: Find out what service is listening on the port. nmap could be helpful for this kind of fingerprinting.

Sounds like a good idea to start off this challenge, so I did just that. Running nmap for port 5553 tells us that the service that is running is a domain service with a version of ISC BIND 9.9.5

With this information, we now know that port 5553 is running a domain service, which means its most likely running a custom DNS service.

For DNS problems, dig can be incredibly helpful. Since I now knew that the service is a DNS service (and is probably vulnerable), I used dig to look up some more information about it while trying to access

I was able to successfully query using the DNS service, and on top of that, exists as a record in the zone file for the DNS! Which ultimately tells me that I should be able to perform a zone transfer.

After performing the zone transfer, the zone file is printed to the screen, and I can see the flag that was put inside.

Mrs Robot

For this challenge, a webpage that prompted us for a secret word was given.

Viewing the source didn't really bring about anything of interest. Maybe the fact that each word that was typed in just redirected to a webpage that was an md5 hash of the input:

This led me to believe that the flag was an HTML file on the webserver, and the name of the HTML file was the secret word in an md5 hash.

I could use something like DirBuster to find the file for me, hashing common words and checking if the file exists, but I might be there for awhile.

Luckily, the site did have a robots.txt file that contained the following:

User-agent: *
Disallow: /481065fd79b253104aeab5ca5c717cd5.html

Well that looks promising!

Upon landing on the page in the robots.txt file, I was presented with the text:

If you're not already familiar with the Robot Interaction Language, a simple google search for both of these terms will bring up a table with all of the translations necessary to get this text into English.

Leveraging the table I got the secret word, winter, and thus the flag itself.


For this challenge, a website with a search box and a login page was given. The goal was to login to the page using the admin account.

I was pretty sure this was going to be a SQL injection challenge, and the hint confirmed that:

HINT: Can you find a way to make the search return more data than expected? What you might get, is not the password yet.

I started off by throwing some quotes at the search box as that's a pretty common approach to get some more information about the version of SQL and the query running behind the scenes. Using the following query:


The result that came back was:

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%'' at line 1

I didn't even use a %, and the behavior of the page did seem to lend itself to being a LIKE query. Inspecting the error more closely, I can see that the trailing SQL query is %'. So, my query of a' must have terminated the query, and %' is what was left over.

With this in mind I felt pretty confident the query was structured as so:

SELECT .. FROM .. WHERE .. LIKE '%' + [input] + '%'

The next step was to start filling in the blanks in order to get some more information about the underlying database.

Using the following query, we can get a dump of all of the underlying tables and their columns

a' union select table_name, column_name FROM information_schema.columns#

The a' is to finish the SELECT statement so that it's valid, and then I can UNION another SELECT statement to pull back information about the schema. Since this is a MySQL database, the pound symbol (#) is used to comment out the rest of the query. It would look something like the following:

SELECT .. FROM .. WHERE .. LIKE '%a' UNION SELECT table_name, column_name FROM information_schema.columns#%'

The hash comments out the trailing %' from the real LIKE query

The results of this query looked like this:

At this point, we have all of the information we need to pull back the data that we're looking for. The users and the password.

Doing another injection using the new tables and columns gives us this query:

a' UNION select username,password_hash from Users#

Resulting in the following records:

There it is! The hash of the admin password.

I was pretty confident the hash was going to be an MD5 hash, so I knew I needed a rainbow table and hope that the hash had been cracked in the past.

I ended up using an online Reverse Hash Calculator which did have the hash already stored.

Using the username of admin and the password of dodge123, revealed the flag.

On Site Challenge

The description for the on site challenge was in the form of a Haiku:

Games are played all night.
This one online, but there, board.
Code is history.

Reading the poem, I saw "but there, board". To me, there is hinting at the location of the flag. It is an on site challenge after all. On site, there is a place where board games are played, so that seemed like a logical conclusion to me.

Upon entering the room, I saw multiple identical QR codes on the wall:

Using a QR code reader, I saw that the QR code is actually a link to:

Looking at the remainder of the Haiku "Code is history" I eventually figure out that its referring to the "View history" of the wiki which contains a strange entry that was added and then immediately removed about a minute later.

Viewing the revision in which lines were added, I saw what appears to be a Base64 encoded string. Putting that string into a Base64 decoder reveals the flag.

CodeMash 2019

What better way to start off the new year than with a conference? This was my first time attending CodeMash, and it was well worth the trip.

CodeMash is an annual conference that is held at the Kalahari Resorts in Sandusky, Ohio. While the conference itself is only two days long, they do offer what they call "Precompilers" which you can just think of as pre-conference workshops.

First Impressions

It seems like many conferences may focus on specific frameworks such as .NET, or languages like JavaScript, CodeMash definitely makes an effort to be diverse in the content that they provide. I attended sessions discussing security, people skills, SQL, and the list goes on.

While not directly applicable to myself, the other thing I noticed is that CodeMash is incredibly family friendly. Not only was the conference held at an indoor waterpark, it's actually two conferences in one. The second name being KidzMash.

KidzMash is another conference that runs parallel to CodeMash, and provides sessions more geared towards.. you guessed it, kids! Topics included robotics, video game development, and even 3D printing. It made the whole experience feel very wholesome and welcoming.

To add onto that, I felt CodeMash had the most community involvement that I've ever seen in a conference. The conference had its own Slack channel where conference goers could ask questions as the conference was going on. You had people asking if anyone wanted to get together to play games, meetup to discuss another topic, etc. You never really felt alone or out of the loop.

New Faces

It was refreshing to see a lot of newer faces giving presenations in each of the talks. Along with being family focus'd, CodeMash almost seemed to be driven to encourage those who may not have spoken before, to get up on stage and give it a shot. A select number of the speakers at the conference have never spoken before, and the conference explicitly calls out that they like to have those individuals come and present.

For those who it is their first time, CodeMash even pairs them up with a couple mentors that can give feedback on their presentation and pass on their expertise from having been there, and done that.

Not only did you get to see some of the bigger names in the tech world (e.g. Jon Skeet!), you got to introduce yourself to some newer members of the community. These individuals bring a fresh perspective and have their own opinions which I feel disrupts the echo chamber that we may find ourselves lost in from time to time.

Capture the Flag

Speaking of new faces, I met quite a few individuals who were also a part of the CodeMash CTF. For those unfamiliar, CTF (Capture the Flag) is an event where you are given a scenario in which you must obtain the "flag" in order to score points. Scenarios may include: a zip file that has been intentionally corrupted and you must figure out how to get inside to unzip the flag, an image that has a hidden message somewhere inside of it, or even a website that has a web page somewhere on the server and you must do everything possible to find it.

It was one of the best experiences I had at CodeMash, and the people involved were equally enjoyable to converse with.

The organizer, Bill Sempf, also runs the Columbus OWASP group and is an expert lock picker! Luckily one of the "open spaces" at the conference was an introduction to lock picking, so I was able to get some hands on experience picking locks which was a great time.

But I digress.

For the CTF, there's plans for next year to have those who were specifically passionate about the it (this guy) to create our own problems, rather than using problems created by someone else. I'm overly excited to be make some CTF problems coming up here in the near future.

Wrapping Up

Taking everything into consideration, CodeMash is definitely worth it. At only 350$ a ticket, you get two full days of sessions, food, networking opportunities, and an after party. Contrast that with the typical pricing of conferences, it's worth at least giving it a try.

Great content, great speakers, in a great location. I will definitely be going back for CodeMash 2020.

Understanding Azure Deployment Slots

Azure deployment slots are a fantastic feature of Azure App Services. They allow developers to have multiple versions of their application running at the same time with the added bonus of being able to re-route traffic between each instance at the press of a button. They can, however, generate a lot of confusion if you don't fully understand how they work.

So what exactly are Azure Deployment Slots?

Let's assume we have a web app running on Azure App Services. We'll call it When you create a new web application, Azure creates a deployment slot for you, typically called production. However, it's possible to add additional deployment slots.

Put simply, a deployment slot is another web application. It has it's own URL, it could have its own database, connection strings, etc. It can be configured any way you see fit. But why would you want to have two web applications? The most common reason is so that we can have a place to deploy new features to, rather than going straight to production, which can be a little risky.

To accomplish this, we would create a deployment slot called staging. The staging slot is where you would deploy all new changes to your application to validate that everything is working before the changes actually go live to your users. Think of it like a test environment. A test environment that's really easy to spin up and manage. Let's create a deployment slot called staging and have it be accessible via

Creating a Deployment Slot

Creating a deployment slot is pretty simple. Open your Azure portal and navigate to your Web App resource. Once there, you should be able to see a menu item labeled Deployment slots.

Clicking on the Add Slot button opens the space to add a new deployment slot. Here you can specify the name of the slot (I used staging) and if you want to copy any pre-existing configurations (such as your current production deployment slot). Press OK and you're all set!

When the deployment slot is first created, it is empty. So you'll want to deploy your latest and greatest changes to your staging environment (or just re-deploy your current production version to get something up and running). Deploying to your new slot is really no different than deploying to your production slot. Using the same tooling, just select the staging slot, rather than the production slot.

At this point, we have two instances of our web application running. One is our production instance, supporting all of our production traffic and another staging environment that we are using for testing the latest and greatest features. When you are satisfied with your tests, you will need to swap the staging and production slots so that your users can benefit from your new features.

Swapping Deployment Slots

Swapping deployment slots routes traffic from the source slot to the target slot. In our case, we want to swap the staging and production slots. This will route our users to the staging app (where our newest changes are) when they navigate to

While that is the easiest way to describe what is happening, there is a lot that is going on behind the scenes that is useful to know.

When Swapping... Source and Target Matter

When performing a swap, you are presented with a source and a target. This may be a little confusing at first. Why would it matter? A swap is just flipping two things! While the end result will be the same, the key takeaway is that up-time is not guaranteed for the source slot.

This is because when you perform a swap, this is what is really happening:

  • First, the staging slot needs to go through some setting changes. This causes the staging site to restart, which is fine.
  • Next, the staging site gets warmed up, by having a request sent to its root path (i.e. '/'), and waiting for it to complete.
  • Now that the staging site is warm, it gets swapped into production. There is no down time, since it goes straight from one warm site to another one.
  • Finally, the site that used to be production (and is now staging) also needs to get some settings applied, causing it to restart. Again, this is fine since it happens to the staging site.

This process guarantees that your destination slot will always been warm and your users won't experience any downtime when the swap happens. Users may experience performance issues when navigating to the staging environment, but this is acceptable as it's not really a production environment.

When Swapping... Settings Are Swapped Too

Spoiler Alert: Not all settings are swapped. It is important to remember that when performing a swap, the settings of a deployment slot are also swapped.. but not all of them.

Some settings make sense to keep specific to the slot, these are called slot settings and can be configured in the Azure portal.

When a setting has been flagged as a slot setting it will not be applied to the target site. This is useful for settings such as connection strings. Maybe you want to have a dedicated database for your staging environment so you create a slot setting to hold a connection string that connects to a database specifically set up for your staging environment.

Some settings will be swapped during the swap process. There are settings that are not marked as a "slot setting" under the Application Settings section. This can be useful for a couple of reasons, one of which could be to introduce a new slot setting.

If at first we apply the setting to staging, perform the swap, and then apply the setting to the staging environment again (the old production app), it's possible to add a new settings without incurring an application restart on the production application.

The Azure portal even tells you which settings will be applied before you perform the swap operation as shown below.

Unfortunately the preview does not list all changes that will be applied to the deployment slot. I learned this the hard way.

When Swapping... The Code Does Not Move

This was something I wasn't always quite sure about until I dug into it a little more and ran some of my own experiments. When you deploy changes to a deployment slot, that is where the changes will forever reside until you deploy over them. Consider the following scenario:

Version 1 of your application is deployed to your production deployment slot.

Version 2 of your application is deployed to your staging deployment slot.

As we learned previously, each deployment slot is its own web application. When a swap is performed, Azure swaps the virtual IP address of the source and destination and applies the appropriate settings. The code stays with their respective web applications, so the staging web app effectively becomes the production web app, and the production web app becomes the staging web app.

Put another way, imagine having two boxes. One box has black licorice it in labeled "production", and the other box has KitKats inside of it labeled "staging".

Note: To get this analogy right, you just need to agree that KitKats are the superior candy.

Your customers are currently going to the black licorice box, but you realize it's time to give them an upgrade. So you swap the location of the boxes. You also swap the labels on the boxes. This puts the "production" label on the KitKat box and the "staging" label on the black licorice box. Directing your customers to the box of delicious KitKats. They obviously rejoice.

Admittedly, it's sort of a silly example, but I hope it clears up the fact that when you perform a swap, we aren't picking up whats inside the box and moving them to a different box. We're simply relabeling the boxes themselves.

Rolling Back Changes

If the ability to be able to test your changes before going live isn't enough of an incentive to begin leveraging deployment slots, the ability to roll back your changes at the press of a button should be enough to convince you.

After performing a swap, our users are now hitting the latest version of our application. If for some reason we missed something and we start noticing errors, all we have to do is swap again to put the system back into its previous state.

There's no need to open up Git, revert the commit, and re-deploy the change. We don't need to deploy anything at all! It's just a matter of routing our users back to the site that was working for them previously.

Testing in Production

There's also this nifty little feature that we can leverage called Testing in Production. Testing in Production is essentially Azure's implementation of a canary test. If you're unfamiliar with the term, it stems from the mining days where miners would bring a canary down with them into the mine. If the canary died, they'd know something was wrong with the air quality, warning them to leave the mine as soon as possible.

We do canary testing by routing a small subset of users to our new feature. Continuing with the production and staging examples, what we do is take 5% of all traffic to our website and actually have them go to our staging environment with the remaining 95% continuing to hit our production environment. If we don't notice anything wrong, we can bump the 5% up to 10% or even 20%, until we've reached 100%. This way if anything were to go wrong, we've mitigated the number of users impacted by a bad change.

If you're interested in trying this feature out, it is relatively simple to get going. Simply click on the Testing in Production menu item from within your App Service.

This will allow you to set the percentage of traffic that you want to see going to your staging slot (5% as shown in the figure) and production slot. That's all there is to it!

Wrapping Up

Deployment slots are incredibly easy to use and offer a wide range of features that make them hard to pass up. If you're hosting your application in Azure, definitely consider them for your current and/or next project!

The Importance of Naming

A story all too real...

We've all seen the joke, time and time again --

"There are only two hard things in Computer Science: off by one errors, cache invalidation, and naming things."

It's true. Naming things can be really hard. There seems to be all of these hidden rules around how we can name our properties and classes.

You: I'm just going to introduce this helper class, FileHelper.cs
The World: No! You can't use Helper in your class! That's just begging to violate the single responsibility principle.

But naming things correctly can save yourself and others a lot of time. I'll share with you an example from earlier this week. Consider this command:

webdriver-manager start

Assuming you had no knowledge of what this command did, I bet you have some guesses. Maybe you would have only one guess, and I'd honestly be Okay with that. You're probably thinking:

Well.. it starts the web driver manager..?

And you'd be right. Almost.

Unfortunately, webdriver-manager start also performs an update. Although the only way you'd be able to figure this out is if you read the documentation (which honestly seems a little buried to me) or if you ran into the same issue I ran into this week.

While not incredibly relevant to the story, if you want to learn more about what webdriver-manager is, you can read the projects npm page or the GitHub repository. The TL;DR is that it is a means to manage E2E tests for Angular projects.

It goes a little something like this...

For most of the things we develop at my company, we put them into Docker containers. This includes our end-to-end (E2E) tests. The Dockerfile for our tests is pretty straightforward, and honestly as to not add too many unnecessary details, the only thing that really matters is that we have the following command in the Dockerfile:

RUN webdriver-manager update

When the image is built, webdriver-manager update will be run, which will install the latest version of the webdriver manager. Docker images are immutable. That is, when they are created, they do not change. This means that the Docker image will be created with the latest version of the web-driver manager.

Now, to start the web-driver manager, we need to run the infamous webdriver-manager start command from within the Docker container.

Though depending on when you created your Docker image and when you started running your container, you're going to get one of two scenarios:

  1. The container will start up just fine and run your tests as you expect.
  2. The container will error when trying to run the webdriver-manager.

This is due to the fact that, unfortunately, webdriver-manager start not only starts, but attempts to start the latest version. Regardless of what version is installed. So it is possible that a new version has been released, and the Docker image is no longer relevant.

Luckily the solution isn't too bad. We just need to update the Dockerfile to update to a specific version. This forces our Docker image to always have version 3.0.0 installed.

RUN webdriver-manager update --versions.standalone 3.0.0

We then need to change the webdriver-manager start command to also include the same parameter:

webdriver-manager start --versions.standalone 3.0.0

Which in turn, forces the webdriver manager to start a specific version.

A simple solution, but a problem that took a decent time to figure out what was going wrong. Never would I have imagined that start did more than just that. Had the command been called startupdate or just left as start with an optional update parameter, the problem would've been much more apparent.

The biggest takeaway from all of this is that your naming should say exactly what it is doing and..

Have no side effects

Side effects are lies. Your function promises to do one thing, but it also does other hidden things. Sometimes it will have unexpected behavior. They are devious and damaging mistruths that often result in strange temporal couplings and order dependencies.              – Miguel Loureiro

Your code should do what it says it does, and nothing more.

Conducting a Good Code Review

As developers, we perform a lot of code reviews (or at least we should be). They are a great way to not only ensure that the code we intend to deploy to production won't have a negative impact on the system, but also a time to learn and better ourselves.

Unfortunately, I have noticed a trend, especially at work. We tend spend the majority of our cycles verifying whether or not the code conforms to standards. In other words, our first instinct is to scan the code for incorrectly placed braces, improperly named variables, etc. Rather than review the actual quality of the code.

There's a lot of great resources available out there on this subject, so I don't want to regurgitate what's already been published, but I did want to highlight a couple points that this article speaks to.

What a code review is:

  • An opportunity to highlight what is being done well. Reaffirm that the developer is taking the correct approach to solving the problem.
  • An opportunity to highlight what can be done better. Offer different solutions and approaches and use it as a chance to improve one's skill set.

What a code review is not:

  • A witch hunt, a time to point out every little fault. Code reviews are a great opportunity to learn. Developers should be eager to have their code reviewed, not shy away from it for fear of belittlement (even if unintended).

How to conduct a code review:

First and foremost, understand the subject matter. Really familiarize yourself with the problem that the reviewee is trying to solve. Go to the developer's office and immerse yourself in the problem. You can't offer solution suggestions if you don't even know what the problem is.

Focus on how the problem was solved. Not how it's formatted. Think big picture. Is the change the right approach to solve the problem in the grand scheme of things?

Code reviews that just point out every little standard violation aren't beneficial to anyone. Use code reviews as an opportunity to knowledge share, and encourage learning. Not as a platform to demonstrate how well you can enforce formatting guidelines.

I would also encourage you to seek out reviews from other developers, especially if you're even the slightest bit unsure if your solution makes sense. Just because you have the deploy permission, doesn't mean you're stuck in a silo. Go forth and get a second opinion.

If you're looking for additional information, here's a blog series that I completely agree with: