The seven deadly sins of firewall admins

“If you can’t explain it simply, you don’t understand it well enough,” said Einstein. With this in mind, here’s a very simple analogy that explains the purpose of a firewall, followed by the seven deadly sins of firewall admins the world over.

By CSC’s Eric Pinkerton

Imagine an airport made up of a number of ‘zones’— check in, departure lounges, customs, tarmacs, planes, etc. When we take a flight, we travel from the car park, through the check-in desks, through security, through customs, through the gate and finally on the aircraft. It is the airport’s responsibility to ensure that we go through all of these stages in the correct order and with each zone we enter, we need to pass some checks:

  1. Are we on a no-fly list?
  2. Do we have a valid ticket?
  3. Do we have valid passport?
  4. Have we filled in a customs declaration?
  5. Are we carrying prohibited items?
  6. Do we smell of explosives?
  7. Does our boarding card match the plane we are boarding?

And so on…

Warning! Authority figure with big laptop ahead.

Warning! Authority figure with big laptop ahead.

Network traffic is similar in this respect. When traffic traverses a firewall, it must satisfy policy conditions not dissimilar to the airport security model:

  1. Where are you coming from? (Source)
  2. Where are you going? (Destination)
  3. What are you doing when you get there? (Protocol)
  4. Are you carrying prohibited items? (Malware)
  5. Do you smell of explosives? (IPS)

And so on…

I spent a great many years managing these devices and their policies, undergoing hours of training and having installed, broke, fixed, upgraded, configured, fine tuned, troubleshot and decommissioned more boxes than I care to remember.

In later life, I became a security consultant and quickly realised that there was a dearth of consultants with actual firewall experience and so, unable to hide my past sins (thanks LinkedIn), my penance was spending many hours reviewing other people’s firewall implementations. I did this for so long, I would often burst in to manic laughter at the site of a configuration because an access list had not been applied to an interface, or that the password was ‘2KFQnbNIdI.2KYOU.’

To shorten my penance, I have compiled a list of Seven Deadly Firewall Sins. This is not meant to be a comprehensive list of everything that can possibly be at fault with a firewall implementation, but I will wager dollars to doughnuts that many of these things are evident to some extent on the firewalls that this article has traversed in order to appear in front of you.

  1. Greed Overly permissive rules

In the perfect world we would cherry pick only the traffic we want from the network buffet by creating dainty rules with a single source and destination to a single port. The reality is that the world is imperfect and so often we find ourselves eating with a shovel.

There are of course cases where it’s better to have one single rule for a network rather than 100 specific rules and where the word “Any” is essential for normal operation, ie. a public webserver by definition needs to be available to everyone. Yet in most cases, we are too greedy with our rules, scooping up great swathes of permissions that we didn’t initially intend.

  • Overuse of the word “Any”

Three letters that seem so innocent, yet can result in an exponential orgy of permissiveness. The word “Any” might serve us better were it replaced with “World + Dog” or “The Great Unwashed.” The trifecta of such debauchery is known as an “ANY ANY ANY” Rule, which refers to the basic format of a firewall ruleset, source, destination and port.

Whilst a firewall rule has many more factors, such as time, user, protocol, logging and action, the key points are where the traffic is coming from, where is it destined, and what type of traffic it is.

Use of “IP Any Any Allow” needs to be justified because odds are it’s either an example of laziness — why create 250 firewall rules when you can create one? — or ineptitude.

For firewall admins, the word “Any” is a Class A drug and, like any addiction, it has serious withdrawal symptoms. The more you use them, and the longer you use them, the more difficulty you will have removing them.

  • VPN traffic is never filtered

It’s common practice to apply an access list to a virtual private network that simply permits all traffic to everywhere for a couple of reasons: firstly this traffic is often policed elsewhere, for example when it traverses another interface; secondly, with some firewall vendors, the access list needs to match at both sides of the tunnel so changes become problematic, especially when you are not responsible for the other side of the tunnel (ie. with a VPN to a partner organisation).

Obviously, if you are opening up your network to a partner organisation, you need to ensure that you are not unwittingly giving them far more access than they need, and the assumption that other access lists will police the traffic may be your downfall.

  • Default drop & log

Firewalls by default silently drop all traffic that is not expressly permitted by a rule. Best practice is to pre-empt this with an explicit default drop-and-log rule at the bottom of the rulebase, so that dropped traffic is logged.

This can be problematic for, say, Internet firewalls that log every single portscan and worm rattling the doors of your organisation. By gaining visibility of the sort of traffic that is being dropped at the last hurdle, you can craft ‘noise rules’ to drop them silently at the beginning of the rulebase, thus reducing the amount of work your firewall has to do, and flagging any unusual activity or misconfigurations.

  1. Pride Why DMZs are so poorly understood

People are often too proud to admit that they don’t really understand what a DMZ is. Short for demilitarized zone, in firewall parlance DMZ refers to an intermediate network used to stage services. For example, if I have a webserver that I need to expose to the Internet, I place it in a DMZ. Now I can police the traffic from the Internet to the webserver, from the internal network to the webserver and finally from the webserver to the internal network.

If the server gets compromised, then it is of less use as a stepping stone to an attacker if it has been isolated in a network that has limited access to other networks.

A better name for DMZ would be “Airlock.” Remember the film “Alien”? Perhaps if those crew members exposed to alien life forms had been quarantined in an airlock and observed for unusual symptoms it would have been a much shorter film.

Too often I see servers on the internal network exposed to the Internet without a DMZ, or I see hosts in the DMZ with additional network interfaces on the internal network begging for a “chestburster.”

  1. Gluttony All the logging you can eat

Logging for the firewall is essential for troubleshooting, incident reviews and anomaly detection. The fact that a firewall can log every packet that traverses it and allow an administrator to view the decisions being made is crucial for visibility of your network. It’s the glass window that lets you view what is happening to the quarantined crew member in the airlock.

You really should be trying to consume as much logging as possible, and send your logs to an external server where they can be easily searched. While you’re at it, you should also sync your firewall with an NTP server so the timestamp is accurate.  PRO TIP: You can create “noise rules” to silently drop uninteresting traffic at the top of the rulebase if you need to cut down the signal-to-noise ratio.

  1. Wrath Why filtering in one direction makes me angry

In years gone by, we often viewed the firewall policy as being a one-way deal — I’ll frisk you on the way in to the airport, but not on the way out. There are two reasons why you should be angry about the lack of egress filtering in modern networks:

  • Exfiltration – If an attacker gets in to your network, eventually he is going to want to get stuff out. By not filtering egress traffic, you are making life very easy for them, and you are unlikely to notice what is happening.
  • Command and Control – If a host is infected with modern malware, odds are it will poll a command-and-control server to say, “I am here, now give me some instructions.” You owe it to yourself to block such requests where possible and, by looking at your logs, you may be able to spot any unusual behaviour, such as your mail server checking Twitter.
One way, or another...

                     One way, or another…

  1. Vendor envy

Modern firewalls come with the sort of endless whistles and bells that allude to just how aggressive the firewall market is. You can’t simply buy a firewall anymore; you have to buy an Ultimate Threat Management Gateway, or a Next-Generation Blade Chassis. The truth is that many of these features are more about marketing and sales.

A solution that does everything, rarely does anything brilliantly. It will probably do the job, but in many cases these sorts of additional functionalities were added for no other reason than “our competitors had one, so we needed one.” If you try to use them all, you will end up with something that is completely useless, because if it takes me three hours of interviews, a background check and a cavity search to get through airport security, I am going to miss my flight.

Also, one box to rule them all often equates to one guy to run them all…

Often the firewall guy is the networking guy who was handed a book and a box on a Friday afternoon and told to put it in place by Monday. If you have read this far, you should have an appreciation already that firewalls can be unforgiving, politically charged, poison chalices at the best of times, but if your firewall admin is under pressure, under resourced, under qualified or simply poorly motivated, this will probably be reflected in your security posture.

  1. Lusting for virtualization

We all love virtualization, and the virtualization of firewalls is a fantastic concept that can be a really useful tool in some scenarios. However, those scenarios are limited to sharing management of the firewall across different entities, such as a service provider providing a managed firewall service or an enterprise that wants to have different departments managing their own firewalls.

The problem in almost every other scenario is that we simply add complexity at an order of magnitude for no tangible gain.  At the end of the day we still have a black box with some network cables coming out of it, and that box makes decisions when packets go in, and if and where they should come out.

I have seen virtualization used in many cases for all the wrong reasons such as, “It made the logical design look better,” or, my personal favorite, “We replaced five physical firewalls with one physical device, and it was easier to migrate to five virtual hosts at the time.”

As Bruce Schneier said years ago, “Complexity is the enemy of security,” and unwarranted virtualization adds complexity in spades.

  1. Sloth Why firmware is rarely, if ever, updated

Firewalls are essentially hardened ‘nix boxes and so they have their vulnerabilities like any other system. Whilst I do not recommend that companies update their firewalls on a monthly basis, I commonly see firewalls that have never been updated despite being in place for five or six years.

Have a look at your vendor’s website and look at the known vulnerabilities for your device. If you are relying on functionality that is mentioned, or if the device is more than two years out of date, you should consider upgrading it to the version recommended by your vendor.

         “But I already patched that device for Y2K!”


Bio-picEric Pinkerton is a principal security consultant for CSC’s Emerging Business Group, which focuses on cloud computing, big data analytics, the Internet of Things, social media and mobile computing. At the age of 20, Pinkerton spent a summer working as a fortune teller by day and as a nightclub doorman in the evenings, so you might say that big data and security is in his blood. 20 years on, having gained experience in IT and security across multiple industries, from introduction and recruitment agencies to Media, ISPs and Telcos, Pinkerton’s still trying to predict the future, whilst ducking the punches.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: