Now, the Netscaler has built in health checking for many different types of backend service protocols: HTTP, FTP, DNS, and more. However, one protocol not included is NTP. If you happen to be load balancing a farm of NTP servers, and want to ensure you’re not routing to a hosed NTP daemon (even if it’s ping-able), it would be nice to be able to health check the service [and take it out of rotation if it's not responding].

Luckily, the Netscaler provides (with documentation) a very flexible way to write your own custom health checks. Using the system, you can test whatever you want, in any way you want, with just a little Perl. Even better, the custom health check subsystem provides a few useful things to simplify the whole process, including:

• passing in the service IP, and port, as arguments to your script
• passing in a custom argument string (key value pairs, user/pass info, whatever)
• handling the timeout for you from a higher level; if your check doesn’t respond within -respTimeout seconds, it will be considered a failure.

Now, for just the basics, this monitor will actually be pretty easy to put together. We are given the IP/port to send an NTP request to, and we don’t need to worry about a timeout, because the underlying subsystem will handle that for us. All we really need to do is send a request and [hope] for a response!

That being said, we’ll need to do a couple things to get to that point. These things are:

1. Load the health check subsystem module.
2. Load the Socket module, and create a socket to use to send the request [using the parameters passed in as args].

From that point, we can build and send our request. Since we don’t need to worry about handling a timeout, we can just sit there waiting for a response. First up:

use IO::Socket;
use Netscaler::KAS;

We’ll need both of these modules – one will tap into the health check subsystem, and the other is so we can send our UDP-based request.

Next is our function, which we’ll just call ntp_probe. It assigns a few variables from $_[0] (the service IP) and $_[1] (the service port) as arguments, which get passed in from work within probe().

So, the magic-fu to this is that, later on, we will pass our ntp_probe function as a coderef to the KAS probe() command. That will ultimately result in our function being called with the arguments passed in; it is similar to this:

$custom_function->($host,$port,$args);

If you look at the KAS code (and since it’s perl, you can), you can see the exact line; this just gives you an idea of what’s going on.

In any case, after that, we’ll create a new UDP socket, send the NTP request message down it, and either:

• return 1 if something wasn’t defined (ip, port, or sock) – failure!
• return 0 if we got a response – success!

Again, we don’t need to worry about timeouts, since the subsystem will handle that for us. So, here’s the function in it’s entirety:

sub ntp_probe {

    my $host    = $_[0];
    my $port    = $_[1];
    my $req     = "\010"."\0"x47;

    if(!$host || !$port) {
        return(1,"Host or port not specified.");
    }

    my $sock = IO::Socket::INET->new(
        Proto     => "udp",
        PeerAddr  => $host,
        PeerPort  => $port,
    );

   if($sock) {
        $sock->send($req);
        $sock->recv($_,1);
        return 0;
    }

    return(1,"Could not create socket");

}

Not bad, was it? Now that we have loaded our modules, and defined a function, the next step is to call the probe() method, with a coderef to our new function, to tell KAS what function to use for probing:

probe(\&ntp_probe);

And you’re off to the races! Just don’t forget to start your script with #!/usr/bin/perl, and it should be good to go

Now, you might be asking “how do I actually configure the Netscaler to use this?” That’s also just as easy. Let’s pretend our new script is called customNTP.pl. Just plop it into the /nsconfig/monitors/ directory, and do something like this from the CLI:

add lb monitor "custom-ntp-mon" USER -scriptName customNTP.pl
   -scriptArgs 1 -dispatcherIP 127.0.0.1 -dispatcherPort 3013
   -interval 20 -respTimeout 3

With that line, you will now have a working NTP healthcheck (20 second intervals, 3 second timeout). Just bind it to your service(s) that need it and enjoy!. Note that the type of monitor is USER; this is the type used for custom health checks.

Well, that’s all for this evening. There are bits and pieces of info like this around the web, on the Citrix AppExpert site, etc. And, even though it wasn’t too difficult to figure out, hopefully this will help someone out some day.

Take care!

At my new employer, I put something together where the collection subsystem won’t refer to the previous total for deltas: it just takes whatever you give it and relays that to the metrics backend. Most applications provide you with a request/sec metric these days, so I figured I’d just use that. Instead of doing deltas on TotalAccesses, I started just relaying ReqPerSec instead. Simple enough, right? For those of you not familiar, here’s an example of server-status output:

Total Accesses: 400
Total kBytes: 800
CPULoad: 5
Uptime: 12345
ReqPerSec: 20
BytesPerSec: 20000
BytesPerReq: 1024
BusyWorkers: 15
IdleWorkers: 5

Well, amidst my metrics collection testing for the new system, I stumbled upon an interesting detail: I would be nailing the Apache instance with a flood of requests using my friend ab (aka: apachebench). However, I immediately noticed ReqPerSec wasn’t accurate at all. For example, I would be averaging a test of 300-400 req/s, but ReqPerSec would be displaying 21 req/s. It was as if I wasn’t sending it requests.

That being said, the TotalAccesses metric was reflecting the requests I was sending, so I knew the web server was receiving them. Interesting.

It got me thinking: “I wonder how Apache determines ReqPerSec?” Time for some source code action! Sure enough, here’s what I found:

ap_rprintf(r, "ReqPerSec: %g\n",
    (float) count / (float) up_time);

Interesting! All Apache does to calculate ReqPerSec is take the total number of requests throughout the life of the parent (count) and divide that by the total uptime in seconds (up_time). So, really, ReqPerSec is just an *average* requests per second over the parent’s lifetime. That’s NOT what we want. Maybe it’s useful in some scenarios, but when you need a live metric to let you know what your server is ACTUALLY doing, this isn’t the one.

If you want an accurate request rate, your best bet is going to be to collect Total Accesses and do deltas off of the previous collection. I’m glad that’s how we did it before!

To a point, this goes into the expense of metrics. For Apache, Total Accesses is just a simple counter. It returns an accurate value live, on-the-fly, and makes it very easy for the sys admin to collect and process. Very little work is done on it’s end. Even the current implementation of ReqPerSec has very little cost: it just does some simple division on-demand, using counters it already tracks anyway.

To keep an actual average request rate, there would be some more resource involved. It would need to set a timer (ie: every 60 seconds), have a callback for the timer (to actually handle the calculation), and store counters in memory to calculate deltas. However, in reality, this will really have a very minimal impact on server performance and system load (a few extra CPU hits every minute, storing two little 64-bit counters, etc).

So what did I end up doing? I hacked up my own mod_status using shared memory!

Using the existing mod_status code, I pulled some code from a shared memory module example and smashed them together to calculate an accurate ReqPerSec metric on the fly, returned when you request server-status.

How does it work?

When you request server-status, it will go into shared memory, pull out the total accesses and uptime from the previous server-status request, do deltas on the current total accesses and uptime, then return the rate. It will then put the current total accesses and uptime back into shared memory for the next poll (base is our shared memory baseline).

ap_rprintf(r, "ReqPerSec: %g\n", (
    ( (float)count - (float)base->prev_req_count ) /
    ( (float)up_time - (float)base->prev_uptime ) )
);

There is of course all of the shared memory code, but the end result is the calculation above. We poll every 60 seconds, so we incur a small cost every minute, but, like I mentioned before, it’s not noticeable, and it gives us the view we need into the environment.

Some of you may also wonder what happens to the first request, since there isn’t any data in our shared memory space yet. For now, it will be inaccurate, but only for the first request in the lifetime of the parent, which I can deal with until I make it better.

Well, that’s all for this one. It was an interesting discovery in the world of Apache metrics collection, and an interesting adventure to hack together the shared memory module. I’m sure there are other scenarios out there like this one; if you come across one, feel free to drop a comment!

I’m off to the Apple Store now- it’s iPad launch day

In any case, one of our metrics publishing systems is wrapped by a few different object classes I put together in Perl (follows a certain protocol created by another group, etc). Of the many data types supported by our metrics receivers, we can do things like 32-bit unsigned integers, single precision (32-bit) floating points, and a few others.

Now, when I would publish the 32-bit uint’s, it would work great- no issues. The receivers would expect a byte stream, so I would use pack() to send my values as binary. For example:

$val=29;
print $sock pack('N',$val);

Easy, whatever. However, when I would send values as a single precision float, it would totally be hosed on the other end. For example:

$val="30.0";
print $sock pack('f',$val);

On the other end, they would get some massive negative number. I don’t deal with this too much, things usually just work. Thankfully, my mind was jogged in regards to standards- there are lots of standards around byte order for integers (16 bit, 32 bit, 64 bit), but not as much for float; one architecture may use big-endian, and another architecture might use little-endian.

In my searches, I found that Intel stores floating points in little-endian byte order, and Sparc stores them in big-endian byte order. Sure enough, I was trying to publish from an Intel-based BSD host to a Sparc based Solaris host. Aha!

Luckily, pack() can *also* handle this for us! With a simple carat added to the templates section of pack(), I can easily pack the value in big-endian byte order and resolve our issue. I do that like so:

$val="30.0";
print $sock pack('f>',$val);

And, ta da! Worked like a charm. Was it difficult to get it working? No. But, it was fun to have to deal with something like this. On any given day, you don’t really worry about endian-ness; things just work. This was one of those days where I actually had to use the old noggin, and that was refreshing.

However, what happens if the app is REALLY in trouble? Your connection queue could very easily grow into thousands of piled up requests all waiting to be serviced. At some point, you’re going to have to start dropping requests from the queue: it will get so large, that even after the app comes back, you may never recover from the pile of requests (or if you do, it might be 60 seconds worth, at which point your user will probably have given up anyway).

So, here we go.

First: how many requests per second (max_requests) can your application slice handle? In this case, your application slice will be however many services are being load balanced. Depending on the max_requests, that will allow us to know how much headroom will be available for a given slice utilization; we will use that headroom to determine how many requests over and above the current utilization are available to help dequeue.

Second: What is our average utilization? If on a given day we are 60% utilized, then we should in theory have 40% capacity available to help dequeue in the event of a surge/backup/whatever. That is, once we begin to dequeue, we will actually be running at a full 100% until we’re back from being under, then we’ll be back at the usual 40%.

Third: How long were we backing up for, ie: what is the worst-case scenario we want to prepare for? If underlying apps rely on a database, and the database locks up for 10 seconds, then that will be 10 seconds of queueing we will get to enjoy. If we can’t afford 10 seconds, then what is the most we’re willing to deal with?

Equation time! Here are the three variables we have to deal with:

max_requests = number of requests/sec that our load balanced slice can handle (ie: 400 .. as in 400 req/s)

utilization = percentage of our average utilization that we want to plan around (ie: 0.6 .. as in 60% avg utilization)

queue_time = number of seconds we were queueing for (ie: 10 .. in which the db locked up)

And here are some things we can derive:

current_requests = ( max_requests * utilization) <--- ie: req/s utilized

available_requests = ( max_requests * ( 1 – utilization ) ) <--- req/s available ... this is also what you should consider headroom

queue_size = ( current_requests * queue_time )

At this point, it’s simple:

time_to_dequeue = ( queue_size / available_requests )

Basically, all we’re saying is, take the number of current requests/sec we’re handling, and multiply it by the number of seconds we were queueing. This will give us the number of requests that are backed up.

Once our queuing has stopped, and we can begin to dequeue, we will still be handling the usual rate of requests (ie: 60% of our max_requests), but we have that 40% of headroom (ie: available_requests) to handle the queue, so we divide our total queue_size up by the available_requests rate, and that will give us the number of seconds our load balancer will take to dequeue.

Expanded, the equation is this:

time_to_dequeue = ( ( ( max_requests * utilization ) * queue_time ) / ( max_requests * ( 1- utilization ) ) )

But wait! The beauty of this is, the time_to_dequeue is going to be the same for any given max_requests (you can see that from the equation). Regardless of our request rate, the rate at which we dequeue will be same relative to the percentage of utilization and a given queue_time.

So, we can simplify, removing anything having to do with request rates, allowing us to place time calculations for anything where we know the utilization and time we were queueing.

time_to_dequeue = ( ( utilization * queue_time ) / ( 1 – utilization ) )

Now, a lot of the times, we need to know how LARGE the queue will be at the given peak (to configure the load balancer accordingly), and in that case, we will need to know request rates.

Either way, I found this to be an interesting little exercise- got my mind thinking about math, and having an equation was much easier to help me build a table of data and pump it into gnuplot.

Enjoy!

After the meeting I was in finished, I was able to sit down and actually put some thought into what was happening. The one thing that stood out was interesting, but until then, I had no clue it was how Perl interpreted [in that scenario]. So, I wrote a short test script, sure enough proving the theory, and was able to fix my error.

Take a look at this:

1
2
3
4
5

my @animals = ("cat","dog","emu","frog");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

which dumps out this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


In my head, the code above would work like this: for each element in the array @animals, copy the data from that element into a new scalar $animal, then print it out. Simple enough. Now, consider this sample:

1
2
3
4
5
6
7
8
9
10
11
12

my @animals = ("cat","dog","emu","lemur");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
    $animal = sprintf("rabid %s",$animal);
}
 
printf("\n");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

I had *EXPECTED* it to output this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


For the first loop, again, my thinking was that we copy each array element’s data into the new scalar $animal, print it, modify it (by adding ‘rabid’ to it), but do nothing with the modification (we are just modifying $animal, which should be assigned by copy, of which would be lost when we iterate to the next element). Then, in our next loop, we iterate over @animals again, initializing $animal yet again for each element, so we just hit the un-modified @animals array and see the same thing.

[un]Surprisingly, I was totally wrong. This was the output I got:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the rabid cat
do not eat the rabid dog
do not eat the rabid emu
do not eat the rabid frog


Wow! What happened? Turns out, like the title suggests, when you use foreach in Perl, it actually assigns $animal to be a pointer (reference, whatever) to that array element, and not a copy. When you make any changes, the change applies directly to the element the array. As another example, it’s for-loop equivalent would be this:

1
2
3
4
5
6

my @animals = ("cat","dog","emu","lemur");
 
for( my $index=0 ; $index < @animals ; $index++ ) {
    printf("do not eat the %s\n", $animals[$index] );
    $animals[$index] = sprintf( "rabid %s", $animals[$index] );
}

Fun! Doing some research after the fact, it turns out there has been some mild discussion on the topic. Some consider it a bug, but in reality, it works by design. What does this mean for you? Well, if you happen to be iterating over an array using foreach, and want to modify each element, and plan on looping over the array multiple times, make sure you understand what’s going on behind the scenes, or else you’ll run into the same issue I did. You can create a temporary variable (which will assign by copy), ie:

my $new_animal = $animal;

or you can iterate using a for-loop and just do it like this:

my $animal = $animals[$index];

There of course may be some cases where you want to take advantage of the referencing feature, and if you do, all the more power to you.

Well, that’s all I have for now. Hopefully this helps someone out some day. Enjoy!

These days, a lot of ISPs have been doing NXDOMAIN redirection. Instead of handing back the NXDOMAIN response, they will intercept it and send you back an A record with an IP of one of their web servers (lets say 1.2.3.4 as an example). Your machine will ultimately think that gegooele.com has the IP 1.2.3.4, connect to that IP, and do a GET request with the HTTP host header of gegooele.com. The ISP’s web server, which specifically handles NXDOMAIN “redirected” HTTP requests, will then take that host header, pass it through some type of dynamic script, and display a nice custom error page with possible corrections, advertisements, etc.

Now, here’s where PowerDNS comes in: there are companies out there whose specialty is to sell DNS appliances and software solutions, solely for the purpose of doing NXDOMAIN redirection. However, if you are in the market for doing something like this (which is a whole separate discussion altogether, since there are strong opinions about this subject), you can save yourself a chunk of change by instead using PowerDNS (pdns-recursor, to be exact) with built-in Lua scripting. PowerDNS is open source, free, awesome.

We’re going to skip the build/compile details for now, and assume you have the latest PowerDNS recursor installed, and that Lua scripting has been compiled in. Chances are, it has. If not, check out http://www.powerdns.com and download the latest source. Once it’s installed, you just need to write a Lua script to modify DNS results, configure your recursor to use the script, and start up pdns-recursor. Let’s get to it.

First, you need to define a Lua script to handle DNS responses. Currently, Lua will only handle two types of responses: nxdomain, and preresolve. Preresolve is a response to give before any resolution has taken place. This is to basically hijack requests with a static response.

To define, we’ll add this line to our recursor.conf:

lua-dns-script=/opt/pdns/bin/nxdomain.lua

Now, let’s write our script. At it’s most basic level, it’s really easy:

function nxdomain (ip,domain,query_type)
ips={}
if query_type ~= pdns.A then return -1, ret end
ips[1]={ query_type=pdns.A, content="1.2.3.4" }
ips[2]={ query_type=pdns.A, content="5.6.7.8" }
return 0, ips
end

This script will first make sure our NXDOMAIN response is the result of a request for an A record. As long as it is, we will build an A record response to return instead of an NXDOMAIN. Per this code, the client will ultimately receive an A rotor response with two records.

Now that our script has been written and added to our recursor.conf, let’s start up the recursor and try it out!

$ /opt/pdns/sbin/pdns_recursor –config-dir=/opt/pdns/etc

Assuming the script is loaded successfully, you will see the following from the startup output:

Nov 25 03:09:36 Loaded ‘lua’ script from ‘/opt/pdns/bin/nxdomain.lua’

Looks good. Now, let’s see what happens when we try to hit our non-existent gegooele.com:

$ host gegooele.com
gegooele.com has address 1.2.3.4
gegooele.com has address 5.6.7.8

And there we go! NXDOMAIN redirection without the need for a commercial solution. Of course, there are more things you can do [with the Lua script]. WIth the proper log level, you can have it log every time the handler is called. For example, you can add a simple print line:

print ("nxdomain handler received: ", ip, domain, query_type)

You can also have it make decisions based on incoming IP, hostname/domain, and query type. However, the point of this was just to show the basics with what you can do. For more Lua syntax, and PowerDNS capabilities with Lua, check out the Lua website and PowerDNS wiki. Enjoy!

However, after a few more runs, it was getting kind of ridiculous: the routine in question was the one that did reads from the file handle. Now, I can understand on a multi-gigabyte file it might take a while (read in each line, parse it through a regex, and populate a data structure). However, it was causing the entire system to hang, and that made no sense at all given it’s relatively easy instructions. Here’s an example of what I was doing:

1
2
3
4
5

my $fd = IO::File->new("data.out",O_RDONLY);
 
foreach my $line(<$fd>) {
   @data = $line =~ /regex_pattern/;
}

Simple enough. So, on the next run, I started tracking the PID. Now, I saw two things: 1) the CPU was pinned, and 2) memory utilization was going through the roof. Bingo. Basically, my multi-gigabyte file was being read into memory, eating up all the remaining free physical memory, which was forcing the box to start swapping (which was eating up the CPU). Now it made sense why things were going ridiculously slow.

Now, what didn’t make sense was the fact this was happening at all. For each line of data, I was only pulling about 10% of it into my data structure (bytes parsed vs bytes per line), so I shouldn’t see memory utilization match that of the file. Additionally, I had worked with large files like this before, on systems with less memory, and never had an issue. What was different?

I took lunch, thought about it, and somewhere in there thought to myself, “hrmm, I wonder if it has to do with the foreach loop.” In the past, I had always used a while-loop, but didn’t really consider there might be an extreme difference between how each was interpreted when reading a file handle. I went back to my desk and modified the code to use a while loop instead:

1
2
3
4
5

my $fd = IO::File->new("data.out",O_RDONLY);
 
while(my $line = <$fd>) {
   @data = $line =~ /regex_pattern/;
}

Sure enough, it tore right through the file, did it’s parsing, and was done very shortly after. Interesting! So, what happened?

Basically, it’s a major difference in how Perl handles while loops vs. foreach loops.

In a while-loop, Perl will blindly shift lines one-by-one off of the array/fd you pass until it reaches an EOF. Once it does, it breaks out and you’re done. This is how it should work (and does exactly what we expect it to do), so that makes sense.

Now, in a foreach-loop, the Perl interpreter needs to know the end of the array before it begins iterating over it (whereas with a while loop we just go until EOF). If the data is already an array, that’s not a problem. However, if the data is a file handle, then the only way it can know the end is by reading the entire file into memory and ultimately create an array of the data to return back to the foreach iterator.

Most of the time, people aren’t running through multi-gigabyte files, so doing a foreach or a while on a file handle won’t really be noticeable, even though there are two completely different logic paths being followed behind the scenes.

Ultimately, use a while loop when reading off of a file handle. It’s cleaner, uses less overhead, and has obviously proven itself to be more fit for this task.

There are a lot of Perl hackers out there (many of whom I have worked with) who may end up commenting or shed some deeper technical insight. By all means, I would love to hear it! If you think I’ve done a good job summarizing and detailing the issue, that works just as well! Enjoy!

Now, that got me thinking. If sshd NEVER hits MaxStartups via the OS X default launchd setup, then it should be relatively easy to DoS an OS X box, right? By default, OS X only allows 256 maximum user processes and 512 file descriptors, so with each new fork of an sshd parent, you can easily eat into that space and essentially block anyone from doing anything once you hit the limit. Even better (or worse), I’m guessing most OS X (and OS X Server) users don’t change any of these defaults: they keep sshd via launchd, and never run into any type of issue where they’d have to increase maxprocs or max file descriptors.

With this idea in my head, I chose to use my wife’s Macbook Pro as our test server; it could use a little exercise. So, I did what any OS X user would do: I went to System Preferences > Sharing, and then enabled Remote Login. Of course, if  I do a ‘ps’ to see if it’s running, I won’t see anything, because launchd is just hanging out waiting for someone to hit port 22.

Next thing, my Macbook Pro will have the same system defaults, so I have to increase those to be able to exceed them limit on the receiving end. That’s easy enough- let’s increase our maxprocs and max fds:

chets-laptop $ sudo ulimit -n 2048
chets-laptop $ sudo ulimit -u 1024

There we go. Now, with my wife’s laptop willing and ready, let’s take out our trusty for loop:

chets-laptop $ for i in `jot 512 1`; do ssh 192.168.1.110 & done

Now, with that running, I jump over to my wife’s laptop (Terminal already open), and I try to do an ‘ls’. Sure enough:

wifes-poor-laptop $ ls
-bash: fork: Resource temporarily unavailable

Whoops! At this point, with a flood of SSH connections incoming, the only real thing I can do to stop the DoS is unload sshd from launchd. However, since the machine has hit the maximum number of processes allowed, I can’t even run the command to unload sshd from launchd. I can’t kill the sshd processes either, because I’d only be killing each individual parent, and launchd would just keep forking them off with each new connection. Regardless, killing won’t work anyway, just like unloading sshd from launchd:

wifes-poor-laptop $ sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
-bash: fork: Resource temporarily unavailable

also tried:

wifes-poor-laptop $ sudo killall sshd
-bash: fork: Resource temporarily unavailable

Ouch. Of course, I also tried opening some apps (System Preferences, Address Book, etc), and those would just bounce and bounce forever; the system had hit the hard limit, so there was nothing that could be done.

At this point, the ONLY way I could stop the attack was to take the machine off of the network, resulting in the sshd forks timing out and exiting. My wife’s laptop was actually wired at the time; if it were on Airport, I’m not sure if the system would have required an extra process resource to actually disable Airport, meaning a machine on the net via Airport would be totally, 100% useless.

Regardless, I think this is something people should be aware of before starting up Remote Login. Of course, running any type of remote login daemon is always a security risk, but if you’re on an open network with a public facing IP, you’re pretty much exposing your machine to being easily DoS’d by someone who knows how to write a for loop. You could, of course, set up TCP wrappers to buffer some of that, run ipfw, and whatever else, but I’m guessing that most OS X users aren’t going to be doing much in the way of this.

My suggestion: set up your hosts.allow/deny, configure ipfw to look for connection flooding on port 22, take sshd out of launchd, and, of course, only run sshd if you really need to. If someone out there wants to lock up your machine, and you’re running the OS X defaults, they can do it in about 10 seconds, and you won’t have a clue what hit you until you suddenly can’t do anything with your machine anymore.

All things aside, I love OS X and think it’s a great OS. There are just a few design decisions here and there that still need to be tweaked. Running sshd via the equivalent of xinetd is one of them. Good luck!

We have a rather heavily loaded administrative server, an Xserve, running Mac OS X Server. Additionally, we run a modified sshd configuration, where sshd runs as a standalone service, instead of via launchd as is with the default system installation. Is it relevant? Yes.

So, Friday evening rolls around (of course), and I get an e-mail from a co-worker saying that when she tries to log in, she gets that oh-so-common error message:

ssh_exchange_identification: Connection closed by remote host

Usually, in all of my experiences, this has been related to TCP wrappers; you’ve got a hosts.allow or hosts.deny set up, and you’re blocking (purposely or by accident), and just need to fix the config. Sometimes you’re totally screwed, but whatever. In our case, however, we aren’t using TCP wrappers. We could, since the default sshd in OS X does support it:

[chet@myhost]$ strings /usr/sbin/sshd  | grep wrap
Connection refused by tcp wrapper
libwrap refuse returns

or, if you have Xcode installed, you can use otool (the OS X equivalent to ldd in Linux):

[chet@myhost]$ otool -L /usr/sbin/sshd | grep libwrap
/usr/lib/libwrap.7.dylib (compatibility version 7.0.0, current version 7.6.0)

However, we don’t have a hosts.(allow|deny), so that’s not the issue. Thinking further, I remembered someone mentioning that the box was under more load than usual, and may have hit some type of system limit (maxprocs, maxttys, etc). To test, I basically flooded the box with ssh connection requests using a super-basic loop that a 3rd grader would write:

$ for connection in `jot 500 1`; do
$   ssh myhost &
$ done

Sure enough, right when I started the loop, I immediately began getting ssh_exchange_identification errors thrown back at me, whereas a single login not amidst a loop would work just fine. Bingo. So, it wasn’t a system limit per-se, but it was some type of sshd limit that was being triggered within the daemon itself.

Looking through the man pages, I came across an sshd_config option called MaxStartups. Here’s what the man page had to say:

MaxStartups

Specifies the maximum number of concurrent unauthenticated con-
nections to the sshd daemon.  Additional connections will be
dropped until authentication succeeds or the LoginGraceTime
expires for a connection.  The default is 10.

This sounded like a totally plausible cause for the issue. When I would flood, it would make requests so fast that sshd couldn’t keep up with authentications, so sshd would reach the MaxStartups limit and start blocking (to protect against DoS attacks, etc). To test it out, I changed MaxStartups from 10 to 200, ran my flood loop again, and sure enough, was able to connect without any errors. Perfect.

However, how come we hadn’t run into it before? Did it have something to do with us running sshd as a standalone daemon instead of via launchd? The standalone configuration is new for us, so we’re still keeping our eye out for bugs. To test, I ran my same flooding loop on an OS X Server machine running the system default (via launchd) sshd configuration. Sure enough, the issue did NOT present itself for that machine. Interesting.

Now, why is this, you may ask? It has to do with the way the sshd daemon runs via launchd.

The launchd service is very similar to xinetd when it comes to running network services: launchd knows what network services are configured under it, and it knows the ports those network services should accept connections from. So, launchd itself will listen on those ports. When a new connection request comes in (example: ssh), launchd will see it’s for port 22/sshd, fork off an sshd parent, and your connection gets handed off to that parent. If another connection comes in, launchd will fork off yet another sshd parent, and that 2nd user will get the 2nd parent. See the issue?

With the MaxStartups option in sshd_config, it relies on the sshd parent being the process handling all incoming connections, and forking off children for each new connection. If it sees that >= MaxStartups children are in an un-authenticated state, it will block new connections from establishing.

In the case of launchd, however, each connection is a parent forked from launchd, so no parent will ever see more than 1 connection (since it will have no children). In that respect, it totally subverts the sshd_config option for MaxStartups, opening up your machine to DoS attacks via sshd floods. Awesome.

In any case, now that we understand the issue, how come it was presenting itself under a scenario where people weren’t trying to flood it with ssh connections? The answer lies somewhere between system load, DirectoryServices, and the pam_securityserver.so PAM module.

When an ssh request comes in, and PAM is enabled, sshd will use the options defined in the /etc/pam.d/sshd file to determine what needs to be done to authorize the user. In our case, we have this configuration (slightly modified from the default OS install, since we’re not running via launchd):

[chet@myhost] $ cat /etc/pam.d/sshd
# sshd: auth account password session
auth       required       pam_nologin.so
auth       optional       pam_afpmount.so
auth       sufficient     pam_securityserver.so
auth       sufficient     pam_unix.so
auth       required       pam_deny.so
account    required       pam_securityserver.so
password   required       pam_deny.so
session    required       pam_permit.so
session    optional       pam_afpmount.so

Looking at the file, we notice a couple of entries for pam_securityserver.so. This module is actually the bread and butter of the authentication process: it queries the local DirectoryServices agent running on the machine to do the user authentication, and will receive a success or failure from the agent. The DirectoryServices agent can be configured to authenticate for local users, remote users from an LDAP directory server, or a mix of both. It’s versatile, but it does a good amount of work. In the case of it being bound to a remote LDAP server, it can also be very slow (depending on if you’re caching account results or not).

Our issue was two-fold: at the time, the host was heavily loaded with some CPU and network intensive processes. The CPU intensive processes were slowing down the communication between pam_securityserver.so and DirectoryServices, and the network intensive processes were slowing down the communication between DirectoryServices and our remote LDAP server. Between the both of these, authentications became super slow, and incoming, un-authentication connections were slowly creeping up (especially due to the scripted cron-job logins that had no logic to drop a hung login – awesome).

However, at that time, things were actually still okay. Slow, but people could still get in. Unfortunately (or luckily), someone sent out a chat message saying the host was acting slow, which of course prompted everyone to try logging in at the same time. At that point, 10 MaxStartups was hit almost immediately (ie: 10 people tried logging in, and since authentications were taking forever, all were stuck as un-authenticated), resulting in the ssh_exchange_identification error being sent back to all subsequent logins immediately (this was a nice bit of detail to note: the sshd daemon itself was still very quick to respond, but as soon as it had to hand an authentication back off to PAM, it just sat around waiting).

Ultimately, the resolution for this was we increased MaxStartups for our heavily-loaded machines (which is still better than the un-regulated launchd sshd), and we will be setting up  separate server to handle all the jobs that were running.

Looking at other options, I noticed while writing this that pam_securityserver.so comes BEFORE pam_unix.so. As a test, I may try using pam_unix.so first, allowing root logins to bypass DirectoryServices, hitting passwd/shadow directly, and hopefully allowing us to jump in on what seems like a hung system due to DS being super slow. I’ll try to get back with an update if I end up testing this.

That’s all from me for now. Feel free to comment on your own experiences, knowledge, other ideas, or if you’ve run into the same thing. Enjoy!