Now, the Netscaler has built in health checking for many different types of backend service protocols: HTTP, FTP, DNS, and more. However, one protocol not included is NTP. If you happen to be load balancing a farm of NTP servers, and want to ensure you’re not routing to a hosed NTP daemon (even if it’s ping-able), it would be nice to be able to health check the service [and take it out of rotation if it's not responding].

Luckily, the Netscaler provides (with documentation) a very flexible way to write your own custom health checks. Using the system, you can test whatever you want, in any way you want, with just a little Perl. Even better, the custom health check subsystem provides a few useful things to simplify the whole process, including:

• passing in the service IP, and port, as arguments to your script
• passing in a custom argument string (key value pairs, user/pass info, whatever)
• handling the timeout for you from a higher level; if your check doesn’t respond within -respTimeout seconds, it will be considered a failure.

Now, for just the basics, this monitor will actually be pretty easy to put together. We are given the IP/port to send an NTP request to, and we don’t need to worry about a timeout, because the underlying subsystem will handle that for us. All we really need to do is send a request and [hope] for a response!

That being said, we’ll need to do a couple things to get to that point. These things are:

1. Load the health check subsystem module.
2. Load the Socket module, and create a socket to use to send the request [using the parameters passed in as args].

From that point, we can build and send our request. Since we don’t need to worry about handling a timeout, we can just sit there waiting for a response. First up:

use IO::Socket;
use Netscaler::KAS;

We’ll need both of these modules – one will tap into the health check subsystem, and the other is so we can send our UDP-based request.

Next is our function, which we’ll just call ntp_probe. It assigns a few variables from $_[0] (the service IP) and $_[1] (the service port) as arguments, which get passed in from work within probe().

So, the magic-fu to this is that, later on, we will pass our ntp_probe function as a coderef to the KAS probe() command. That will ultimately result in our function being called with the arguments passed in; it is similar to this:

$custom_function->($host,$port,$args);

If you look at the KAS code (and since it’s perl, you can), you can see the exact line; this just gives you an idea of what’s going on.

In any case, after that, we’ll create a new UDP socket, send the NTP request message down it, and either:

• return 1 if something wasn’t defined (ip, port, or sock) – failure!
• return 0 if we got a response – success!

Again, we don’t need to worry about timeouts, since the subsystem will handle that for us. So, here’s the function in it’s entirety:

sub ntp_probe {

    my $host    = $_[0];
    my $port    = $_[1];
    my $req     = "\010"."\0"x47;

    if(!$host || !$port) {
        return(1,"Host or port not specified.");
    }

    my $sock = IO::Socket::INET->new(
        Proto     => "udp",
        PeerAddr  => $host,
        PeerPort  => $port,
    );

   if($sock) {
        $sock->send($req);
        $sock->recv($_,1);
        return 0;
    }

    return(1,"Could not create socket");

}

Not bad, was it? Now that we have loaded our modules, and defined a function, the next step is to call the probe() method, with a coderef to our new function, to tell KAS what function to use for probing:

probe(\&ntp_probe);

And you’re off to the races! Just don’t forget to start your script with #!/usr/bin/perl, and it should be good to go

Now, you might be asking “how do I actually configure the Netscaler to use this?” That’s also just as easy. Let’s pretend our new script is called customNTP.pl. Just plop it into the /nsconfig/monitors/ directory, and do something like this from the CLI:

add lb monitor "custom-ntp-mon" USER -scriptName customNTP.pl
   -scriptArgs 1 -dispatcherIP 127.0.0.1 -dispatcherPort 3013
   -interval 20 -respTimeout 3

With that line, you will now have a working NTP healthcheck (20 second intervals, 3 second timeout). Just bind it to your service(s) that need it and enjoy!. Note that the type of monitor is USER; this is the type used for custom health checks.

Well, that’s all for this evening. There are bits and pieces of info like this around the web, on the Citrix AppExpert site, etc. And, even though it wasn’t too difficult to figure out, hopefully this will help someone out some day.

Take care!

In any case, one of our metrics publishing systems is wrapped by a few different object classes I put together in Perl (follows a certain protocol created by another group, etc). Of the many data types supported by our metrics receivers, we can do things like 32-bit unsigned integers, single precision (32-bit) floating points, and a few others.

Now, when I would publish the 32-bit uint’s, it would work great- no issues. The receivers would expect a byte stream, so I would use pack() to send my values as binary. For example:

$val=29;
print $sock pack('N',$val);

Easy, whatever. However, when I would send values as a single precision float, it would totally be hosed on the other end. For example:

$val="30.0";
print $sock pack('f',$val);

On the other end, they would get some massive negative number. I don’t deal with this too much, things usually just work. Thankfully, my mind was jogged in regards to standards- there are lots of standards around byte order for integers (16 bit, 32 bit, 64 bit), but not as much for float; one architecture may use big-endian, and another architecture might use little-endian.

In my searches, I found that Intel stores floating points in little-endian byte order, and Sparc stores them in big-endian byte order. Sure enough, I was trying to publish from an Intel-based BSD host to a Sparc based Solaris host. Aha!

Luckily, pack() can *also* handle this for us! With a simple carat added to the templates section of pack(), I can easily pack the value in big-endian byte order and resolve our issue. I do that like so:

$val="30.0";
print $sock pack('f>',$val);

And, ta da! Worked like a charm. Was it difficult to get it working? No. But, it was fun to have to deal with something like this. On any given day, you don’t really worry about endian-ness; things just work. This was one of those days where I actually had to use the old noggin, and that was refreshing.

After the meeting I was in finished, I was able to sit down and actually put some thought into what was happening. The one thing that stood out was interesting, but until then, I had no clue it was how Perl interpreted [in that scenario]. So, I wrote a short test script, sure enough proving the theory, and was able to fix my error.

Take a look at this:

1
2
3
4
5

my @animals = ("cat","dog","emu","frog");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

which dumps out this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


In my head, the code above would work like this: for each element in the array @animals, copy the data from that element into a new scalar $animal, then print it out. Simple enough. Now, consider this sample:

1
2
3
4
5
6
7
8
9
10
11
12

my @animals = ("cat","dog","emu","lemur");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
    $animal = sprintf("rabid %s",$animal);
}
 
printf("\n");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

I had *EXPECTED* it to output this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


For the first loop, again, my thinking was that we copy each array element’s data into the new scalar $animal, print it, modify it (by adding ‘rabid’ to it), but do nothing with the modification (we are just modifying $animal, which should be assigned by copy, of which would be lost when we iterate to the next element). Then, in our next loop, we iterate over @animals again, initializing $animal yet again for each element, so we just hit the un-modified @animals array and see the same thing.

[un]Surprisingly, I was totally wrong. This was the output I got:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the rabid cat
do not eat the rabid dog
do not eat the rabid emu
do not eat the rabid frog


Wow! What happened? Turns out, like the title suggests, when you use foreach in Perl, it actually assigns $animal to be a pointer (reference, whatever) to that array element, and not a copy. When you make any changes, the change applies directly to the element the array. As another example, it’s for-loop equivalent would be this:

1
2
3
4
5
6

my @animals = ("cat","dog","emu","lemur");
 
for( my $index=0 ; $index < @animals ; $index++ ) {
    printf("do not eat the %s\n", $animals[$index] );
    $animals[$index] = sprintf( "rabid %s", $animals[$index] );
}

Fun! Doing some research after the fact, it turns out there has been some mild discussion on the topic. Some consider it a bug, but in reality, it works by design. What does this mean for you? Well, if you happen to be iterating over an array using foreach, and want to modify each element, and plan on looping over the array multiple times, make sure you understand what’s going on behind the scenes, or else you’ll run into the same issue I did. You can create a temporary variable (which will assign by copy), ie:

my $new_animal = $animal;

or you can iterate using a for-loop and just do it like this:

my $animal = $animals[$index];

There of course may be some cases where you want to take advantage of the referencing feature, and if you do, all the more power to you.

Well, that’s all I have for now. Hopefully this helps someone out some day. Enjoy!