sidenotesand other random technical bantering, by chet nichols

So I like using Perl, if anyone hasn’t realized yet. Python seems awesome and all, but I haven’t had the time to start writing things in Python. I’m still at the stage when, if a tool needs to be put together, it needs to be done ASAP; I can’t delay trying to figure out Python particulars.

In any case, one of our metrics publishing systems is wrapped by a few different object classes I put together in Perl (follows a certain protocol created by another group, etc). Of the many data types supported by our metrics receivers, we can do things like 32-bit unsigned integers, single precision (32-bit) floating points, and a few others.

Now, when I would publish the 32-bit uint’s, it would work great- no issues. The receivers would expect a byte stream, so I would use pack() to send my values as binary. For example:

$val=29;
print $sock pack('N',$val);

Easy, whatever. However, when I would send values as a single precision float, it would totally be hosed on the other end. For example:

$val="30.0";
print $sock pack('f',$val);

On the other end, they would get some massive negative number. I don’t deal with this too much, things usually just work. Thankfully, my mind was jogged in regards to standards- there are lots of standards around byte order for integers (16 bit, 32 bit, 64 bit), but not as much for float; one architecture may use big-endian, and another architecture might use little-endian.

In my searches, I found that Intel stores floating points in little-endian byte order, and Sparc stores them in big-endian byte order. Sure enough, I was trying to publish from an Intel-based BSD host to a Sparc based Solaris host. Aha!

Luckily, pack() can *also* handle this for us! With a simple carat added to the templates section of pack(), I can easily pack the value in big-endian byte order and resolve our issue. I do that like so:

$val="30.0";
print $sock pack('f>',$val);

And, ta da! Worked like a charm. Was it difficult to get it working? No. But, it was fun to have to deal with something like this. On any given day, you don’t really worry about endian-ness; things just work. This was one of those days where I actually had to use the old noggin, and that was refreshing.

I’m not sure how I’ve never run into this issue before. In some work I was doing recently, I ran into what I thought was a nasty bug [in my code] but couldn’t explain it. Without thinking, I started trying to undo this, fix that, hack this, and ignore that.

After the meeting I was in finished, I was able to sit down and actually put some thought into what was happening. The one thing that stood out was interesting, but until then, I had no clue it was how Perl interpreted [in that scenario]. So, I wrote a short test script, sure enough proving the theory, and was able to fix my error.

Take a look at this:

1
2
3
4
5

my @animals = ("cat","dog","emu","frog");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

which dumps out this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


In my head, the code above would work like this: for each element in the array @animals, copy the data from that element into a new scalar $animal, then print it out. Simple enough. Now, consider this sample:

1
2
3
4
5
6
7
8
9
10
11
12

my @animals = ("cat","dog","emu","lemur");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
    $animal = sprintf("rabid %s",$animal);
}
 
printf("\n");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

I had *EXPECTED* it to output this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


For the first loop, again, my thinking was that we copy each array element’s data into the new scalar $animal, print it, modify it (by adding ‘rabid’ to it), but do nothing with the modification (we are just modifying $animal, which should be assigned by copy, of which would be lost when we iterate to the next element). Then, in our next loop, we iterate over @animals again, initializing $animal yet again for each element, so we just hit the un-modified @animals array and see the same thing.

[un]Surprisingly, I was totally wrong. This was the output I got:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the rabid cat
do not eat the rabid dog
do not eat the rabid emu
do not eat the rabid frog


Wow! What happened? Turns out, like the title suggests, when you use foreach in Perl, it actually assigns $animal to be a pointer (reference, whatever) to that array element, and not a copy. When you make any changes, the change applies directly to the element the array. As another example, it’s for-loop equivalent would be this:

1
2
3
4
5
6

my @animals = ("cat","dog","emu","lemur");
 
for( my $index=0 ; $index < @animals ; $index++ ) {
    printf("do not eat the %s\n", $animals[$index] );
    $animals[$index] = sprintf( "rabid %s", $animals[$index] );
}

Fun! Doing some research after the fact, it turns out there has been some mild discussion on the topic. Some consider it a bug, but in reality, it works by design. What does this mean for you? Well, if you happen to be iterating over an array using foreach, and want to modify each element, and plan on looping over the array multiple times, make sure you understand what’s going on behind the scenes, or else you’ll run into the same issue I did. You can create a temporary variable (which will assign by copy), ie:

my $new_animal = $animal;

or you can iterate using a for-loop and just do it like this:

my $animal = $animals[$index];

There of course may be some cases where you want to take advantage of the referencing feature, and if you do, all the more power to you.

Well, that’s all I have for now. Hopefully this helps someone out some day. Enjoy!