RedWarriorNet

You know when you move a piece of furniture that’s been in one plac a very long time and suddenly discover something underneath that has been there almost as long?  There’s that feeling of having discovered a piece of the past?  Well, it turns out that moving servers to a new datacenter is often a lot like that, particularly in a large enterprise where things often get lost in memory.  We are currently working on moving one of our main applications to the datacenter and one of the system administrators made an astounding discovery yesterday afternoon.

He couldn’t get some remote printing to work with a terminal server he installed here at the new datacenter.  The only difference is that, unlike back at the old datacenter, there is a new network for these systems at the new datacenter so, until the rest of the application servers are moved, this terminal server is on a separate network.  After much troubleshooting with the vendor, he came over to the network engineering cube area with the most remarkable question I think I’ve been asked at 4:00 on a friday…”Hey, do you guys block DECnet?”  After a moment of thinking back to the first chapter in most of my TCP/IP books, I remembered that what he was talking about was the precursor to IP and a layer 3 protocol in its own right that dates back to the very early days of the internet.  The ensuing conversation was a mixture of astonishment and curiousity mixed with equal parts consternation.

Of course, we don’t “block” DECnet.  We didn’t even know they were using it!  The problem really is that we don’t ROUTE DECnet.  DECnet has its own methods of routing and we have nothing in place to move this kind of traffic from one network segment to another.  This is why we were able to be happily oblivious to this kind of layer 2 traffic going on as long as the two hosts using it were on the same segment.  We wracked our brains, wondering if we could encapsulate it somehow within an IP packet or else use a GRE tunnel of some sort…anything to move that over our routed WAN link to the datacenter.  Finally, we shrugged and suggested using a Metro-E connection to temporarily span that subnet from the old datacenter  to the new until the servers are moved.  The system administrator is also looking for a solution to get this system to use IP rather than DECnet, which is definitely something that needs to happen long term.  A bit of further digging showed that since this traffic doesn’t even use MAC addresses, our switches have likely been treating it like broadcast traffic and just flooding it out all ports in the same vlan.  It looks like the only way we’d be able to route this is using DECnet over TCP.

Before this dinosaur is laid back to rest, though, I’m going to try to see if wireshark will capture some of this traffic.  I feel almost like I’m in a Geico commercial and have suddenly met a Caveman and I’m curious to see how he behaves.  I’m also curious to see how our switches treat this traffic over ethernet.  I’m not sure if the rest of the team shares my fascination, though…they seemed a bit more irritated that we were asked to deal with a layer 3 protocol that should have been taken out of production decades ago and also that we were asked to deal with it a week before a server move.  

Next time I move my couch…I’m going to be prepared for anything!

I just found out through working with a TAC Engineer on an issue where our brand spanking new Catalyst 4900M won’t accept the command to place a Policy Based Route map on a vlan interface that PBR will not be supported until an upcoming IOS release, most likely in June.   Ugh.  This is not the first weirdness we’ve had with our 4900M’s.  The first was with one of the 4900M’s we have deployed in our new datacenter core.  This switch displayed some weird behavior when we were troubleshooting what appeared to be an issue with a fiber uplink in a twingig module we had plugged into one of the ten-gigabit ports.  We shuffled around some known-good GBICs in these twingigs and next thing we knew, the ports had error-disabled due a Cisco “feature” that keeps you from using illegal hardware, giving us an error of “Unapproved GBIC” in the output of a show interface status.  A reboot got us our ports back, but to this day one of the ports still shows that error even though it works and TAC hasn’t been able to help us be rid of it.  Apparently, on most of the higher-end switches, the database that keeps track of GBIC serial numbers times out entries after a certain period so that you can reuse a GBIC from one module in the switch to another without much trouble…in the 4900M’s they have yet to perfect this.

Cisco TwinGig Module

Don’t get me wrong, the 4900M’s are definitely good hardware and allow you to double your port density in modules you don’t want to use TenGigabit in by using the twingig modules that give you 2 1Gig ports instead, while still giving you the option down the road of using half as many tengigabit ports.  That can be a very handy feature when combined with the port density they already have and is the reason we have 4 of them in the core of our datacenter.  I just wish Cisco had worked out a few more kinks before releasing them.

But enough complaining…it’s good to have job security in these uncertain times!  My BCMSN studies are coming along well and I’m up to chapter 8 in the self-study guide.  That doesn’t sound like much, but the self-study guide’s chapters are all pretty meaty.  I also managed to watch 4 hours straight of CBT Nuggets this weekend before needing a nap.  My lab setup is almost finished and I hope to get it accessible from my desk as soon as there is a lull in the datacenter cabling.  Until then, I hope to keep pushing on my reading.  I’ve already really enjoyed some of the information on MST, which allows you to combine multiple instances of spanning tree into one instance, saving resources and allowing you to more easily use redundant links for load-balancing instead of just leaving them in blocking.  I’ve also really enjoyed the section on etherchannel, particularly doing more sophisticated load-balancing over multiple etherchannel links.