Metro Ethernet Frustrations and the fix

For those that have been following the last few weeks, you’ll know that I was implementing a Metro Ethernet (MAN) network for my company using a Cisco ME-3750 switch, and the frustrations that came with it.

Well, I still went forward using the ME-3750, just not using the ES Ports.  The circuit came up fine.  I thought I was done.  Ran a few simple tests…. yep, communication is working.  So I left to the airport.

A couple of days later, communications started failing.  Intermittent results.  Basically, it stems from ARP not working correctly.  I would try to ping a device, then look at it’s ARP table.  Nothing….. hmmm… wierd.

I then look at the Switch’s mac-address table… yep…. the MAC’s are populating.  So, I go the distance, and place two network sniffers on each end.

The end result, was that sometimes (and only sometimes), an ARP request would get sent out, but the reply would never com back across the link.  Since I was monitoring the actual trunk ports, this must be a problem with the provider.

Well, come to find out, AT&T (who was doing the fiber to copper hand-off for me) has a mac-address table with a maximum entry of 50 entries.  50 entries!!  We have a decent sized network, and we are moving servers to a co-location DR site.  C’mon… we’ll max 50 entries in no time.

I didn’t even know AT&T caps mac-addresses.  Do other providers do this, as well??

Technorati Tags: Cisco, Network, metro-ethernet

Cisco discontinues the Pix Firewalls

Well, it has finally happened.  It was only a matter of time.  Since the Cisco ASA (Adaptive Security Appliance) did exactly what the Pix does, and them some, why support two lines?
Cisco announces that they will stop sales for the Pix Firewall in January 2009.  Support, however, will be continued until 2013.

See Cisco’s press release.

So, let’s talk about these ASA’s.  For those that do not know, the ASA is actually the PIX underneath, with modularity to allow you to expand it to a specific appliance, such as Application Inspection or Virus/Malware/Spyware inspection.  I’m actually using the one with the CSC module, which includes the Virus/Malware/Spyware inspection.  The ASA actually inspects SMTP,HTTP,POP3, and IMAP packets.

When I first purchased and used it 18 months ago, Trend Micro (who owns the scanning engine of the CSC module) had quite a few bugs in it, so, I didn’t like it at first.  Too many problems.  However, over the last 18 months, their updates and bug fixes have seem to stablize it a little bit

You can learn more about the ASA at Cisco’s website, if you aren’t already familiar.

Technorati Tags: Cisco, Network, Security

3750-metro frustrations. Worth it??

I am incredibly frustrated, and I’m hoping this post will save countless others from yelling out loud, after finding out you wasted money and hours of troubleshooting for no reason.

First off, a bit of history. Back in my historical posts, I mentioned my thoughts on the 3750-metro series switches. At first, I wasn’t all that impressed with the metro-series. I mean, it seems to be more for the service provider than for the customer. But, if Cisco recommends it for a customer, I should listen, right?

Well, Cisco sales reps are more interested in selling products, than in giving the best answers, and this is a perfect example. So, make sure to listen up, and pay attention.

In building our metro network, we have a fiber line, provided by AT&T. On the customer side, AT&T installed a Cisco switch to terminate the fiber, and hand-off copper. I then, take the copper into my network. But, after connecting my 3750-metro interface (The ES port, for “Enhanced Services”), I never saw a link. Hmmm… did AT&T enable their interface? I asked…. yep. Do I need a cross-over, or did they build the cross-over in their patchpanel? Nope. Straight-through. So, I must provide the cross-over
(And for those that will ask… no I do not trust the auto-sensing MDIX).

Still no luck.

Then, I asked AT&T the interface characteristics: (100Mbit - Full Duplex). Hmmmm…. shouldn’t be a problem. I’ll set my interface to that. What?? I can’t. It only accepts 1000? Let’s look at the docs:

Now, I consider myself a respectful and considerate human being (at times). However, I must say, when I realized that, I was glad I was in an isolated room with no one around. Because I yelled and cussed as I used to when I was a sailor in the Navy. At this point, I realized I have spent between $6k to $8k more than I needed to (I bought 2 metro switches, one for each side of the link), and I just wasted 3 to 4 hours of troubleshooting. I should have gone with my initial feelings about the 3750-metro.

So, what is the difference between the 3750 and the 3750-metro? From Cisco’s website:

Q. What is the difference between the Cisco Catalyst 3750 Metro Series and the Cisco Catalyst 3750 Series?

A. The Cisco Catalyst 3750 Metro Series is built for Metro Ethernet access in a customer location, enabling the delivery of more differentiated Metro Ethernet services. These switches feature bidirectional hierarchical QoS and Traffic Shaping, intelligent 802.1Q tunneling with class-of-service (CoS)
mutation, VLAN translation, MPLS, EoMPLS, and Hierarchical Virtual Private LAN Service (H-VPLS) support, and redundant AC or DC power. They are ideal for service providers seeking to deliver profitable business services, such as Layer 2, Layer 3, and MPLS VPNs, in a variety of bandwidths and with different SLAs. With flexible software options, the Cisco Catalyst 3750 Metro Series offers a cost-effective path for meeting current and future service requirements from service providers.

And what is this “ES Port” thing. What does it do for me?

Q. What are the Enhanced Services (ES) ports?

A. The Cisco Catalyst 3750 Metro Series includes two SFP-based ES ports. The ES ports support Metro Ethernet features that are vital for delivering profitable business services, such as Layers 2 and 3 and MPLS VPNs, in several bandwidths and with different SLAs. Supported features on the ES ports
include EoMPLS, MPLS, MPLS VPNs, bidirectional hierarchical QoS, intelligent 802.1Q tunneling (Q-in-Q) with CoS mutation, and VLAN ID translation.

So, to summarize, pretty much every enhancement the metro-line offers, is in the ES ports. Other than the redundant power-supplies, this is a wash. The regular Cisco 3750 still has a powerful QoS engine, and the same IOS commands. Now, I’m sure there is more in the metro software image than the standard 3750, but without the ES Ports, what’s the point??? The provider will provide the Q-in-Q tunneling, the heirarchical QoS, etc. What good does it do??

I’m still a little bitter, but by the time you post your comments, maybe I’ll feel better, and listen more. So feel free to let me know what you think. Right now, I’m disappointed, and frankly, quite pi$$ed, because I feel like I was taken advantage of.

My only advice is…. if you get the Cisco 3750-metro, please make sure the provider will hand off a 1000 Gigabit connection. Otherwise, you are sitting on an expensive 3750.

The only thing I can think of, is to place a media-converter in the middle, so I can use the 100Mbit ES interface. But, that just adds another point-of-failure, and this metro network was supposed to alleviate the failures… not add to them…..

((Sigh))

Cleveland - MAN (Datacenter recolation)

Well, I’m off to Cleveland this week. I’ll be bringing up our first MAN to a new co-located datacenter at Expedient.  We currently have one cabinet there, but look to getting a second cabinet in a few more months, depending on how great it works out.

The only thing I do not like about the 3750-metro switches is that they don’t have a netflow export option.  Personally, I think they should.  Being a multi-layer switch, and intended to be on each side of a MAN, the netflow export would be perfect to analyze the traffic across the MAN link.

What do you all think?  Agree to disagree?

For me, it will be a learning experience.  This will be my first time using the Cisco 3750-metro series switches (I’ve used the 3750’s before but the metro’s allow more fine-grained control on QoS and packet-shaping, though, I lose the Gb interfaces).

I’ll need that QoS and packet-shaping skils, since we only have a 20M fiber link between the two locations, and we’ll be sending voice and video through it, on top of regular data traffic.

To be honest, I’m not that confident in my QoS knowledge.  Yes, I know the fundamentals (at least I think I do…. uh oh), but I’m always nervous I’ll forget one tiny little thing, which will cause disastrous results.  Ah well…. I feel that way everytime I touch a keyboard key.

Wish me luck!

Technorati Tags: cisco, 3750, metro, expedient

Powered by ScribeFire.

New Cisco Certification

Cisco has created a new certification called CCDE (Cisco Certified Design Expert).  So, what’s so different between a Design Professional (CCDP) and a Design Expert (CCDE)?

Seniority and includes Business strategies, says Cisco:

“There’s not going to be a lot of CCDEs walking the street,” said David
Bump, a portfolio manager with Cisco. “It’s a very senior credential;
it’s a very exclusive credential.”

Basically, you will take the technical knowledge of the CCDP (and other certifications), but rather than design a technical network around specific requirements, you would be the one to create those requirements, maintain budgets, and make sure those requirements fits the business needs, not just the technical needs.

The 2-hour qualification exam is released today at Pearson VUE centers.  Once you have qualified, you will be able to take the 8-hour long Proctored exam, made available this fall.

No doubt, many CCIE’s will try to get this “higher” status.  Me?  Well, I’ve been using and building Cisco networks for almost 7 years now.  I still don’t have my CCNA!!  No time for the weary, I guess.  Yeah yeah… I know I should…..

For those of you, who didn’t know that, I probably just knocked my credibility down by 2 notches or so.  Sorry to let you down.

Link to Referenced work:

Cisco announces Design Expert Certification for network engineers

Cool…

Technorati Tags: cisco, certification, ccde

Powered by ScribeFire.

Cisco Timestamps - Converting

This was so freaking confusing for me! In Java, I would convert my timestamps one way, but then, if I needed to convert them in Excel, I was 70 years off?!? WTF.

Well, here’s how you convert those timestamps to something meaningful, and why you are getting different results with different systems.

First, some terms:

Unix Epoch: number of seconds elapsed since Jan 1, 1970

NTP Epoch: number of seconds elapsed since Jan 1, 1900

I have no idea why the different Epochs, but did you notice that the difference is 70 years? YEP!!!

Okay, so Microsoft Products use the NTP Epoch (i.e. SQL Server, Excel, Access, etc) as a reference to build it’s Date objects. Unix, Macintosh, Java, and C / C++ uses the Unix Epoch as a reference for it’s date objects. Cisco uses Unix Epoch to export its timestamps.

So, basically, we have to add 25,569 days (70 years, approx) to the NTP Epoch, in order to get valid results in NTP Epoch-type systems (Excel, SQL Server, etc).

Convert a timestamp in JAVA:

public static void main(String[] args){

// declare our timestamp in seconds

long timestamp = 1198167416;

// Since timestamp is in seconds, but Java Date works in milliseconds,

// convert to milliseconds
Date mydate = new Date(timestamp*1000);

// Format how the date is displayed to us

SimpleDateFormat formatter = new SimpleDateFormat(”dd MMM yyyy HH:mm:ss”);

// Print the date to standard out.

System.out.println(formatter.format(mydate));
}

Convert a timestamp in Excel (Assuming your timestamp to convert is in column A1):

A1/86400+25569

Then, just format the cells as “Date”. So, to explain the Excel formula:

1). Divide the timestamp by the number of seconds in a day. This will give you the number of days

2). Add 25569 days to the already converted days, to take into account the difference between Unix Epoch and NTP Epoch shift.

Cisco IOS - Order of Operations

So, for the first time, I had to NAT an IPSec tunnel for a vendor, due to overlapping networks.  I know the fundamentals, but have never actually done it.

First, define the vpn traffic…. check

Next, define the  nat traffic…. check

Map the traffic to the cryptomap….check

Create the access-list to filter the VPN Traffic…..no check…. ummm…. okay.. so here is where I needed some help.  Does the ACL get hit first, and THEN NAT?  If so, I’ll need to use the NAT address in the ACL.  But, what if NAT gets hit first?  Then I’ll have to use my private address in the ACL.

What to do?  Well, visit the irc chat room #cisco, that’s what.  They sent me to an incredible post which details the operations of both ingress and egress in a Cisco IOS system.

Very handy!  ….. <Aaron is printing>….

Styx Update

Styx now has it’s own webpage. I had to put it in googlepages since Javaforge won’t do web hosting Which makes me irritable, since they were doing web hosting when I first checked them out!

The site is in it’s extreme infancy, since I just started it today, but plan for much more information in the very near future. I can’t wait for this project to really take off!

Anyway, the project site is located here.

I’m calling it a 0.01 release. It won’t be a 0.1 release, until I have two tasks completed. Which one of them is already 95% done.

The Cisco 3750-Metro swtich?

Well, I’m about 4-6 weeks out from finishing my co-lo project.  We are moving some of our critical servers and phone system to a hardened datacenter facility.  In doing so, Cisco kept insisting that we purchase the 3750-Metro switches.  I was just going to get the regular 3750’s.

Well, I guess the 3750-Metros does have some nice features, but not all of it is useful.  For example, they were telling me, “But, Aaron….. this switch does HEIRARCHICAL QoS.”.  In lamens terms, it allows you to merge different QoS policies.  But that only helps when you are the provider with multiple customers and different SLA’s.  I’m the customer.  Why would I need that?

The packet shaping option is kind of nice, I’ll admit.  But, in order to take advantage of it, you MUST uplink into an SFP port.  They are the only ports that can use the advanced IP feature set.

I’ll let you know more, once I receive the switches, which should be this week, actually.  Does anyone have a comment on the 3750-Metros?

Technorati Tags: cisco, 3750, metro, switch, networking

Powered by ScribeFire.

My Netflow Collector/Analyzer Project

I thought I would run a quick post, to let you all know where I’m at.  For those that have been following for a little while, you’ll know that I’ve started a bit of an adventure to create an easy-to-use, but very powerful, Netflow Analyzer.

Why another one?  Well, each analyzer has their benefits and their weaknesses.  Some have steep learning curves to configure and report on.  Others are not very portable across OS’s, and still others are not very flexible with writing your own reports, or takes a seasoned Unix sysadmin to compile and install.

So, I’ve started my project, written in Java, to help alleviate all those pains I just mentioned, and make it easy to extend.

I’ve decided on a code-name, until a proper project name can be determined (any suggestions??).  I’ve codenamed the project “Styx” based on the Greek Mythological river Styx.  While I can give a clever little speech on a few metaphores…. for now I’ll leave it to your imagination.  I’ve created a project on JavaForge, but haven’t uploaded anything yet.

Anyway, I’ve made some decent progress.  The fundamental engine I’m using is a dead project called JNCA (Java Netflow Collector Analyzer).  I’ve received the author’s permission to use it.  But, since it’s fairly old, I’m replacing some core functionality, such as adding Apache’s log4j, and cleaning up the code to be properly wrapped in a java container.

Anyway, I’ll be using LOTS of Open-Source technologies and projects to kind of mash and mold this project.  It will be interesting to see what I come up with.  If it’s half as cool as what I can imagine in my head, it WILL ROCK!!!  My only concern is using an RDBMS system to store the data.  If you have a busy network, you could be looking at millions of records per day (or more), easily.  So, I’ll have to work on some kind of buffering/aggregating system to alleviate that a little.  Dunno.

One of the biggest resources I’ve been using is Cisco’s new book “Network Management: Accounting and Performance Strategies“, which I’ve been meaning to write a complete review of for quite some time!  I feel so ashamed.  This book rocks, and I have not given it the respect it deserves!

No ideas on when Styx will be available… time will tell.

Stay tuned!

Technorati Tags: cisco, netflow, collector, analyzer, java, styx

Powered by ScribeFire.