Teneo !!!

Aaron’s blog on Networking, and Enterprise Technology

Should I continue building a netflow analyzer?

So, I’ve been developing a netflow analyzer off and on for a year now. It’s called Styx.

Do we really need another netflow analyzer? I mean, there are so many out there, both open-source and commercial.

I wanted to make something easy to use, and understand, but do we really need it?

Uggghhhhh…. what to do……

Advertisements

7 responses to “Should I continue building a netflow analyzer?

  1. Mike Patterson August 2, 2009 at 5:29 am

    Hi Aaron, I’m not sure how busy you are but, we could use some help coming up with ideas on ours. Do you have any interface suggestions you are willing to share?

    We are working on Scrutinizer v7 for NetFlow and sFlow analysis.

    Sincerely,

    Mike

  2. James August 3, 2009 at 11:19 am

    There’s always room for one more. Especially if you can think of something unique to build into yours that none of the other free or commercial ones have.

    I’m currently building an app which will enter a fairly crowded space in the next few weeks when I release it. My app contains far more feature than any of the currently available free competitors, is almost on par with many expensive commercial ones, yet mine will be free (gonna ask for donations though). That’s how I plan on diferentiating myself from the others. It sometimes doesn’t take much to set yourself apart.

    Also, you will never learn more about a particular technology than when you build an app that uses it. If you complete this project, you may start to be known as “Mr. Netflow” 🙂 You could still complete this project as little more than a learning experience. (And resume builder)

    –cheers

    • Aaron Paxson August 3, 2009 at 12:24 pm

      Thanks James! I appreciate the feedback!

      Yes, there are some advantages of my app (trying not to reinvent the wheel). But I ask myself if the advantages are worth the time. I guess time will tell?

      🙂

  3. snetherland August 5, 2009 at 7:12 pm

    Aaron,

    I appreciate you bringing this up as this hits very close to home for me. If your looking to invest into the open source community, then your expertise may be just as helpful in developing an already established app, or just adding ease-of-use functionality. To give you an idea of why this interests me, our company is constantly in need of a good collector that not only can be used to provide solid statistical data for bandwidth usage, but also trigger alerts when thresholds are passed, automate tracking more in-depth flows when certain traffic patterns are seen, as well as a variety of other features that are supported in some commercial products. Our major hurdle is cost justification. We have used NTOP as a collector, as well as a few other open source applications. While they do at least provide us with basic bandwidth usage and a picture of our traffic patterns across our transit path, I feel so much more could be done with this data that is simply not available in freeware offerings.

    I do not believe I am alone, as I know of a few other techs in the same boat. Someone with your knowledge and skill set could be incredibly helpful in either assisting people to get the most out of an open source offering(let’s just say NTOP for arguments sake), or in creating a well-documented application that supports many of these features. I think I can speak for several of my associates, as well as for myself, when I say we would be very grateful for any help you could provide.

    While on the subject, what free collector do you feel is the easiest to implement? Which do you feel has the most features? Which is the most customizable?

    I don’t mean to tie you up with questions. I would appreciate feedback from any of your readers that could help.

    Thanks for your time.

    • Aaron Paxson August 5, 2009 at 9:29 pm

      snetherland,

      All very good points! And all for the reasons why I want this project to succeed. Unfortunately, I am only one, and work is taking many of my hours!

      The project that I’m building is called Styx ( http://styx.javaforge.com ). As for the thresholds, I will be building a threshold correlator, and send events to my favorite network manager (OpenNMS – http://www.opennms.org ). This will allow threshold triggered events to be correlated to notifications, alarms, escalations, trouble-tickets, etc.

      I will be implementing a drools rules-based engine (eventually). Drools ( http://jboss.org/drools/ ) will allow you to build your own rules for forensic analysis like intrusion detection, or specific traffic patterns).

      This is obviously my wish list, and will take some time, but you have really helped me realize I should push through my hesitance, and just get it done!

      If NTop is not doing what you want, then you may want to look at flow-tools. It’s command-line based, and uses proprietary storage, but is flexible if you want to learn the commands.

      Styx will store all the data in SQL database. Obviously, this poses alot of problems with performance for reporting and ad-hoc querying, but am confident I can get passed the issues using OLAP reporting.

      I hope this helps! And keep up your ideas! If you know of any java developers that want to help, I could really use some help and guidance!

      Cheers!

  4. Jurgen Kobierczynski August 18, 2009 at 6:13 am

    Hmmm… I’ve build a NetFlow analyzer (JKFlow) myself and although it was a rewarding project, I don’t receive very much feedback anymore. Analyzings flow can be fairly complex and depends on your priorities.

    • Aaron Paxson August 18, 2009 at 7:47 am

      Hi Jurgen! Yes! I’ve looked at JKFlow. Unfortunately, not only am I not a Perl guy, but I also wanted to store my data in a more “easily accessible” fashion, i.e. via database SQL, rather than RRD.

      Unfortunately, this may be the piece that may kill my project, as SQL is not near as fast as local I/O.

      Thanks for the comment!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: