Categories
Data Center hosting Networking Security WISP

Homeland Security US-Cert e-mail on Network infrastructure

A few days ago Homeland Security published an e-mail on threats to network devices and securing them.  Rather than cut and paste I exported the e-mail to a PDF. Some good best practices in here.

TA16250A The Increasing Threat to Network Infrastructure Devices and Recommended Mitigations

Categories
UBNT WISP xISP

Random UBNT EdgeSwitch 16 interface photos

Categories
Networking Wireless WISP

Ignitenet 60GHZ Interface shots

Categories
Mikrotik

Mikrotik Router OS 6.36.2

To upgrade, click “Check for updates” at /system package in your RouterOS configuration interface, or head to our download page: http://www.mikrotik.com/download

v6.36.2 forum topic discussion,
http://forum.mikrotik.com/viewtopic.php?f=21&t=111450

What’s new in 6.36.2 (2016-Aug-22 12:54):

*) arm – show cpu frequency under resources menu;
*) capsman – fixed upgrade policy;
*) ccr/crs – fixed SFP+ interface ddmi info reporting function. Info is now refreshed on regular intervals;
*) conntrack – fixed ipv6 timeout display;
*) conntrack – fixed removing icmpv6 connections;
*) dns – avoid unnecessary dynamic server address saving in storage;
*) dns – allow to set query-server-timeout and query-total-timeout only greater than 0s;
*) dns – fixed lockup when dynamic dns server address 0.0.0.0 was received;
*) export – updated default values in /system routerboard settings menu;
*) partitions – fixed crash on repartition when there is not enough free space;
*) sstp – fixed disconnects on transmit for multicore systems;
*) switch – fixed configuration reload on CRS switches;
*) winbox – make queue tree default queue type default-small;

Categories
Data Center Networking UBNT

Ubiquiti EdgeSwitch 16XG First look

So today UPS dropped off a brand new EdgeSwitch 16XG.  I won’t bore you with all the cool stats.  You can read the official product literature here.  This is just a first look.  Future posts will dive into configuration, testing, and other such things.  For those wanting the cliff notes version of what this switch is about:

  • (12) SFP+ Ports
  • (4) 10G RJ45 Ports
  • (1) RJ45 Serial Console Port
  • Non-Blocking Throughput: 160 Gbps
  • Switching Capacity: 320 Gbps
  • Forwarding Rate: 238.10 Mpps
  • (12) 1/10 Gbps SFP+ Ethernet Ports
  • (4) 1/10 Gbps RJ45 Ethernet Ports
  • Rack Mountable with Rack-Mount Brackets (Included)
  • DC Input Option (Redundant or Stand-Alone)

UBNT EdgeSwitch 16XG

UBNT is following a natural trend in the switch world.  As more and more networks are looking at 1Gig being their minimum, the switches are reflecting this.  Gone are the days of 10/100 ports. Now are going toward 1/10 gig ports, even on copper. 10/100/1000 switches still have their place, but usually not on switches with 10 gig ports.

IMG_3910
Accessories included with the 16XG

Out of the box the switch isn’t anything sexy.  I feel like it should have a shiny UBNT logo somewhere.

UBNT EdgeSwitch 16XG

I like the fact that none of the ports are shared ports.  You can use all 16 ports. It always annoys me when I buy a switch and can’t use all the ports because they are shared on the bus.

An interesting feature on this switch is a redundant DC input option.  This can be anything from 16-25volts and be able to support 56watts. This results in a minimum of a 2.2 Amp power supply.   This is assuming a full load on the switch as well. For the WISP market this could be a very handy option.  You could install the switch where it is drawing from AC power but in the event of AC outage it will switch to a DC source.  One of my questions to UBNT is if you can run it off total DC.

UBNT Redundant Power
DC Input Jack

Now on to some nitpicky design things.  None of these really affect the performance of the switch, just are annoyances.

-The console port not being on the front. In today’s dense rack environments we are putting patch panels and Transfer switches in the backs of the rack.  If we have to get to the back of the front mounted devices then anything other than power becomes an annoyance. This is not an issue if you install every new switch with a console cable back to a console server like we do, but even that doesn’t always happen.

Ubnt 16XG Back

-The SFP cages should stick out just a tad from the front.  During inserting and re-inserting SFPs I actually pushed the cage back a little.  This resulted in some of the SFPs not clicking in correctly.  The little tabs holding the top of the SFP cages aren’t sturdy enough to hold some repeated clicking in and out.

EdgeSwitch 16XG SFP

After seeing this I was prompted to open the switch and see what is under the hood.

Inside of EdgeSwitcvh 16XG

I think this will be a hugely popular switch for anybody looking to do 10Gig. At a $600 approximate price these are, by far, the most cost effective 10 Gig switch out there.  Many manufacturers have tacked on one or two, sometimes 4 SFP+ ports, but if you need to go beyond that you are talking 4 digit pricing.  This is something we have struggled with MidWest-IX. It usually leads to us buying something on the used market that has the port density we need.

There you have it for a first look at this switch. More articles to follow that include:
-Interoperability Testing
-Configuration tasks
-Questions I and you, the reader, have for UBNT

Categories
Mikrotik

Mikrotik RouterOS 3.36

Lots of things fixed in this release.

What’s new in 6.36 (2016-Jul-20 14:09):

*) arm – added Dude server support;
*) dude – (changes discussed here: http://forum.mikrotik.com/viewtopic.php?f=8&t=110428);
*) dude – server package is now made smaller. client side content upgrade is now removed from it and is downloaded straight from our cloud. So workstations on which client is used will require access to wan. Alternatively upgrade must be done by reinstalling the client on each new release;
*) firewall – added “/interface list” menu which allows to create list of interfaces which can be used as in/out-interface-list matcher in firewall and use as a filter in traffic-flow;
*) firewall – added pre-connection tracking filter – “raw” table, that allow to protect connection-tracking from unnecessary traffic;
*) firewall – allow to add domain name to address-lists (dynamic entries for resolved addresses will be added to specified list);
*) wireless – wireless-fp is discontinued, it needs to be uninstalled/disabled before upgrade;
*) address – allow multiple equal ip addresses to be added if neither or only one is enabled;
*) address-list – make “dynamic=yes” as read-only option;
*) arm – fixed kernel failure on low memory;
*) arp – added arp-timeout option per interface;
*) bonding – fixed 802.3ad load balancing mode over tunnels ;
*) bonding – fixed bonding primary slave assignment for ovpn interfaces after startup;
*) bonding – fixed crash on RoMON traffic transmit;
*) bonding – implemented l2mtu value == smallest slave interfaces l2mtu;
*) capsman – fixed crash when running over ovpn;
*) certificate – added automatic scep renewal delay after startup to avoid all requests accessing CA at the same time;
*) certificate – cancel pending renew when certificate becomes valid after date change;
*) certificate – display issuer and subject on check failure;
*) certificate – do not exit after card-verify;
*) certificate – force scep renewal on system clock updates;
*) chr – fixed CHR seeing its own system disk mounted as additional data disk;
*) clock – fixed time keeping for SXT ac, 911L, cAP, mAP lite, wAP;
*) clock – save current time to configuration once per day even if there are no time zone adjustments pending;
*) cloud – fixed export order;
*) console – fixed get false function;
*) console – show message time in echo log messages;
*) defconf – changed channel extension to 20/40/80mhz for all ac boards;
*) dhcp-pd – correct server listing for commands;
*) dhcp-server – fixed radius framed route addition after reboot on client renew;
*) dhcpv6-client – fixed ia lifetime validation when it is set by dhcpv6 client;
*) dhcpv6-relay – set packet link-address only when it is manually configured;
*) dhcpv6-server – fixed binding last-seen update;
*) disk – added support for Plextor PX-G128M6e(A) SSD on CCR1072;
*) email – fixed send from winbox;
*) email – removed subject and body length limit;
*) ethernet – fixed incorrect ether1 link speed after reboot on rb4xx series routers;
*) ethernet – fixed memory leak when setting interface without changing configuration;
*) fastpath – fixed kernel failure when fastpath handles packet with multicast dst-address;
*) fetch – support tls host name extension;
*) firewall – added udplite, dccp, sctp connection tracking helpers;
*) firewall – do not show disabled=no in export;
*) firewall – fixed spelling in built-in firewall commentary;
*) gps – fixed longitude seconds part;
*) health – fixed broken factory voltage calibration data for some hAP ac boards;
*) health – fixed incorrect voltage after reboot on RB2011UAS;
*) icmp – fixed kernel failure when icmp packet could not be processed on high load;
*) ippool6 – fixed crash on acquire when prefix length is equal with pool prefix length;
*) ipsec – add dead ph2 detection exception for windows msgid noncompliance with rfc;
*) ipsec – added dead ph2 reply detection;
*) ipsec – don’t register temporary ph2 on dead list;
*) ipsec – fix initiator modecfg dynamic dns;
*) ipsec – fixed AH with SHA2;
*) ipsec – fixed checks before accessing ph1 nat options;
*) ipsec – fixed mode-config export;
*) ipsec – fixed route cache overflow when using ipsec with route cache disabled;
*) ipsec – fixed windows msgid check on x86 devices;
*) ipsec – show remote peer address in error messages when possible;
*) ipsec – store udp encapsulation type in proposal;
*) kernel – fixed possible kernel deadlock when Sierra USB mode is being used;
*) l2tp – fixed crash when rebooting or disabling l2tp while there are still active connections;
*) lcd – reduced lowest backlight-timeout value from 5m to 30s;
*) license – do not expire demo license right after fresh installation of x86;
*) log – added whole scep certificate chain print;
*) log – increase excessive multicast/broadcast warning threshold every time it is logged;
*) log – make logging process less aggressive on startup;
*) lte – added allow-roaming option for Huawei MU709, ME909s devices;
*) lte – added cinterion pls8 support;
*) lte – added support for Huawei E3531;
*) lte – added support for ZTE ZM8620;
*) lte – added use-peer-dns option (will work only combined with add-default-route);
*) lte – changed driver loading for class 2 usb rndis devices;
*) lte – display message in lte,error log if no response received;
*) lte – display message in lte,error log when PIN is required;
*) lte – fix crash on SXT LTE while resetting card while at high traffic;
*) lte – fixed access technology logging;
*) lte – fixed connection for Huawei without cell info;
*) lte – fixed modem init when pin request present;
*) lte – fixed modem network configuration version checks;
*) lte – fixed network-mode support after downgrade;
*) lte – Huawei MU609 must use latest firmware to work correctly;
*) lte – improved multiple same model modems identification;
*) lte – show uicc for Huawei modems;
*) lte – use only creg result codes as network status indications;
*) mesh – fixed crash when connection references a mesh network but it is not available any more;
*) modem – added support for Alcatel OneTouch X600;
*) modem – added support for Quectel EC21 and EC25;
*) modem – added support for SpeedUP SU-900U modem;
*) nand – improved nand refresh feature to enhance stored data integrity;
*) ovpn – enable perfect forwarding secrecy support by default;
*) ovpn – fixed compatibility with OpenVPN 2.3.11;
*) pppoe – allow to set MTU and MRU higher than 1500 for PPPoE;
*) pppoe – do not allow to send out bigger packets than l2mtu if mrru is provided;
*) proxy – limit max ram usage to 80% for tile and x86 devices;
*) queue – reset queue type on interfaces which default queue type changes to no-queue after upgrade;
*) rb2011 – fixed ether6-ether10 flapping when two ports from both switch chips are in the same bridge;
*) rb3011 – fixed port flapping on ether6-ether10;
*) rb3011 – fixed reset button functionality;
*) rb3011 – fixed usb driver load;
*) rb3011 – fixed usb storage mounting;
*) rb3011 – improved performance on high cpu usage;
*) route – added suppport for more than 8 bits of options;
*) route – fixed ospf by handling ipv6 encoded prefixes with stray bits;
*) sniffer – fixed ipv6 address matching;
*) snmp – fixed get function for snmp>=v2 when oid does not exist;
*) snmp – fixed interface stats branch from MikroTik MIB;
*) snmp – report current access technology and cell id for lte modems;
*) snmp – report ram memory as ram instead of other;
*) ssh – add rsa host key size parameter;
*) ssh-keygen – add rsa key size parameter;
*) ssl – do not exit while there still are active sessions;
*) ssl – fixed memory leak on ssl connect/disconnect (fetch, ovpn, etc.);
*) sstp – fixed dns name support in connect-to field if http-proxy is specified;
*) supout – erase panic data properly on Netinstall;
*) switch – fixed switch compact export;
*) timezone – updated timezone information from tzdata2016e release;
*) traffic-flow – added ipfix support (RFC5101 and RFC5102);
*) tunnel – added option to auto detect tunnel local-address;
*) tunnel – fixed rare crash by specifying minimal header length immediately at tunnel initialization;
*) upnp – fixed nat rule dst-port by making it visible again;
*) usb – I-tec U3GLAN3HUB usb hub/ethernet dongle now shows up correctly as ethernet interface;
*) usb – implement possibility to recognize usb hubs/ethernet-dongles (if usb hubs/ethernet-dongles are not recognized with this version – send supout.rif file);
*) userman – fixed crash on database upload;
*) userman – use ipnpb.paypal.com for payment verification;
*) wap-ac – fixed performance problems with 2.4GHz wireless (additional reboot after upgrade required);
*) webfig – do not allow to press OK or Apply if current configuration values are not loaded yet;
*) webfig – reduced refresh time for wireless registration table to 1 second;
*) winbox – added 2ghz-g/n band for wireless-rep;
*) winbox – added icons to bridge filter actions similar to ip firewall;
*) winbox – added support for ipv6 dhcp relay;
*) winbox – allow to reorder hotspot walled-garden & walled-garden-ip rules;
*) winbox – do not allow to specify vlan-mode=no-tag in capsman datapath config;
*) winbox – do not show filter for combined fields like bgp-vpn4 RD;
*) winbox – do not show mode setting for WDS interfaces;
*) winbox – fixed crash on disconnect in secure mode;
*) winbox – fixed crash when using ctrl+d;
*) winbox – fixed safe mode;
*) winbox – improve filtering on list fields;
*) winbox – report correctly dude users in active users list;
*) winbox – set default sa-learning value to “yes” for CRS Ingress VLAN Translation rules;
*) winbox – show action column as first in bridge firewall;
*) winbox – show error when telnet is not allowed because of permissions;
*) wireless – fixed multiple wireless packages enabled at the same time after upgrade;
*) wireless-rep – added initial API support for snooper;
*) wireless-rep – fixed crash on nv2 reconnect;
*) wireless-rep – fixed scan-list unset;
*) wireless-rep – treat missing SSID element as hidden SSID;

Categories
Mikrotik

An open letter to Mikrotik about bug fixes

This isn’t your typical “rag on Mikrotik” post.  I see some frustrations with the Mikrotik process, mainly in regards to getting ongoing bugs and issues fixed. Having a persistent bug continue for large amounts of times tends to make for a frustrating experience.  Mikrotik has made leaps and bounds in their Changelogs over the past couple of years, which has been a huge help in the decisions of what software versions to upgrade (or even downgrade) to.  But I think things get lost in the process. This results in ongoing bugs, which tend to get unburied if someone makes enough noise.

One of the biggest things I would like to see is a public bug tracking system like Redhat’s Bugzilla tracking system.  This would benefit the community as a whole and help users see some of the outstanding issues when they go to implement things.  Forums are a great tool, but due to the nature of them, you get a fair amount of mis-information and unrelated chatter.  Just because Joe says he is seeing a bug, doesn’t mean he has a confirmed bug.  Having a confirmed bug system that has information and able to have moderated comments would be beneficial in many ways:
1.Users with long term bugs they are experiencing or waiting on would be able to keep informed on open status of bugs.
2.Would cut down on the “non-scientific” nature of forums. Information could be specifically submitted in support of a confirmed bug. Bug reports normally include the conditions that need to be met or existing for the bug to manifest itself. Users can then confirm, under those specific conditions, if they are experiencing a certain bug.
3.Bugs that are important to users will get reported more often. This should lead to the more important bugs being upvoted by the community thus getting them fixed earlier. If your particular bug has low numbers you have a reference as to why it’s not being addressed in a timely manner. Companies have to give resources to places they get the most bang for the buck.

Not only would this keep Mikrotik accountable, but it would keep the community accountable.  Properly reporting bugs and reproducing them is a process. It takes effort on both the user and the developer. In the end, it makes for a better product.

I have the utmost respect for Mikrotik and their staff.  Several folks there I consider friends. I think, before growing pains get too out of hand some sort of additional feedback options would be helpful for the community at large. Mikrotik is getting there. Things like making bug fix versions and release-candidate versions available, along with changelogs has been a huge help for planning and just keeping up on what’s being addressed.

What prompted this was I had a client over the past weekend who started having OSPF issues. Many hours of troubleshooting later, and only talking to some other folks who were seeing the same issues, I was able to determine a specific RouterOS version was to blame.  Being able to attach data to a specific bug report, or having Mirkotik open up a new bug based on information I submitted would have been a great help to others.  A forum of blog post would have been too general. Forums posts also tend to bring out the “I am seeing that too” and they are not meeting the same conditions you are.

Mikrotik implement a bug tracking system! Bugzilla is even on GitHub.

Categories
Networking WISP xISP

Leap second to be added in 2016

http://www.space.com/33361-leap-second-2016-atomic-clocks.html

A “leap second” will be added to the world’s official clocks on Dec. 31 at 23 hours, 59 minutes and 59 seconds Coordinated Universal Time (UTC), which corresponds to 6:59:59 p.m. EST; the clocks will read 23:59:60 before ticking over to midnight. The goal is to keep two different timescales in sync with each other.

So, why is this important to you as an ISP?
The trouble is that even as they use the leap second, UNIX and Linux define a day as something that is unvarying in length. “If a leap second happens, the operating system must somehow prevent the applications from knowing that it’s going on while still handling all the business of an operating system,” says Steve Allen, a programmer with California’s Lick Observatory.
Source:http://www.wired.com/2015/01/leap-second-rattle-internet-theres-plot-kill/

Many patches and fixes have been put in place to adjust for this in most Operating systems.  The concern, even a year after the last one, is software that has not been updated to account for dealing with leap seconds.

Categories
BGP Networking

How I learned to love BGP communities, and so can you

BGP communities can be a powerful, but almost mystical thing.  If you aren’t familiar with communities start here at Wikipedia.  For the purpose of part one of this article we will talk about communities and how they can be utilized for traffic coming into your network. Part two of this article will talk about applying what you have classified to your peers.

So let’s jump into it.  Let’s start with XYZ ISP. They have the following BGP peers:

-Peer one is Typhoon Electric.  XYZ ISP buys an internet connection from Typhoon.
-Peer two is Basement3. XYZ ISP also buy an internet connection from Basement3
-Peer three is Mauler Automotive. XYZ ISP sells internet to Mauler Automotive.
-Peer four is HopOffACloud web hosting.  XYZ ISP and HopOffACloud are in the data center and have determined they exchange enough traffic amongst their ASN’s to justify a dedicated connection between them.
-Peer five is the local Internet exchange (IX) in the data center.

So now that we know who our peers are, we need to assign some communities and classify who goes in what community.  The Thing to keep in mind here, is communities are something you come up with. There are common numbers people use for communities, but there is no rule on what you have to number your communities as. So before we proceed we will need to also know what our own ASN is.  For XYZ we will say they were assigned AS64512. For those of you who are familiar with BGP, you will see this is a private ASN.  I just used this to lessen any confusion.  If you are following along at home replace 65412 with your own ASN.

So we will create four communities .

64512:100 = transit
64512:200 = peers
64512:300 = customers
64512:400 = my routes

Where did we create these? For now on paper.

So let’s break down each of these and how they apply to XYZ network. If you need some help with the terminology see this previous post.
64512:100 – Transit
Transit will apply to Typhoon Electric and Basement3.  These are companies you are buying internet transit from.

64512:200 – Peers
Peers apply to HopOffACloud and the IX. These are folks you are just exchanging your own and your customer’s routes with.

64512:300 – Customers
This applies to Mauler Automotive.  This is a customer buying Internet from you. They transit your network to get to the Internet.

64512:200 – Local
This applies to your own prefixes.  These are routes within your own network or this particular ASN.

Our next step is to take the incoming traffic and classify into one of these communities. Once we have it classified we can do stuff with it.

If we wanted to classify the Typhoon Electric traffic we would do the following in Mikrotik land:

/routing filter
add action=passthrough chain=TYPHOON-IN prefix=0.0.0.0/0 prefix-length=0-32 set-bgp-communities=64512:100 comment="Tag incoming prefixes with :100"

This would go at the top of your filter chain for the Typhoon Electric peer.  This simply applies 64512:100 to the prefixes learned from Typhoon.

In Cisco Land our configuration would look like this:

route-map Typhoon-in permit 20  
match ip address 102  
set community 64512:100

The above Cisco configuration creates a route map, matches a pre-existing access list named 102, and applies community 64512:100 to prefixes learned.

For Juniper you can add the following command to an incoming peer in policy-options:

set community Typhoon-in members 64512:100

Similar to the others you are applying this community to a policy.

So what have we done so far, we have taken the received prefixes from Typhoon Electric and applied community 64512:100 to it.  This simply puts a classifier on all traffic from that peer. We could modify the above example to classify traffic from our other peers based upon what community we want them tagged as.

In our next segment we will learn what we can do with these communities.

Categories
Mimosa

Mimosa G2 First Look Video