Tag: troubleshooting

Categories
Mikrotik Networking WISP xISP

mAP installer toolbox

One of the problems installers run into on a few networks we manage is having the right tools to properly test a new install. Sure, an installer can run a test to speedtest.net to verify customers are getting their speed.  Anyone who has done this long enough knows speedtest.net can be unreliable and produce inconsistent results. So, what then? Or what happens if you need to by-pass customer equipment easily? Most installers break out their laptop, spend a few minutes messing with settings and then authenticating themselves onto the network. Sometimes this can be easy, other times it can be challenging.

mAP with extenral battery pack

In steps the Mikrotik mAP.
What you are about to read is based on a MUM presentation by Lorenzo Busatti from http://routing.wireless.academy/ with my own spin on it. You can read his entire presentation on the mAP in PDF at : https://mum.mikrotik.com//presentations/US16/presentation_3371_1462179397.pdf . The meat of what we are talking about in this article starts on Page 50. If you want to watch the video you can do so at https://www.youtube.com/watch?v=VeZetH9uX_Y . The focus of this article starts around 21:00.

I have taken Lorenzo’s idea and have several different versions based upon the network.  In most of our scenarios, the ethernet ports are what plug into the CPE or the customer’s equipment, and the technician connects to the mAP over wifi.  This post covers using the mAP as an installer tool, not a traveling router. Lorenzo covers the travel option quite well in his presentation.

In this post, we focus on networks which use PPPoE. PPPoE networks usually are the ones who take much time to set up to diagnose.   What we have done is set up an uncapped user profile that is available on every tower.  Authentication can be done with local secrets or via radius.  Depending on your IP design the user can get the same IP across the network, or have an IP that assigned to this user on each tower/routed segment. We could do an entire article on IP design.

On our Mikrotik, we setup ether1 to have a PPPoE client running on it.  When the installer plugs this into the customers CPE the mAP will automatically “dial-out” and authenticate using the technician user we talked about earlier.  Once this connection has is established, the mAP is set to turn on the red “PoE out” light on the mAP using the following code.

/system leds
add interface=pppoe-out1 leds=user-led type=interface-status

Note. Our PPPoE interface is the default “pppoe-out1″ name. If you modify this, you will need to modify the led setup as well to match.

The red light gives the technician a visual indicator they have authenticated and should have internet. At the very least their mAP has authenticated with PPPoE. There are netwatch scripts mentioned in the above presentation which can kick on another LED indicating true internet reachability or other functions.  In our case, we can assume if the unit authenticates with the tower, then internet to the tower is up.  While this isn’t always the case if the Internet is down to the tower you quickly know or the NOC quickly knows.  At least you hope so. We chose the PoE out led because we are not using POE on this setup and a red light is noticeable.

Once the technician has a connection they can connect to an SSID set aside for testing.  In our case, we have set aside a “COMPANY_TECH” SSID. The tech connects to this on their laptop, and they are online.  Since this is a static profile, you can set it up just like a typical customer, or you can give the tech user access to routers, APs or other devices.  Our philosophy is you set up this SSID to mimic what a customer account experiences as closely as possible.  It goes through the same firewall rules and ques just like a typical customer.

To further enhance our tool we can set up a VPN.  This VPN can is accessible from the laptop with a second SSID named “COMPANY_VPN”. Once the technician switches over to this SSID they have access, over a preconfigured VPN on the mAP, to the network, from where they can access things customers can not, or at least should not be able to access. Many modern networks put APs, and infrastructure on separate VLANs not reachable from customer subnets.  The VPN comes in handy here. You can access these things without changing security. If you plan on using this router internally, the type of VPN you choose is not as important as if you plan to modify the config so you can travel as is the case with the above MUM presentation. If you plan to travel an SSTP VPN is the most compatible.  If it’s just inside your network, I would suggest an l2tp connection with IPsec.

Our third configuration on this is to set up the second ethernet port to be a DHCP client.  This setup is handy for plugging into the customer router for testing or for places where DHCP is the method of access, for example, behind a Baicells UE.  If your network does not use PPPoE, you could have one ethernet be a DHCP client, and the other be a DHCP server. We have found having the technicians connect wirelessly makes their lives easier.  They can plug the unit in and not have to worry about cables being too short, or getting behind a desk several times to plug and unplug things.

So why go through all this trouble?
One of the first things you learn in troubleshooting is to eliminate as many variables as you can. By plugging this into your CPE, you have a known baseline to do testing. You eliminate things such as customer routers, customer PCs, and premise wiring.  The mAP is plugged directly in CPE, whether it be wired or wireless. Experience has shown us many of the troubles customers experience are traced back to their router. Even if you provide the router, this can eliminate or point to that router as being a source of the problem if a technician needs to visit the customer.

Secondly, the mAP allows us to see and do more than your typical router. From the mAP we can run the Mikrotik bandwidth test tool from it to the closest router, to the next router inlines, all the way out to the internet. A while back I did an article titled “The Problem with Speedteststs“.  This article explains many of the issues testing just using speedtest.net or other sites.  Being able to do these kinds of tests is invaluable.  If there are four Mikrotik routers between the customer and the edge of your network all four of them can be tested independently. If you have a known good host outside your network, such as the one we provide to our clients, then you can also test against that. 

Having a Mikrotik test tool like this also allows you access to better logging and diagnostics.   You can easily see if the ethernet is negotiating at 100 meg or a Gig.  You can do wireless scans to see how noisy or busy 2.4GHZ is.  You have easy to understand ping and traceroute tools.  You also have a remote diagnostic tool which engineers can remote into easily to perform tests and capture readings.

Thirdly, the mAP allows the installer to establish a good known baseline at the time of install.  You are not reliant on just a CPE to AP test, or a speedtest.net test.

How do we make this portable?
You may have noticed in my above pictures I have an external battery pack hooked up to my mAP.   I am a fan of the Anker battery packs

Distributors such as ISP Supplies and CTIconnect have the mAP.

Finally, you will need a USB to MicroUSB cable

If you want you can add some double sided tape to hold the mAP to the battery pack for a neat package. I like the shorter cable referenced above in order to have a neat and manageable setup.

No matter what gear you use for delivering Internet to your customers, the mAP can be an invaluable troubleshooting tool for your field staff. I will be posting configs for Patreon and subscribers to download and configure their mAPs for this type of setup, as well as a road warrior setup. In the meantime, we do offer a setup service for $200, which includes the mAP, battery, USB cable and customized configuration for you.

Categories
Networking

The importance of Network Monitoring Systems (NMS)

One of our open tickets on MidWest-IX is a member reporting slow speeds on their exchange port. After having them send us some data and a few e-mails back and forth we began looking at their switch port on the fabric.  Right away we noticed errors on the port. After a counter reset the errors were still incrementing

 19 runts  0 giants  1210 CRC  0 no buffer
 1329 input error  0 short frame  0 overrun  0 underrun  0 ignored

This led us to look at our LibreNMS data for this port.  A quick look shows on October 31st the port started seeing input errors.

By dilling down we are able to see exactly when this started happening

We now have responded to the customer to see if anything changed that day. Maybe a new switch, new optic, or software upgrade.  By having this data available in an NMS we were able to cut down on troubleshooting by a huge margin.  We now know when the issue started and are closer to the root cause of this.  Without this data, we would be spending more time trying to diagnose and track down issues.

Categories
Networking Wireless WISP xISP

Why every ISP should be deploying hAP Lite to customers

So Mikrotik has a very cheap hAP Lite coming out.   This is a 4 port, 2.4 b/g/n router/access point which retails for $21.95. Baltic networks has pre-orders for $18.95.

Why should you deploy this little gem and how? We have found over the years routers account for more than half of the support issues. In some networks this number is closer to 80-90%. Whether it be a substandard router, one with out of date firmware, or poor placement by the customer.

Deployment of the hAP lite can be approached in one of two ways.  Both ways accomplish the same goal for the ISP. That goal is to have a device to test from that closely duplicates what the customer would see. Sure you can run tests from most modern wireless CPE, but it’s not the same as running tests m the customer side of the POE.

Many ISPs are offering a managed router service to their customers.  Some charge a nominal monthly fee, while others include it in the service.  This is a pretty straightforward thing.  The customer DMARC becomes the wireless router.  The ISP sets it up, does firmware updates, and generally takes care of it should there be issues.  The managed router can be an additional revenue stream in addition to providing a better customer experience.  Having a solid router that has been professionally setup by the ISP is a huge benefit to both the provider and the customer.  We will get into this a little later.

Second option lends itself better to a product such as the hAP lite. With the relative cheap cost you can install one as a “modem” if the customer chooses their own router option.  The actual method of setup can vary depending on your network philosophy.  You can simply bridge all the ports together and pass the data through like a switch.  The only difference is you add a “management ip” to the bridge interface on your network. This way you can reach it.  Another popular method, especially if you are running PPPoE or other radius methods, is to make the “modem” the PPPoE client.  This removes some of the burden from the wireless CPE onto something a little more powerful.   There are definite design considerations and cons for this setup.  We will go into those in a future article. But for now let’s just assume the hAP is just a managed switch you can access.

So what are the benefits of adding one of these cheap devices?
-You can run pings and traceroutes from the device.  This is helpful if a customer says they can’t reach a certain web-site.
-Capacity is becoming a larger and larger issue in the connected home.  iPads, gaming consoles, tvs, and even appliances are all sharing bandwidth.  If you are managing the customer router you can see the number of connected devices and do things like Torch to see what they are doing. If a customer calls and says its slow, being able to tell them that little Billy is downloading 4 megs a second on a device called “Billy’s xbox” can help a customer. It could also lead to an upsell.
-Wireless issues are another huge benefit.  If the customer bought their own router and stuck it in the basement and now their internet is slow you have a couple of tricks to troubleshoot without a truck roll.  If the hAP is in bridge mode simply enable the wireless, setup an SSID for the customer to test with and away you go.  This could uncover issues in the house, issues with their router, or it might even point to a problem on your side.
-Physical issues and ID10T errors can be quickly diagnosed.  If you can’t reach your device it’s either off or a cabling issue.  If you can reach the hAP and the port has errors it could be cabling or POE.

These are just a few benefits you can gleam from sticking a $20 Mikrotik device on your customer side network. It becomes a troubleshooting tool, which makes it money back if it saves you a single truck roll. The implementation is not as important as having a tool closer to the customer.  There several vendoars you can order the hAP lite from.  Baltic Networks is close to me so they are my go-to.  http://www.balticnetworks.com/mikrotik-hap-lite-tc-2-4ghz-indoor-access-point-tower-case-built-in-1-5dbi-antenna.html .

This isn’t practical for business and Enterprise customers, but you should already be deploying a router which has these features anyway right? 🙂

Categories
Bitlomat Mikrotik Wireless xISP

Capacity of a UBNT AP vs the number of clients

Almost all the time I get asked: “How many clients can an AP handle?” . My answer is always a very long and drawn out one. There is no set in stone answer. There are many factors which can affect this. I will go into some of these and then explain how to calculate this.

Some things that we will assume.
1.You are calculating on an 802.11N Ap with some kind of polling (TDMA, NSTREME, AIRMAX, etc)
2.You know the MCS values and/or data rates at channel widths.
3.When I say in an ideal situation I mean basically in the lab. This is our baseline. This means no outside noise, everything is working properly, and all the connected clients are excellent.

Before I get into what affects how many clients can an AP handle we need to shift our thinking a little. We don’t think in terms of how many clients can an AP handle. We need to think in terms of how much capacity an AP has. This is very important to think in these terms. If you do so things will become more clear and more quantifiable.

So now, on to what affects the total capacity of an AP.

1.The channel width. In and ideal situation you will get more Capacity out of a 20 mhz channel than you will a 10mhz channel.
2.Noise. In the real world you will have interference. If you have interference the noise floor drops, customer signals can’t reach maximum modulation, and there are retransmits.
3.Plain old signal. Things such as trees, distance, fresnel zone, and antenna gain all affect signal
4.The speed you are giving to each customer.
5.Overselling. The concept of overselling has been around since the dial-up days. You are betting your customers are not all online at the same exact time doing the exact same stuff. So you can oversell your capacity. I will explain this a little more in a bit how this factors in.

Okay, so let’s dive into this. I am going to use a Ubiquity Rocket M5 as an example. Again, this can be applied to any polling type N radio.

Say we have a Rocket M5. At a 20MHZ channel the best modulation this M5 will do is MCS 15 at 130 Megs of over the air. What do you mean Over the Air? Well there is a difference between actual throughput and the Wireless Data Rate (aka over the air). Your actual throughput/capacity will be 1/2 of the over the air rate minus a little for overhead. I factor in 10% overhead for easy figuring.

Back to our figuring. You have 130 megs of capacity on your AP in an ideal situation on a 20 mhz channel. If we do our math:
130 / 2 = 65 Megs of Capacity to sell on the AP.
Now here comes the overselling part.
If we oversell at a 2:1 ratio we have 130 Megs of capacity on the AP.
If we oversell at a 3:1 ratio we have 195 megs of capacity on the AP.

We can do higher ratios, but it starts to become a moving target. With the spread of Netflix, Youtube, Hulu, and other streaming services the average customer is sucking down more and more bandwidth for longer periods of time. Think of a restaurant with so many tables. If your customers are staying longer and longer, you don’t have as much seating capacity to turn over for new people to sit down and consume your food. This is for another blog post.

So, let’s say we are overselling at 3:1. We have 195 megs of capacity. We now need to think about what packages we are selling to our customers. If they are all say 5 meg packages, this means we can safely sell 39 connections to the AP. 195 / 5 = 39. You can figure up the math if you have 3 Meg, 10 meg, or a mixture.

Now to the real world (aka why do my customers hate me and my AP sucks?).

The following is a real AP in the wild.  Blacked out to protect the innocent from script kiddies.

ubnt-main-screen
Couple of things to Note (circled in Red).

20 MHZ Channel
Capacity at 45% . This is more important than anything, even CCQ.
43 clients associated.

Let’s apply our math we learned earlier. We know a 20 mhz channel nets us MCS15 – 130 Megs

Here is the kicker.  Our capacity is at 45%.  This means we only have 45% of 130 megs of Over the air capacity.  Take this in half (130 / 2= 65   45% of 65 = 29.25.
This means all 43 of these customers are sharing 29 megs of capacity on the AP.  And the quality isn’t the greatest (37%).  So this means there are retransmissions going on between the client and the AP. The client can’t talk as fast as it is capable of in most cases. This means you can’t oversell the AP as much due to the quality of the signals being poor.  It is important to note I am talking about the quality and capacity of the signals, not signal strengths.

If those 43 people are all paying for, let’s say, 2 Megs download.  That means your AP needs to support a minimum of 86 megs. Thats without overselling.  We only have 29 megs in the current state!

We need to get those capacity numbers up.  How do we do that?

1. Channel selection. A noisy channel will drag everyone down.

2. Antenna gain.  This can be done at both the client and the AP.  A higher gain or better quality antenna can cause the clients to “hear” better.  You might not get an increase in signal strengths, but you are looking for an increase in quality. I use a loudspeaker metaphor.  You can hear a loudspeaker from a far distance, but you might not always be able to make out what is being said.  If you can somehow make out what is being said more clearly, then you don’t have to have the speaker turn up the volume.

3. Shielding. This helps eliminate the amount of stuff a client or AP hears.

4. Channel Width.  Sometimes dropping the channel width down can increase signals, thus raising the overall capacity.  Keep in mind it will lessen the overall capacity of the AP.

5.Simply getting rid of customers that shouldn’t be installed.  We have all done installs that were iffy.  These can drag down the overall capacity.

I hope this has helped understand.  The biggest thing I want you all to take away from this is think in terms of the amount of capacity you have to sell, not the number of connections.