Xodium's Adventures In Business Networking
UPDATE! I've published a follow up to this in which I fix the problems I encountered here, hopefully for good. To put it simply, the problem was a rogue DHCP server, NOT the Connection Pro. Whoops.
Or: xodium gets in over his head, possibly.
This began several months ago, and everything seemed innocent enough on the surface: "Hey, xo, my work is switching to Comcast Business and my boss would love some help in executing the switchover + maybe some guidance in buying some new hardware since our computers need to be replaced."
Sure, why not. It's been a minute since I've done actual IT stuff, and the whole thing seemed simple enough on the surface. Just dismantle the remnants of their old AT&T service, install their new machines, and we're off to the races, right?
Ha. I wish.
I. Connection Pro No
Unfortunately, I was not present for when the details were being worked out for the Comcast service, and my client opted for Comcast's cellular backup, the Connection Pro. I can't prove it, yet, but I really in my heart feel this thing has been causing a lot of the pain and suffering, but maybe it's something else.
The reason why this is so regrettable is for my client's use, the Connection Pro is entirely useless. My client was worried about a Comcast outage taking out their ability to ring out customers and access records, but the problem is all their machines and printers need to be on a single, unified network. And while the 10Mbps speed of the Connection Pro (when it's on cellular) would be entirely sufficient for their uses, Comcast--far as I know--has a 3-device limit for the Connection Pro.
Which meant when I went to plug in their 16-port switch with a handful of machines on it, the Connection Pro said absolutely not and none of the devices behind it could reach the internet. Worse, because their credit card readers are Wi-Fi only and the Connection Pro explicitly does not cover wireless devices, well, as you can guess, this makes the Connection Pro entirely useless for my client's purposes. If Comcast goes down, enough of their network goes down that they're still stalled out, effectively.
(Plus, again, the Connection Pro fractures the network such that anything behind it wouldn't be able to interact with the printers not directly connected to it.)
Fine, whatever, we'll just skip the Connection Pro and move on without it, right?
I carried on and just bypassed the Connection Pro entirely, plugging their existing switch into the CBR (Comcast's abbreviation for the gateway they give to businesses), and that seemed to work just fine. I was mostly there to install their new computers, so I carried on with that, seeing that hey, they loaded Google's homepage, we should be good. I hope.
Nope. Next day rolls around and they can't access any of their records, because as a part of all this they moved all their record keeping to some cloud-based platform, and trying to access the login page would just time out. Just to make sure, I whipped out my phone, hotspotted to one of their machines, and tried. It worked.
I noticed right off that strange things were afoot. I could reach sites like Google, Facebook, Reddit and the like, but sites like Speedtest would just spin forever. Fast.com gave me an expected result (900ish Mbps, maxing out the gigabit connection) so the connection on a basic level seemed fine. Jumping into PowerShell and doing some quick pings, I noticed a strange occurrence: Pings destined for 192.168.1.1 (the gateway's IP) were getting bounced to 192.168.0.254. Huh?
I tried several things. Different browsers. Hell, I even tried using my laptop on their network and it exhibited the very same behavior. Tried everything short of factory resetting the CBR (because their phone system runs through it and I absolutely did NOT want to break that).
What ultimately fixed this was me thinking "hey, what if I just...set a manual IP, such that I get to tell the computer the default gateway is 192.168.1.1?" (And yes, I did try just setting the DNS servers to something other than Comcast's before doing this. It did not work.)
Sure enough, doing this caused pings destined for the gateway to land at the gateway, and also everything worked. I could now get to the login page for their cloud stuff, and they were able to log in and access everything as normal. Seeing as I needed to get them back up as fast as possible, I assigned every computer in the building a manual IP, made a note of it in their network closet, and that was that.
I don't know off the top of my head what caused all this, and why this was the fix. Like I said before my hypothesis is that because their internet plan has a Connection Pro attached for it, the CBR is trying to somehow account for it and mis-routing stuff. But as much as I want to say that's it, it simultaneously doesn't make sense, since Comcast expects you to plug in non-essential stuff to the CBR directly, even if the Connection Pro is present.
I do know Comcast was called on this (with notes provided by myself) and all they could say is "this is up to your IT department to fix, not us." (Which, uh, I guess I'm the IT department? Woo?)
They also refused to remove the Connection Pro, because a contract was signed, and therefore we just had to deal with it.
But eh. It would be fine, I guess. I had a means to get around it that 100% worked. Even the credit card processing machines could be manually assigned (when they didn't work either, go figure) so it seemed like the problem was fixed and that's the end of this post.
Nah, life ain't ever that easy.
II. Vendor Equipment, Yay
So usually, when I do gigs like these, I have a very strict "I do not mess with vendor stuff" policy. And by that, I mean I don't deal with software platforms and stuff that businesses use to do business stuff. My experience ends at the computer systems you use to access this stuff.
So, that fix we just did? It works great! But only when the device connecting to the network can be manually assigned (and told the right gateway IP). What I didn't count on is my client deciding to upgrade some equipment in the back rooms...to stuff that now depended on an internet connection to operate.
Worse yet, this equipment is extremely locked down. When first being set up there was no way to get in to configure it until it had an internet connection to activate it. And of course, the Comcast connection being what it is in the state that it's in...it ain't going to connect to its servers, not without being manually assigned.
We did get it to connect, albeit briefly, by doing something unorthodox and putting all the new equipment (there are like, four separate piece of vendor equipment, keep this in mind) on a router at the vendor's behest (a Netgear Nighthawk R7000) and connecting said router's WAN port to the Connection Pro's LAN ports. This worked for a few days before it just broke again.
This one...is one I've been slamming my head against for weeks now, thinking I've found the fix! But then that fix doesn't work and we're back at square one.
Attempt the first was to test the line with a cable tester. It checked out. I didn't bring any of my tools with me otherwise, so I was limited to just trying to reconnect different stuff and got absolutely nowhere. So it goes.
Attempt the second was just...taking their equipment hub (not to be confused with an ethernet hub) and testing it against various other connections in the building. Same with my laptop. I seemed to get better results if I bypassed the RJ45 coupler going into the back room, but this wasn't consistent. I decided to get a new coupler, but that didn't fix it. I tried plugging their hub directly into the cable coming through the wall. Nope.
I finally decided to just...disconnect the hub, drag it to their network closet, and connect it directly to the CBR. It started working. Well, shit. Looks like the cable going to the back room is going bad, looks like I'm going to have to either do a new cable run, OR set up a wireless bridge. Might as well do this simple first, so I made a note to set up a wireless bridge and bring it over.
Attempt the third went about as well as you'd expect. I thought I had the fix. It was the cable, right? Just bypass it and we're good, right?
If you said "not a damn chance", well...you'd be right.
I had some old TPLink range extender that would happily work as a wireless bridge. Not only does it extend wireless networks, it has an ethernet port on the bottom that it'll pipe a connection out to. Great! I got it set up on my home network to make sure it all worked, and everything seemed to check out. I put it aside, waiting for a good day to head over.
...and today was that day. I went over, laptop and bridge in hand, confident that finally, this nightmare would be over.
As the back room was kinda busy when I got there, I decided to just set up the bridge from scratch and do some checks to make sure it worked as intended. And it did. My laptop was getting a pretty decent connection through it, about 30Mbps both ways. Not the fastest, but fine enough for what they were doing. And because I was able to key in a static IP on the extender itself (as well as tell it the true default gateway), we shouldn't have any routing issues. I hope.
Right?
I got in, installed the bridge, and of course, nothing. All connection checks save the one to the NTP servers still failed. It was pulling a proper IP, and it being able to connect to the NTP servers told me it was still able to get SOME connection, but clearly it wasn't able to reach the vendor's servers (which use Microsoft Azure, for the record).
I tried pinging the servers myself behind the bridge. Nothing. WTF?
I tried connecting their router to the wireless bridge. Nope.
I tried connecting their hub to the bridge directly. Nope.
At this point and running out of ideas, I try one last thing. I reset the wireless bridge and this time tell it to connect to my phone's hotspot. I connect the bridge to their hub directly, and...green across the board. Huh.
I tried connecting the bridge to the vendor's switch, again. Nothing came up, and we were back to where we were again.
I connected the bridge back to the hub again, and...yep we're back to only being able to see the NTP servers, and nothing more. What...the...hell?
I connected my laptop back up to the bridge to make sure I still had connection, and I did, so that wasn't the issue. It was approaching closing time, so I had to give up here, and at this point came to the conclusion that I really believe this is something to do with the way this vendor's equipment interacts with their backend servers. It will connect sometimes, but then promptly lose the connection entirely and fail to reconnect.
I also wanted to suspect the switch they used to connect up all their equipment to the hub might be faulty, but I was unable to verify this. There was another spare switch on site but I couldn't find where the power adapter was stashed.
At this point I kinda just had to throw my hands in the air and give up. Maybe it might still be the cable going to the back room. But given what I just saw and given how it still wouldn't work even behind a wireless bridge, I'm inclined to believe it's something to do with the vendor's provided equipment. Either their switch, or their router. I'm sure Comcast's weird routing still plays a part at some point in this, but given it still had issues keeping a connection going when I put it on an entirely different internet connection...I'm assuming Comcast isn't solely to blame, here.
I suppose if I really wanted to be sure I should have toggled the VPN on my phone just to make double sure (as I have an OpenVPN instance set up to route back to my home connection). But I'm also not sure that would have changed anything, either.
This is where I'm at. No idea really where to go next with this. I have ideas, but given I have to work around their business hours for the most part and I have to muck about in a way that doesn't knock them offline...it's tough to truly try solutions in hope of a more permanent fix to things.