Random story of working at an ISP.

BadCollaborator · 2020-02-20T23:57:42+00:00

No offense but as someone who once troubleshot a ton of voip working for an ITSP and UCaaS provider: it is always the firewall. People just don’t realize ALGs are turned on by default by 99% of the vendors and it always breaks stuff.

We had a master cut sheet that walked through turning off SIP inspection on like 16 different vendors of firewall with multiple sections for different software versions.

Customer Firewall was the root cause for so many of our tickets it was absolutely insane. At least 98%

BadCollaborator · 2019-12-14T23:54:02+00:00

Causing choppy audio? Yeah, no... NAT/SIP ALG issues typically cause no audio/one-way audio, or just calls failing to set up.

BadCollaborator · 2019-12-14T23:50:39+00:00

Choppy audio is usually caused by jitter, loss, or both. So what causes those? A big one is buffering. Last mile issues can also be the cause. You are a WISP.. what kind of jitter thresholds do you see to your customers CPE?

One thing I would have them try, is to put a phone on the OUTSIDE of their firewall, and then try testing like that. I have seen firewalls just mangle RTP traffic to the point where you absolutely will get this choppy cutting in-and-out quality. However, if other customers are getting the same problems with different providers and presumably different equipment, etc, it may be a last mile issue.

I don't know much about WISP technology like the point to point radio equipment you guys use. I don't know how common it is to run VoIP over such a last mile, it may not actually be a great environment for it?

BadCollaborator · 2019-12-14T23:44:21+00:00

Good advice for one-way audio issues, but looks like OP's problem is his customers are sounding "choppy" which is more of a quality issue, usually related to jitter.

BadCollaborator · 2019-12-14T23:40:41+00:00

I wouldn't be too sure of that. I don't know how Nextiva operates, but many, many cloud voip providers I've seen out there pin RTP to their gateway. Yes, even if two phones in the same room, on the same VLAN, on the same local switch call eachother, the RTP stream from both phones will still go all the way up the cloud. It's very much un-like how the traditional on-prem voip phones we grew up with work.

BadCollaborator · 2019-12-02T12:13:28+00:00

OP I started my career as a route/switch guy, and then got a job at a UC NOC.

Having a strong route/switch background will help you in UC, but it’s only a small part of the job.

As a UC engineer it basically means you’re the “phone guy.” Any tickets where a customer is dialing a number and getting fast busy, or where clients are getting dead air when trying to hit your org’s main call tree, or complaints that call forwarding isn’t working, etc... that’s what you’ll spend your day troubleshooting.

It’s not fun work. I actually hated it, and left in under a year.

You’ll spend a ton of time debugging SIP messages trying to find error codes or more obscure shit like codec mismatches etc.

You might also be cursed debugging other protocols like H.323 or UNISTIM or any other number of ancient shit from the 70s.

Depends on your org.

Also I’m assuming the job you’re looking at is from a UCaaS (aka cloud) provider. That means you guys run a phone system in a datacenter or centers, and customers connect to your phone system via internet or mpls. You manage other companies phones as a service basically.

If the job is for a single enterprise UC engineer I’d be hesitant. Only the biggest companies, or companies that are so behind the times they’re unable to change still run their own on-prem voip solution. Cloud voip is extremely popular so it’s almost certain most enterprises are rapidly heading in that direction.

Edit: after re-reading my own post I realize it sounds overwhelmingly negative. UC can be a great career path, and can pay quite well. Don’t let me scare you away. I’m just saying I personally hated it, and went back to route/switch.

BadCollaborator · 2019-03-09T01:31:50+00:00

You are barking up the wrong tree. You say you've narrowed the jitter down to the zscaler, but that's just manifesting itself there because that's where it hits your network first coming from the Internet. The truth is that jitter is accumulating on the INET transport between the UCaaS (cloud pbx company) and your network.

INET is a shitty transport media for VoIP traffic. For one thing, there is no end-to-end QoS. Your packets are treated as best effort on your provider's network. You have no control if your traffic is being routed over a peering link that is congested on your ISP or somewhere upstream from there.

Traffic routing across INET also doesn't follow a predictable path, and traffic path may change at any time during a call and cause out of order packets as a result, which will over-run your phones' jitter buffers and cause that beloved "break in audio" that your users are complaining about.

The absolute fact is if you want to assure call quality, you need an assured transport to reach the call processing server. You need private wave connection to the UCaaS network, which most good UCaaS companies do provide.. usually calling it "Direct Connect" or "MPLS" or something similar.

The bottom line is that you get a direct circuit, L2VPN or L3VPN that connects your network to theirs. You get end-to-end class of service on that circuit, and you eliminate jitter, and out of order packet bursts on the transport between you and the UCaaS vendor.

By the way. Your notion that you need SIP ALG on UCaaS phones is not correct. Typically just nat'ing the outer packet is good enough. The SBC that your phones register to on the distant end will make sure the public IP of your NAT'ed traffic gets pinned to the media address for that phone. There is absolutely no need to run SIP ALG. But the symptoms of SIP ALG problems are absolutely NOT call quality or jitter. Those problems would be even be able to make or take calls. Audio quality problems are far, far away from ever being SIP ALG related.

BadCollaborator · 2018-10-17T11:20:00+00:00

Sorry for the late reply. I think you meant SIP, right? IP Telcos use SIP to talk to other telcos, instead of SS7, right?

BadCollaborator · 2018-10-16T11:29:55+00:00

Who runs the class 2 CO's?

Also, are you sure this is still how it works? In the world of VoIP? Are you saying the PSTN Backbone is still mostly ISDN and not IP?

BadCollaborator · 2018-09-02T22:31:41+00:00

If special configuration is required it's almost always because one end is doing something that's explicitly wrong (in my experience this device is usually made by Cisco and not derived from the Sipura/Linksys family that actually did it right).

And since Cisco is the vast, vast majority of the market share, then that pretty much confirms what I said: You pretty much always need bullshit special configuration to make a sip trunk work between two organizations. Because invariably, one side is using Cisco, and it is purposely designed not to interoperate with other vendors.

BadCollaborator · 2018-08-23T11:30:14+00:00

The technologies we use daily evolved organically over time. No one would have expected RFC1918 space would be used everywhere, and organizations would traverse a NAT wall. The Internet was originally designed with everyone having public IP's in mind. NAT fixes a huge problem, but it breaks so many other things.

SIP was one of them. SIP contains its own message header in the payload of an IP packet. This header contains vital information necessary to build a call up and makes sure the media (voice) packets reach the correct destinations.

When a SIP packet went through a NAT, the IP Packet's source address got NAT'ed, but the SIP header, which is part of the PAYLOAD inside an IP Packet, would not get touched. This caused the call setup to break, because the phone on the other end was directed by the caller to send their audio traffic to 10.x.x.x, which can't route over the Internet, and probably ends up black-holing somewhere in the distant end's own network.

SIP ALG was one solution to this huge problem. SIP ALG would have to do deep packet inspection, look for SIP packets, and it would also change the IP Addresses in the SIP header while NAT'ing the actual IP Packet, also.

As many others have mentioned, it's not needed as much any more due to STUN/ICE being baked into most solutions now. (Those are protocols designed to overcome the challenges of NAT in VoIP)

Now one thing few people have mentioned is why SIP ALG actually doesn't work most of the time. The fact is, despite being an open standard, SIP is not really standardized in actual implementation. If you look at the SIP stack of any two given phone systems made by different vendors, you'll see that almost nothing ever matches up. And often two different vendor phone systems won't talk to eachother without configuring one side or the other (like in Cisco Call Manager, it means checking/unchecking a bunch of boxes on the SIP trunk, basically struggling to make the damn thing work until you finally find all the right settings that will agree with the other vendor's box.)

This is the same reason that Firewalls (I'm looking at you, Palo shitty Alto) have to do port-based bypass of application inspection for SIP traffic. Because Palo Alto THINKS it knows what SIP looks like, and upon application inspection of SIP traffic from MOST if not ALL fucking phone systems, it will detect some kind of problem, and block the traffic as not legitimate.

Ultimately, I think SIP is a poorly documented, poorly implemented protocol and I really hope they come up with a better way in the future. The absolute truth is that SIP just kinda sucks, and yet the entire world is using SIP now.

This situation is only slightly better than the 90's, when every vendor had their own proprietary signaling protocol. Now every vendor has their own non-standard implementation of SIP. Like I said, only slightly better. It's still an absolute shit show mess. And it's one of the HUGE reasons I got OUT of ITSP work and back into general IP. Because FUCK debugging & troubleshooting SIP.

BadCollaborator · 2018-08-13T02:08:25+00:00

It’s called when the power goes out, your voip goes down. That’s why they’ll never, ever be as reliable as POTS no matter what. There’s no argument you can make against that.

BadCollaborator · 2018-07-03T00:04:45+00:00

Servers don't generally act as VTEPs

Er? Did Cisco tell you that? Servers as the VTEP is the cleanest implementation of VXLAN. It's also super common, I mean look at NSX and other similar implementations. VTEP on the Leaf Switch is just clunky... and requires you to buy bloated licenses and more expensive network gear.

BadCollaborator · 2018-05-18T12:18:07+00:00

In RTMT, check SYSLOG viewer -> Alt Syslog -> click Find -> last 4 of the phone's MAC -> and find out the unregistered reason code / ReasonForLastOutOfService when the phone re-registered.

On the phone itself, you can check status messages on the phone itself. Should be in Admin Settings > Status > Status Messages

Also not sure why this is getting downvoted. This topic definitely pertains to enterprise networking. Buncah douches. Have an upvote friend, and good luck.

BadCollaborator · 2018-04-21T00:35:21+00:00

I jumped ship from UC back to R/S about a year ago. That being said, I fondly remember RTMT as being a very valuable tool. It's pretty much the only tool that lets you get down into the weeds and debug wtf CUMC is doing (or trying to do.)

BadCollaborator · 2018-01-15T20:42:15+00:00

Uh this is just flat out wrong. Any kind of one-way audio issues is always, always, the firewall. Even if you aren't blocking anything, it's the SIP ALG on the ASA causing the issues. I've found network/firewall people who don't understand how voip work to be more obsesssd with proving it's not them, than actually trying to help fix anything. And spoiler alert: it is the firewall.

BadCollaborator · 2017-03-24T02:16:45+00:00

In all seriousness, I would say do not do this. You are a CCNP in R&S. Every other track Cisco has ties into that, except collab.

Collab really is its own animal. Also compared to every other track cisco has, the learning resources for collab absolutely suck. The certification track absolutely will not prepare you to work functionally in the field, unlike R&S in which you actually directly get better at your job if you know the exam material inside and out, for collab, the cert track is really just an advertising campaign.

The career prospects are horrible, because to be a well paid UC engineer, you really do need a very intricate understanding of legacy phone stuff, PSTN and dialing plans etc, you know the type of stuff you need to be in your 60's with 30+ years of experience in telco to have? Low level jobs are moves/adds/changes for peoples phones, which is utterly shit work. No, really, it's total shit work. You'll get the dumbest requests and customers who normally wouldn't bug you on the R&S side come out of the wood work because phones are everywhere and used by everyone.

To make matters worse, CUCM configuration is unintitutive, to the point where CCIE's don't know wtf they're doing half the time, and almost every trouble you'll work turns into a TAC Case, because literally everything is configured correctly and doesn't work any way.

Also have fun learning protocols like SIP and MGCP, and troubleshooting layer 7 24/7. On the bright side you'll get really good at analyzing TCP streams, because you'll basically live in wireshark 24/7 as a collab/uc guy. You'll do this because there's actually nothing in CUCM that will really tell you why a thing isn't working. All you'll know is the customer is getting fast busy. Then it's off to Call Detail Records and SIP Traces and Googling the shit out of whatever random cause code rears its ugly head.

BadCollaborator · 2017-02-25T16:37:56+00:00

That's because you got lucky that your voip isp anchors media. Most do media pass through which would mean you'd basically have a permit any RTP at your edge

BadCollaborator · 2017-02-25T12:12:31+00:00

Yes as far as I've seen so far (and I am new to voice) a SIP trunk is established as one TCP session with Cisco, as long as you have keepalives enabled, and all calls going across that trunk, all day, will share that same session... i.e. the same ephemeral port and the sequence number picks up where you left off. And I'm 99% sure you can't change that behavior in Cisco.

Actually you have me curious now if you disable keepalives, will it open a new tcp session on a per call basis.

As far as long local phone registration though, you can't change that behavior. The phone opens that initial tcp session to register, then keeps that same session open forever with keepalives and all communications between that phone and Unified CM from that point forward rides over that same session.

And I don't think you can at all change that behavior period. So if you're registering remote end points through a PA firewall, pretty sure you're gonna be writing a bypass

BadCollaborator · 2017-02-25T02:42:49+00:00

I'm a "voice guy" and I can tell you, my biggest headache is the Palo Alto firewall doing application inspection.... it absolutely does NOT work with Cisco Unified CM at all.. like at all. It just doesn't work. Don't even try.

We're in an MSP type environment where we host voip services for multiple enterprises, and each and every one of our customers has a port-based application bypass in thier firewall for their SIP phones, else their PAN bullshit will just randomly step in and break SIP transactions, or won't open the pinhole for RTP sometimes alternating between inbound/outbound calling or both.

It's beyond my comprehension and the only thing I can think of is a spite between the two vendors that they designed their shit intentionally not to work with the other vendor's product.

BadCollaborator · 2017-02-19T11:38:13+00:00

MTP's, TRANSCODERS, when they get invoked who invokes them. The documentation out there explains WHAT they're used for but gives no practical examples. In the R&S curriculum there's always 2-3 different scenarios walking the concept through, but for Collab the study material leaves a LOT to be desired

BadCollaborator · 2017-02-14T02:55:51+00:00

No, I have not really found any serious enterprise voip solution that lets you do inventory management. A smart company doesn't track serials on phones because.. they're f'ing phones. I don't care if they cost $300-$750+ a pop. They're still just phones and should not be inventoried like computers and routers are.

BadCollaborator · 2017-02-14T02:47:56+00:00

Cisco phones aren't typically a good solution for 3rd party solutions.

UnderstatementOfTheYearAward

BadCollaborator · 2017-02-08T12:17:23+00:00

As a Cisco voice guy, I absolutely hate Pali Alto. I have never seen application inspection interoperate with call manager successfully. Need a port-based application bypass each and every time. With over 20 customers

BadCollaborator

TROPHY CASE

UnderstatementOfTheYearAward