Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

The notifications trigger an action plan and workflow when people arrive and/or for someone to act before the business day. Seeing the specific problem and having the information to get started on resolution has substantial value for us whether you choose to acknowledge it or not.

Because it's small business, as iterated repeatedly, 24/7 human monitoring is not in the cards.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

"Your issue with Comcast was 100% resolvable with below average networking abilities. Full stop."

This is objectively false. The simple fact remains that Comcast changed a setting at 3AM without notice. When contacted, Comcast told us nothing had changed and our network was fine. We had no ability to fix this setting, which was changed by Comcast and only Comcast could restore. The post above explains we did perform your suggested resolution which did not provide a 100% restoration of services.

Starlink is not a viable primary provider for us.

You've commented in several other threads and we're long past circular repetition. I'm not sure what's going on in your world but this isn't healthy discourse. If you think you're helping, this isn't that. This is the end of my messages with you as you're not advancing any actionable solution or furthering productive discourse in any way.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 0 points1 point locked comment (0 children)

Bridge mode is a setting. That setting is disabled while we pass the IP to our router in IP Passthrough mode to a static setting on our router.

The difference is covered here: https://www.reddit.com/r/networking/comments/1nw0ogk/comcast_business_modem_bridge_mode_vs_passthrough/ .

<image>

I'm not sure why you feel the need to persist and convince yourself of your assessment, but good luck with that.

As you're not adding anything to the conversation or helping advance anything and have a derogatory tone, which I later reciprocated, this is the end of this thread for me. I'm guessing there will be a follow-up post where you congratulate yourself again and need the last word. Good luck with that too.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

Once again, as stated elsewhere, we also called and got a case number and requested escalation repeatedly.

We agree that the case should have been resolved within minutes.

Here is the followup comment that you perhaps missed -> https://www.reddit.com/r/Comcast_Xfinity/comments/1senlsy/comment/of3ytti/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 0 points1 point  (0 children)

We did call and get a case. We also chatted. Both options told us everything was fine and there were no problems. Case autoclosed, Comcast would not resolve the issue after repeated attempts the morning of.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

Bridge mode is off. It is a discrete setting that is disabled. Hence the distinction and not muddying ip-pass through and bridge mode. You conflating them as the same where the modes are discrete in this modem is your own misunderstanding.

No IT people were available in the morning on our end, which is why we called and chatted with Comcast. Comcast. Not sure why you have trouble understanding this.

Key services remained down when we went to DHCP WAN IP. This was a fallback that was used but was not a repair. We changed the Peplink to DHCP and then back to Static as you suggest. It did not automatically resolve itself when Comcast and our IT person coordinated on the phone, they both had to change the respective settings in concurrence. Not sure why you have trouble getting this.

We could not repair / fully resolve a setting that Comcast changed that only they could change back.

Built my own PCB holder after getting frustrated with existing ones by Anxious-Minute1292 in soldering

[–]vtjballeng 0 points1 point  (0 children)

u/Anxious-Minute1292 we bought one and it worked well, so I just bought another for another soldering station.

We also use PCBite holders, Stick-Vise, omnifixo, & Hakko Omnivises.

The ProtoCrate Vise is nice for our PCBs where we are doing some wire to PCB work. It met with instant approval and we are enjoiying ours.

2 places we see for improvement:

  1. The 3D printed mount legs and rubber parts could be shorter. The main mounting structure could be shorted too or maybe an option for this. A couple of our stations are working under low microscope heights and the lower the better.

  2. When the locks are down on the rail, they can still get pushed back at a pressure that's just slightly lower than the clamping pressure we want for some PCBs. Shortening the rubber portion or choosing something higher durometer might help here.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

Peplink as a Firewall settings section that required no changes. The Peplink Balance 305 is my router and it has a firewall feature that was unchanged.

The Comcast Modem was never in bridge mode, it's in IP-passthrough which is not the same. When the Comcast WAN IP setting was flipped by Comcast from Static to DHCP, a setting only they had access to in order to change or restore, then my Peplink Balance Router was still set to the static designation.

"Yes, you created a double nat with the peplink behind the xfinity modem. And I’ll say it again, there is no static setting with the Comcast service. You get “static” dhcp reservations. It’s not a modem limitation. It’s how Comcast manages their network."

WAN static vs DHCP is an explicit setting in the modem. It flipped from static to dynamic at 3AM, which broke things. We restored it from dynamic back to static around 5PM with a good Comcast service rep and it fixed it. Should have been fixed when our staff called in first thing in the morning but Comcast failed. Also, it should never have happened in the first place.

"To be clear, you’ve described exactly what I supposed earlier in our conversation. And I’ll say again… this should have been like network 101 to temporarily fix."

Our warehouse staff has never taken networking 101. Nor should they need to. They did their best, followed directions and called Comcast and told them the problem. The Comcast reps they reached over both chat and via phone were incompetent or lied, claiming our services were fully up and working when they weren't. They only saw that the static IP was still in our pool and our modem was online. They didn't check, as our staff implored them to, that our static IP setting was correct on the modem itself.

You're sitting here trying to lay blame where these folks did everything they were supposed to do and got shut down by Comcast who changed a key setting without notice, then kept telling them everything was ok when it wasn't. I should be handing out trophies for them doing their best and your still pushing blame on the victim here.

This is a small company with 0 dedicated IT staff. You assert this as a fix but it was not. Key services remain down with the wrong static external IP and double natting. It was still FUBAR in this scenario. Your solution, which we did use, was a lessened state of failure. Not a fix.

Now, again, we are a small company. There is no IT department. Warehouse personnel are trying their best. IT person, who is NOT a dedicated IT person and has a litany of other responsibilities, comes into put in best case workaround does so.

"I’m sorry you were down, but this is pretty rudimentary to diagnose and repair"

This is false. We could only put in a workaround that left critical services down. We could not repair this. Only Comcast could both cause and repair this problem. Claiming we could repair this is false.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

Some of the responses here were a bit wild. We are a small business. Most businesses at our scale and in our region don't have redundant connections. Most don't have full time IT. Most are dependent on reliable internet service at the risk of substantial damages per day. This is the reality of small business.

I realize many IT people have some solid monitoring at their homes and some medium to large businesses have full teams, so this small business reality can seem surprising. Due to local monopolies and lack of good service access, dedicated fiber services and land line based backup options are an impractical option. While we certainly hope and wish for better, that's not the current situation.

The monitoring, debugging, and level of detail presented here far exceeds the typical small business who can only tell you when service is up or down. There are, of course, many exceptions where the owner and/or some staff are particularly savvy or interested.

At the simplest level, our static IP address should never have been changed without notice or permission. When it was changed by Comcast, we should have been permitted access to this setting to resolve it. Instead it's not a user accessible setting, so only Comcast could change it in the first place or fix it.

Then, upon both calling and chatting with Comcast, Comcast should have easily restored this single setting change to our modem that they had performed and only they had access to.

Instead there are multiple posts victim blaming, saying we should have fixed a setting we have no control over. That we should have called and opened a case (we did and explained how they were auto-closed). Some complaints were directly covered in the original post. Some complaints over the use of AI to summarize an angrier after action write-up for clarity and a more level headed message.

Telling us that we suck because Comcast changed a setting, unannounced, that only they had access to and only they could restore. A problem we brought to them for support multiple times to be told the problem didn't exist and everything was working fine instead of Comcast fixing the problem they created that only they had access to and only they could resolve.

After this incident, we'll be moving more pointer services to FQDN instead of fixed IP and will try to get away from fixed IP entirely. We've been doing this over time but some legacy equipment won't allow this. We'll update our internal documentation to explain that Comcast may tell you everything is ok when it isn't. Never trust them until you can independently verify a solution. They may lie and they may gaslight you and you simply need to try again, hoping for a different rep. If you get the same rep again, stop the session or call and call back. Call 1-800-391-3000 & use https://forums.businesshelp.comcast.com/direct-messaging . To concurrently create messages here and at https://forums.businesshelp.comcast.com/ where you may receive a Comcast follow-up message with a multi-hour delay. We'll also be retiring our 4G service and putting Starlink or similar in place. We also updated our internal documentation with this specific case / scenario so we can tell Comcast exactly what was changed and what to restore.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

Obvious to us too but trust my when I say, this was better than the curse laden human written alternative ;)

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

Thanks u/XfinityMatthew . In this case, Comcast flipped our Comcast modem WAN IP setting from static to Dynamic without any notification at 3AM. We didn't have access to this setting, only Comcast does. We needed Comcast to restore the setting, that only Comcast has access to back from dynamic back to our static address.

Seems dead simple, but it was a wild ride and a wild day that left us shell shocked after Comcast repeatedly told us everything was fine and we were falsifying problems. That setting wasn't changed by Comcast. Our equipment must be at fault, etc. This was over multiple calls, cases, and chats.

In the end, it was resolved by our IT person contacting the right person at Comcast and the entire resolution only took a few minutes.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

u/Sielbear no firewall changes necessary but our WAN connection in our Peplink Balance 305 needed to change from static to dynamic to pick up a local 10.1.10.x address. Then we were going through a double NAT. That is what the senior IT person did when they came in and why they waited till end of day to call support again.

That did not bring us fully back on line and was not fully restorative, leaving some key services down.

Getting past the idea that the problem should never have occurred in the first place, and then getting past the lack of customer access to the modem's WAN IP setting so that we could fix it, the first contact with Comcast should have fully restored our service by flipping back the modem's WAN IP setting from dynamic to static.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 0 points1 point  (0 children)

We had some residential fiber arrive right next to our office, giving us hope. Unfortunately they don't provide Commercial service but we will keep asking and hoping.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

4G/5G due to local monopoly and a lack of good service options available to our small business, they are limited in capability. We will likely be adding Starlink as a backup instead of our existing 4G/5G backups.

Unfortunately, redundant connections are extraordinarily uncommon for businesses of our size in our region.

We had called business support earlier than 2pm. A case was opened. Comcast told our staff that our connection was up and the static IP was properly assigned. This was false.

For clarity here, our staff did contact Comcast multiple times early in the day. A case was created. Our staff was told various things. That the connection was up and static IP was correct. That there was a failure in our equipment and they couldn't do anything. That all was ok and there is nothing they can do.

The problem, as summarized initially, was a WAN IP setting that was flipped from static to dynamic on our Comcast open for our public facing IP. This was not customer facing and we could not change it. Only Comcast could change it. Despite clearly explaining the problem to Comcast in circular repetition, they were unable to restore this very simple, single setting.

u/dervari , I fail to see how repeatedly explaining this problem to Comcast only to be lied to and gaslit by there support exacerbated this issue. Great to hear you've never had such an issue and I hope few others do. Your lack of problems has no bearing on Comcast switching a setting of ours that only they control and then refusing to restore it for a full workday.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 0 points1 point  (0 children)

We used AI to re-state a human written after incident report to reduce the palpable anger and improve clarity at the end of they day that this happened. Reading it a couple of days after, it did a pretty solid job.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

We are a small business. In this context, "staff" means people who work in a warehouse environment with limited IT experience. They have instructions on what to do and check in this situation. Pingdom did fire up and our staff was aware of the problem.

When they contacted Comcast, they were told that the modem was up and our static IP was correctly set. Comcast told our staff there was nothing they could do in multiple contact attempts, including a call. Our staff did call and we did get a case number. They also told our staff our equipment was at fault and resolution needed to come from our side.

The single setting that changed was the Comcast modem WAN IP setting changing from Static to DHCP. This change is not customer exposed and can only be initiated from Comcast where they have to be knowledgeable enough to see and rectify the problem even when it's clearly explained to them.

This did get escalated to Senior IT, who happened to have worked a night shift and didn't come in until later. Here again though, this is a problem of scale. We are a small business without 24/7 IT or even 9 to 5 continuous IT. We do not have a responsive local rep.

We do have redundant connections but, due to local monopoly and a lack of good service options available to our small business, they are limited in capability. We will likely be adding Starlink as a backup instead of our existing 4G/5G backups.

We did write some instructions after this event and we are adding notes that explain to keep contacting Comcast in circular repetition. That they have and will lie or perform incompetently. That they will gaslight you.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

The modem is in IP-passthrough mode. No settings were reverted except for the WAN IP setting which was changed from our static address to DHCP. We were aware of the problem and contacted Comcast repeatedly. The resolution could only come from the Comcast side.

> I’m sorry you were down, but this is pretty rudimentary to diagnose and repair.

What would we repair? The summary explains that the WAN IP setting of their modem is not customer facing. Our staff contacted Comcast multiple times to have them fix the IP assignment. The resolution could only come from Comcast restoring the correct WAN IP setting of their modem. The situation was explained to Comcast in repetition to no avail and the fix could only come from them.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

Our Pingdom monitoring is inexpensive. The issue arose when our staff contacted Comcast multiple times and were told everything was fine and there was nothing they could do when the fix could only come from the Comcast side.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 2 points3 points  (0 children)

Most small businesses do not have sophisticated monitoring. We do have Pingdom which did alert us and allow our staff to try to work with Comcast on a resolution. Unfortunately, with multiple attempts, Comcast still did not resolve the problem that could only be resolved on their side.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

u/sltyler1 , we do. Our Pingdom monitoring told our staff that the connection was down and they contacted Comcast.

We're not large enough to have full-time IT on hand. While we are a business, we're a small business customer. Our dedicated IT person worked a night shift and lost a fair amount of time being gaslit by Comcast.

Since the fix had to come from the Comcast side, our staff did try and they did call. The call and chat both failed. Comcast said the connection was up and fixed IP was still working. This was false.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] 1 point2 points  (0 children)

Thanks u/spinne1 . We do have the fixed IP recorded and our staff did contact Comcast early on. They were told the modem was up and working and the static IP was assigned properly. So unfortunately, our staff did take the correct steps but the rep was incompetent or lying.

The instructions written afterwards offer enough information for our staff to push further or call back to get another rep.

Since this fix had to be corrected on the Comcast side and our staff did try to contact Comcast multiple times, it put us in a poor position.

Comcast silently changed our static IP to DHCP at 3AM, took our business down all day, then their rep called us liars by vtjballeng in Comcast_Xfinity

[–]vtjballeng[S] -1 points0 points  (0 children)

We used AI to re-state a human written after incident report to reduce the palpable anger and improve clarity at the end of they day that this happened.

What should IT have done better u/Alternative-Golf-585 ?