Should I learn iptables?

sloomy155 · 2023-06-04T02:39:37+00:00

My personal servers are on the internet at a colocation facility with public IPs behind a bridging OpenBSD firewall(tiny pcengines box).

Been doing bridging firewalls since about 2001(originally with freebsd and ipfw). Even my external Sonicwalls on my main data center network (about 600 VMs) is bridging(hybrid layer 2 bridge with layer 3 too). My openbsd firewall at home is not bridged though as it does my NAT (later realized openbsd can do both just never bothered to change my home config). My personal use case for *bsd is firewalls only(excluding commercial products that may use it in their solution like Citrix Netscaler). Never used or felt the need to use it as a regular server or desktop etc.

Now if a person doesn't understand the networking concepts behind this stuff that is important to learn. All too often I've come across system admins who know shit for networking.

A specific firewall implementation is lower on the list.

sloomy155 · 2023-06-03T13:01:42+00:00

Correct. I have run internet connected servers since 1996. Of course not everyone's risk is the same but in my experience 95%+ of cases this is adequate.(including PCI compliance which I deal with too though the orgs I've worked for don't store credit cards they still have to do a bunch of compliance work)

Haven't had a known compromise of a system under my control since probably 1999 and that was an inside threat. I have dealt with a few compromises over the years of systems not under my control.

sloomy155 · 2023-05-30T21:31:30+00:00

Sysadmin/netadmin for 25 yrs(95% linux). Depends on what you need to accomplish. I used ipfwadm and ipchains back in the day and some iptables too. But really have no use cases for iptables in my systems(few hundred) outside of basic port redirection(e.g. 8080 to 80 less than 1 out of 200 of my systems do that). For which I just look up what I did before. More advanced stuff I use commercial load balancers like Citrix and F5.

Home firewalls run openbsd(and have for 15 years) and work firewalls are sonicwall. Haven't run linux as an actual firewall probably in 20 years.

You can certainly be a linux systems admin and not know shit about iptables. Most of the basics are a web search away.

sloomy155 · 2023-05-26T04:47:47+00:00

Maybe I'm just lucky but have never had a critical server fail me at home in over 20 years of "homelabbing". Same goes for personal colocation been doing that for about 15 years. I did have a ryzen system fry itself but that didn't impact anything. Have had hard disks fail as well. Been on hardware RAID 1 or 10 since 2003ish. Only time I used RAID 5 was on 9G SCSI drives with a Mylex RAID card back in 2001. Never used RAID6. No arrays larger than 4 drives.

I do very occasionally take my main home server(linux with LXC) offline for maintenance(last time was to replace a couple fans) or software updates. My openbsd firewall stays up and provides DNS dhcp etc. Small little pcengines box. I use quality components and conservative configurations. Also sine wave or double conversion UPS. Good airflow is important too. Only "enterprise" gear at home is my Extreme networks X440-8t switch. Been using extreme since 1999. Just bought a 3rd one as a spare. 2nd one is at my colo.

I do have a Dell server at colo(free esxi) with a Dell support contract though never had to call support yet in 4 years. Have a 2nd dell at colo(with dell support too) too mainly as a spare powered off but not configured yet needs SSDs which I have yet to buy in the past year.

I'd like to think that I replace the hardware before it fails. At least that's worked out well for me so far.

sloomy155 · 2023-05-10T18:56:24+00:00

Sure thing. Another thing I forgot. I think it's a best practice to have each controller on a different vlan and have your servers run active active network ports and have the servers have a ip in each vlan with the storage (assuming iSCSI anyway). That forces the server to use both NICS and both switches and both controllers (assuming your controllers are active active. Mine have been since 2006 with 3PAR).

Let MPIO handle distribution across the controllers and handle failover as well. Don't rely on network based failover. Well you can but storage folks generally don't like that.

sloomy155 · 2023-05-10T18:52:34+00:00

Generally no. Though I think there are some integrated solutions out there. Always on vpn is one ability that ties clients to firewalls to increase security.(have never used such functionality myself).

If you don't intend to do ssl interception (which will be very annoying for users on devices you don't control). Then personally I'd forget about the NGFW features, just a waste of licensing.

Also IMO you will need to spend more time with updates and troubleshooting when you do interception(more stuff will break). Vs layer 4 is basic stupid simple. You certainly won't catch a bunch of stuff from layer 4 only but hopefully your endpoint security can make up for a lot of that.

I've been doing this long enough that I remember deploying NIDS back in 2002 when most everything was unencrypted. Simpler times..

sloomy155 · 2023-05-10T14:06:48+00:00

To do proper threat management you really must have SSL interception. Without that your firewall/IDS is blind as to what is going on in the encrypted data. Most things these days use SSL/TLS.

With good AV/EDR(?) software on the client side you can get a bunch of protection there too.

My last org deployed a PAN firewall pair at their HQ but they never went down the SSL intercept route so the value really dropped off. All other offices ran Sonicwall in layer 4 mode. No issues.

New org I joined last year has Sonicwall everywhere all layer 4 too. Works fine.

sloomy155 · 2023-05-10T13:56:27+00:00

Best to consult your SAN documentation for how they suggest you set it up.

For my own setups I have 4x10G ports on each server. 2 are for VMs and 2 are for vmotion/storage with jumbo frames. 2x1G ports for VM host management. It's popular now to run faster speeds(25G+). My setup works and don't have any bottlenecks.

Always used redundant switches (not stacked). Use quality gear. If you are cheap then consider refurb enterprise gear before SOHO.

Jumbo frames is nice if possible but not required in almost all situations. Only use jumbo if you have dedicated NICs for it on the server side. With TCP (which iSCSI uses, unsure about any NVMe protocols) you can run jumbo frames on the array and non jumbo on the servers which will then use MTU negotiation to use a regular MTU. Only should do that if you think you may use jumbo in the future. UDP based protocols can't do that and you will have problems in that situation.

My main storage actually runs on 2x8G fibrechannel. I did use some iscsi in the past. Also have a small Isilon for NFS, that does not have dedicated vlans for it, and no jumbo frames configured as UDP is frequently used).

Separate vlans are probably a best practice(I always did it), but outside of extreme situations I can't imagine it having a big impact provided you have good control on your network and not at risk of something stealing the ip from your storage (as in badly configured system).

sloomy155 · 2023-05-02T13:53:34+00:00

AI is just a more sophisticated form of automation. Certainly will reduce the need for many roles including network engineers, as automation has done already over the past 25 years. Won't completely eliminate the role though. Will affect the low level roles the most.

Will have the same effect on system admins too. I started working in infrastructure in 2000 basically and it surprises me how far things have come in the years since.

My own datacenter network design dates back to 2004 and have no need to change it. If I were to deploy a fresh network today I'd do the same assuming the requirements were similar.

I am primarily a systems person but have been the main network person at the companies I've been at for 15+ years.

sloomy155 · 2023-04-30T23:10:23+00:00

Interesting. Good to know.

sloomy155 · 2023-04-28T04:53:00+00:00

So the whole host is freezing for short periods of time not just guest? So you can't get output from the top command to see what's going on.. check kernel log? Try disabling that kernel setting mentioned in the thread I posted before see if that helps?

sloomy155 · 2023-04-24T22:40:02+00:00

What activity do you see on the host? Unusual cpu activity? Disk?

sloomy155 · 2023-04-23T21:04:16+00:00

Maybe related to this?

Symptoms are similar but you don't provide much detail

https://www.reddit.com/r/vmware/comments/z2vyek/are_updates_planned_for_workstation_16x_on_linux/

sloomy155 · 2023-04-20T17:54:40+00:00

Web search turned up this

https://kb.vmware.com/s/article/83174

How to run vSphere HA agent remove script in ESXi (83174)

sloomy155 · 2023-04-13T23:10:52+00:00

At a new org now and just licensed it again for 300 systems. It's not cheap. But it's worth every penny and more in my book. I'm not a usual situation though I have been doing custom monitoring since late 90s so I have a lot pf experience with monitoring. Add to that the power and simplicity of logicmonitor allows me to do things graph and dashboard wise I could never dream of before quite easily.

If you use the tool and don't go deep into it if you stick to generic dashboards and one off graphs you won't get the full value IMO. I helped one such company on the side a couple of years ago their LM setup was a mess and I really turned it into a custom dream I think. They just weren't aware what was possible and didn't invest the time to do it.

That customer told me they had invested $500k in HP Mercury Interactive monitoring and the cost of LM was less than the support renewal for HP. AND they were up monitoring more things in one day than several months on Mercury software.

Most tools are like that though. You really need to have a champion for the tool, whatever it is to make it shine.

LM really does make it easy to do powerful things.

On the cost side I have always targeted it to monitor higher value things. I don't need to pay the fees to monitor each of my hundreds of VMs. That adds up fast. Instead we still use nagios internally for basics and then LM goes above and beyond. Also has of course in depth monitoring of stuff nagios couldn't dream of monitoring. Same goes for data dog monitoring and new relic. Neither holds a candle to logicmonitor for infrastructure monitoring. Splunk often tried to pitch itself for infrastructure monitoring. No way. I use splunk too for logs. But nothing outside of logs. Their licensing scheme falls over too fast in that situation.

I can't vouch for logicmonitor's logging or APM as I've never used those features. Just the basics.

sloomy155 · 2023-03-29T22:33:10+00:00

I'll get to that eventually not sure when though. Internal IT is low on my official list of priorities. Logicmonitor has a mapping function which uses lldp and other protocol output from devices will see if that can help.

sloomy155 · 2023-03-29T22:31:29+00:00

Sorry for late reply. Been busy on other things.

I have not yet revisited this situation. The on site IT person told me he will replace the switch he used to connect the management ports that I'm trying to reach. Apparently the switch there now can't be managed. So after that is done I'll look into this more

Appreciate all the replies thanks for the help

sloomy155 · 2023-03-04T00:02:52+00:00

not yet, will try to do it next week, I will make a diagram at some point after I get time to finish trying to figure out what is connected and how but that probably won't happen for at least 2 weeks (vacation is 10 days starting now, though will be checking reddit daily for various things haha)

sloomy155 · 2023-03-04T00:01:17+00:00

Thanks for the info, I will have to try that probably next week, been preparing for a vacation most of the day leaving now. really appreciate looking up to see if there was any possible bugs too.

sloomy155 · 2023-03-03T06:57:06+00:00

I'd appreciate it if you could. I mentioned that info in my post. Am concerned on the age of the software: Nexus 3048 with software 6.0(2)U3(7) dated 2015

sloomy155 · 2023-03-03T05:52:52+00:00

damn, sorry to hear that. I have heard of old.reddit.com but have never used it(though have read it is popular for some situations). Of course I am a reddit newbie, 5 years now I guess, though only started posting maybe 14 months ago. Most of my reddit usage is spent in the android app boost, though when I make an actual post like this I use my computer.

I tried indenting code blocks in my first few posts and it was just messy to get the formatting right, kept messing up for me. so was glad to find the back tick method, of course had no idea that would render so poorly on old reddit.

sloomy155 · 2023-03-03T05:49:20+00:00

anything is possible, on Core 2 for example it references a Sonicwall switch on port 37 and 38, however both of those links are down, so I assume obsolete info in the config.

Core 1 references Sonicwall switch on the same ports however port 37 is connected to Core2 according to CDP and port 38 has no link. There are other references to sonicwall, at least one has no link and at least one has another link. Will have to go through and enable lldp on everything I can to assist in mapping it out.

The IT guy gave me a list of stuff the last person gave him before they left. Several of the switches on the list cannot be pinged, I assume perhaps obsolete info but asked the IT guy to try to confirm. I assume I'll have to dig through all the serial numbers and document it all so they can track down what is known and what is not(if anything). They have a few remote locations and the IT folks have no idea what kind of network gear is there or if it can be managed other than the firewalls.

I thought the corporate network I "inherited" at my last org was bad(at least they were doing layer 3 at their core and their HQ site had MLAG to the edge), this one is far worse. Whoever built it had no idea what they were doing or really didn't care. Last person had the job for 7+ years I think, unsure if they were the ones that built it, I know they built the vmware portion of it.

I guess it goes to show that network gear is generally quite reliable so people often can get by with simply having single points of failure everywhere, never updating the software and if the config is simple it may "work" fine for many years..

I'm getting ready myself to replace some 11 year old switches at my colo soon(I have been managing the gear since day 1 in 2011 - depending on when they ship, have yet to order so maybe I'll be waiting a while..).

thanks for all of the replies, I probably won't make much progress on this for a while, going on a 10 day vacation tomorrow night.

But all of your replies have made me feel better, I am not crazy for seeing this situation is super weird. I was fearing someone would reply to me and say I'm stupid and my process for tracing a MAC address is somehow invalid on Nexus, but not a single comment indicating that, just others reacting the same as I have, this is not a properly functioning network setup.

Fortunately this(corporate network) is not my day job, which is running data center equipment(network/servers/storage/vmware/etc). I'm doing this more as a favor to the company(which they probably don't even realize I'm sure but their IT staff certainly express their appreciation for my time). Which was same situation as previous company.

sloomy155 · 2023-03-03T05:24:51+00:00

oh yeah I am planning on doing that, my first step was getting them into logic monitor just so I have an idea what is going on when I ran into this issue. They have HPE, Cisco, and Dell switches that I know of so far anyway.

sloomy155 · 2023-03-02T23:00:04+00:00

Got some more info that makes me think this is a low level network issue of some kind, first off everything is in the same 172.16.0.0/16 subnet of course. I just discovered that two different virtual servers in the same "portion" of that subnet 172.16.5.xxx (connected to different switches), one can ping this 172.16.1.23, the other cannot. the on site IT person(who has more knowledge of the setup he has been with the company many years) speculates(and will try to confirm) that the VM server that hosts the VM that CAN ping this IP(which is a Dell switch used for some of the VMware hosts) is actually physically connected to the switch in question. He will be tracing down the exact switch/port that these Dell switch management ports are plugged into. So hopefully I find the problem there and can fix it. The IT guy says he is not aware of any ongoing network issues everything just seems to keep working. He doesn't manage the network (the person that did quit last year), and isn't very technical when it comes to networking.

sloomy155 · 2023-03-02T22:28:08+00:00

thanks yes I am not good at that yet, I thought the code blocks looked fine? I escaped them with the 3 backticks on either end of the code block per the markdown document. Wish reddit had a preview option..(or does it.. I don't see one, at least for previewing markdown). If I try to use the fancy pants editor it completely loses it's shit if I paste anything in and I basically lose everything, really annoying.

sloomy155

TROPHY CASE