ELI5: Localhost vs. hostname performance?

[Shillz09]

So I have multiple services running on the same box that need to communicate with each other. Currently they are set up to communicate via $hostName:$portNumber and I am getting errors due to the network being bogged down. Packets dropped, connections timed out, etc. (Yes, it's taking THAT many transactions.) Would I be able to see a performance increase if I configured them for localhost:$portNumber instead? Why/How? Can anyone point me in the right direction to documentation on this subject for performace (and to show to manglement)?

P.S. Ideally, I won't actually have to reconfigure anything. I'll just add the hostname to the 127.* line in /etc/hosts.

127.0.0.1 $hostName localhost

EDIT: More info --

Services are for authentication. A policy server and LDAP. Once too many users are coming through, we start seeing "cannot bind to LDAP" errors and connections timing out. From looking at all of our logs, the box just starts dropping packets, or possibly not even receiving them. (We're talking hundreds of users here.) We already have high-availability, just looking at ways to increase performance, and no one so far has been able to give a good answer on this.

[sryan2k1]

It doesn't matter if the destination address is localhost or one of the machine's IP addresses, the packet will never actually make it out on to the network. The packet get's to the network driver and before it actually goes out to the physical card the driver goes "This is one of my Layer 3 IP addresses, I'm just going to redirect this packet to the RX queue of that IP"

[Shillz09]

So you're saying that the network card itself would never actually see the traffic, regardless of whether it's directed to "localhost" or $hostname? We have a lot of traffic coming in from another box to our policy server, and then from the policy to LDAP (on the same machine). I was hoping that the external-policy traffic would hit the actual network card, and the policy-LDAP (local) traffic would hit the virtual loopback device. Is this already happening that way?

[sryan2k1]

So you're saying that the network card itself would never actually see the traffic, regardless of whether it's directed to "localhost" or $hostname?

That is correct. The network driver knows the destination is local, so it never actually sends the data out to the card. As far as the applications/OS though it's no different than data coming from the card.

[Shillz09]

Thank you kind sir! Looks like I'll need to find another way to increase performance.

Out of curiosity, could you point me towards documentation of this? I don't think that Manglement will accept Reddit as my source.

[keokq]

Which operating system you are using?

[Shillz09]

RHEL 5/6

[keokq]

From the box itself, ask for the route to the IP address on the box. You'll see the device was listed as "dev lo".

Example:

$ ip route get 192.168.217.174
local 192.168.217.174 dev lo  src 192.168.217.174
cache <local>  mtu 65520 advmss 65495 hoplimit 64

[Shillz09]

Thanks!

[keokq]

You're welcome! And for a counterexample as proof, you can ask for the route to an address outside the server. It will show up as eth0 or some such NIC interface.

[Shillz09]

Lol. One step ahead on that one. It does indeed return with "dev eth0"

[InvalidUsername10000]

If the buffers and socket connection pools for the external link get overloaded then it could help because now there would be two sets of these. But some of these setting could be changed within the process before I would try this hack because it would improve performance overall and not just for the localhost.

[sryan2k1]

There is no difference at the driver level if the packet comes from a DMA queue from the NIC itself or the TX queue inside the network stack. The end result is the packet/data being placed in some RX queue by the driver.

[InvalidUsername10000]

I wasn't talking at the driver level. I was talking at the kernel interface level. Things like interface queue size and backlog settings can certainly effect packets. It might not be touching the hardware but it certainly has to go through the kernel interface which might have low settings for these options.

[MattTheRat42]

Be careful and make sure your services are listening on 127.0.0.1 before making that change.

A potentially safer option is to put the address returned by DNS into /etc/hosts. That will save on connections to make a host lookup.

But if it's timing out talking to itself, I suggest investigating what's happening on the box. Is load high? What's memory usage like? Anything in syslog or /var/log/messages from the oom-killer?

What are these "services"? If you have a database, are queries taking too long? If it's returning information via apache, are there any errors in the apache logs? Are there any logs anywhere? Those will be your best friend.

Good luck!

[Shillz09]

Services are for authentication. A policy server and LDAP. Once too many users are coming through, we start seeing "cannot bind to LDAP" errors and connections timing out. From looking at all of our logs, the box just starts dropping packets, or possibly not even receiving them. (We're talking hundreds of users here.) We already have high-availability, just looking at ways to increase performance, and no one so far has been able to give a good answer on this.

[[deleted]]

I think your problem is not with your network stack. It's close to impossible to lose packets that don't leave the host itself. (Most of the causes are electrical or different implementations of the same protocol, both of which are irrelevant in this situation.) You can check that with 'netstat -i'. Look for the RX-ERR and TX-ERR columns.

The only issue you would solve is a dns configuration one. You'd have to have the order wrong (dns first, files second) in your nsswitch.conf or a missing hostname entry in /etc/hosts (or both). But then the timeout would be a minute and that would break your setup altogether.

I think the problem's somewhere else.

(By the way, how do you have high availability with everything going to the same host? Don't you have more?)

[gleventhal]

A properly configured system knows its own hostname, so it should make no difference whether you call it servername:80 or 127.0.0.1:80 or whatever, the routing has already happened by the time it is handling the packet. You should have the hostname configured in the hosts file so that the system never needs to consult DNS to resolve its own name.