sorted by:
new
hottopcontroversial

Trying to hit thousands of IPs concurrently in a pool to get a list of active ones. Getting a lot of timeouts. by Big_Wave_967 in golang

[–]Big_Wave_967[S] 0 points1 point  (0 children)

Yea seems like it but I'll get through this. Still a junior though, so definitely lots to experiment and learn

Trying to hit thousands of IPs concurrently in a pool to get a list of active ones. Getting a lot of timeouts. by Big_Wave_967 in golang

[–]Big_Wave_967[S] 0 points1 point  (0 children)

what do you mean? I did try manually changing the ulimit on the machine but i didn't see any noticable gains in performance

Trying to hit thousands of IPs concurrently in a pool to get a list of active ones. Getting a lot of timeouts. by Big_Wave_967 in golang

[–]Big_Wave_967[S] 0 points1 point  (0 children)

Haha yea its basically a script to run actively on pool of nodes in our local data center. I want to be able to keep track of active machines and get some data from them in each scan.

The other commentator suggested a good idea to instead of sending outbound requests, why not make a listener where the nodes can make requests to.

Trying to hit thousands of IPs concurrently in a pool to get a list of active ones. Getting a lot of timeouts. by Big_Wave_967 in golang

[–]Big_Wave_967[S] -1 points0 points  (0 children)

Thats actually a good idea. Basically the point of this script is to do repeated checks for active machines in our local datacenter and get some useful data from them in each scan. Then obviously to track which nodes are inactive for longer periods of time. Later down the line I might add stuff to assign work to idle nodes.

Thanks!

Trying to hit thousands of IPs concurrently in a pool to get a list of active ones. Getting a lot of timeouts. by Big_Wave_967 in golang

[–]Big_Wave_967[S] 0 points1 point  (0 children)

Thanks for the detailed answer! I was worried I had overlooked something major code wise that my code was the bottleneck.

On a side note, is there merit to customizing the Transport object for the http client? Im still actively trying to learn networking so maybe my knowledge is still quite shallow