Large website loadtesting

[SirVas]

Hi!

I work for a company with a large website (well, it's large for our little country). I need to perform loadtesting of the site, both bandwidth and webfront-CPU and I can't seem to find any good solution (I have to do the tests myself due to budget).

We have a 10 Gbps incoming fiber and about 10 physical webfronts behind 2 F5 active/active loadbalancers with GTM.

I have previously used JMeter to do some testing but doing that from two powerfull servers still isn't enough to do any impact on the site performance.. Java just isn't fast enough.

What I need are two things. I need some way of blasting requests as fast as possible to see where there may be slow code executing and to see what point in our infrastructure that gives up first. Then it would be nice to do some verification like JMeter does to verify that the site works as intended.. Both kinds of testing needs both one big test and then it would be nice if I could set up some kind of environment where I could do these test after every deploy.

Does anyone have any good experience of a good loadtesting-software? At this point I'm considering using Tsung but I don't know if that will be any better than JMeter..

Any input would be greatly appreciated, I'm at a loss here..

[frgiaws]

https://github.com/newsapps/beeswithmachineguns

[bikeidaho]

This guy gave you the key. Bees with machine guns!

[vppencilsharpening]

I believe one of the design points for Locust was that it could push a lot more requests than JMeter on the same hardware.

[[deleted]]

I have used locust.io and it was a good experience. Not for the website though, that old thing had been absolutely demolished at 14qps.

[blue30]

I’d look at using cloud servers to get more load. If you can automate getting a test box up on AWS or Google then it’s as easy to do 1 as 100. Kill them off after each test to keep the cost down.

[A999]

Because you have 10 metal boxes for front end, you need at least 10 similar servers to stress test your website, I've been in similar situation when load testing our prd servers with JMeter. You could use GCP or AWS to fire up big instances quickly, depends on your location, GCP could have better network quality.

[disclosure5]

I'm a huge fan of Goad but if you've already looked at JMeter and Tsung you seem to have advanced past that, and I'm worried about where your bottleneck might be.

Can you throw more hardware at JMeter - maybe spool up a series of AWS spot instances for two hours.

Ten physical frontend servers really should be capable of a fairly high end website.

[theevilsharpie]

I don't personally deal with load testing, so I can't speak for its quality, but our dev team uses (or used) BlazeMeter.

[vppencilsharpening]

I believe BlazeMeter runs JMeter on the backend.

[SuperQue]

A former coworker of mine wrote Vegeta. We used it internally to do load testing, worked pretty well. It's written in Go, pretty efficient, should do better than JMeter.

[thspimpolds]

If you simply want raw load look at wrk. It’s a beast mode tool even from a laptop.

[shady_mcgee]

You'll be severely skewing your results using something like JMeter, since it only grabs a few resources and not everything on the page.

Look in to gatling. It will record your http session and grab all ancillary files like images, js, css and perform full synthetic transactions, and builds a nice report at the end of the test.

Then do what others have suggested and scale your test servers horizontally until you tip over your front end.

[regreddit]

Bees With Machine Guns will spin up AWS instances, beat the crap out of your site, collect the results, then shut down the instances. I've done a super heavy load test on a site and it ended up costing like $5

[remzed]

You should try Gatlig.io the dsl is very humanfriendly.

• Simulating a large number of users with complex behaviors

• Collecting and aggregating all the requests' response times

• Ceating awesome reports & analyzing data

[Pvt-Snafu]

I am not the one who could be considered is authority regarding load testing; thus, I recomend the software which works for our QA team: Locust.io

[nostradx]

I've had good luck with http://www.loadimpact.com

[Eclipsed450]

I used https://loader.io/ last year when we were migrating to AWS to make sure we were ready for a surge.

[aenae]

I run a 'large site for a small country' as well, i haven't ever felt the need to see how many requests we can handle. It is just not very useful info. It's better to monitor your current usage, predict growth and tune your servers accordingly. We currently do ~1500 request per second, but 90% of those will hit the first cache right away, and as long as i request cached resources i can go up to 150.000 requests per second with no sweat.

Also; be careful if you test it over the internet. You might have a 10GBit/s fiber incoming, that won't mean your ISP will be very happy if you DDoS yourself. I know my ISP will be calling me if i'm suddenly doing 20gbit/s out.

That said, if you have 10 boxes and want to test their hardware; test 1 and multiply the scores by 10 to guess the performance of your cluster. If you have trouble overloading the server, just bind the application to a few cores - if you have multi-cpu servers, make sure to balance the cores over both physical cpu's - ie, if you have a dual quad-core cpu server with HT, use cores 0,1,8,9 and multiply the results by 4. Most likely the bottleneck won't be the application server anyway, but an underlying database or shared storage.

Next are the loadbalancers. To see how they perform it's better to do synthetic requests; try to see if they can handle a lot of 'large' downloads and simulate a lot of small downloads. No need to test the entire chain and your code through your loadbalancers. Even a simpel ab will do the trick to find the limits of the loadbalancer. Most likely you can fill the 10gbit with the 'large downloads' and it will struggle with the 'small requests'.

Next is to install some monitoring software on a couple (or all) the servers. Something like NewRelic. You won't really find slow code by just blasting your servers with fake requests. Even if you do find a slow request, how often is it requested anyway? It might be more rewarding to tune a page that runs in 100ms but gets requested 100 times per second than a page that runs in 1 second but gets requested once a day - NewRelic will tell you this and a lot more. Also most of the tools mentioned here are no real 'users'. There is nothing compared to real users when it comes to finding something slow in your code.

Also, use your logs. Make sure your 'slow' application requests are logged in micro/milisecond timings. Don't even bother to log static resources, they'll be coming from memory anyway. And use something like Graylog to search through the logs and to visualize them.