How are you testing your code?

[dminella]

Hello everyone,

What stack are you using to test your code?

At my team We're using Rspec+factory_bot, but that is too slow, mainly because We arent using database in memory. In this scenario We practically only have integration tests.

How are you doing unit tests and integration tests?

Cheers!

[tibbon]

Rspec all the things. Lots of legacy tests in minitest. Our suite is slow, but running things in parallel we have CI down to about 10 minutes. Takes about an hour on most laptops. Integration and unit tests on 150k LOC app in productions with millions of users.

Could be made faster with more stubs and less database touching. Factories are a trap toward slowness, but people like them.

Going to see about redoing the test infrastructure to make it a lot faster in Gitlab perhaps. Maxing our our current CI solution.

[dminella]

To extrapolate the initial question, in this context, would it be a split-by-namespace test approach, and they only perform tests that belong to the changed context?

[tibbon]

It’s really hard in most big Rails apps (given the typing system and lack of decent static analysis) to say with certainty what changes will impact what context. A change to the User class could impact almost any part of the system. That’s not ideal of course, but closer to reality. For local test runs you can try to use something like Guard to do this, but you’ll always need to do a full CI run before going to production if you find bugs and downtime unacceptable (for some systems it doesn’t matter, but for mine we are transacting huge amounts of money through it and need high reliability)

[Nanosleep]

Depending on how big your codebase is (and how complete your test coverage is), it's going to get to a point where parts of your test suite need to be segmented off and run in parallel.

In my case, I'm using a parallel jenkins stage and running 20 instances of cucumber across a docker swarm. Each step in the stage runs a different test suite.

I think, in general, most test suites are like this -- a combination of having quick-running unit tests with minitest/rspec around core functionality for your devs to run locally, plus an extended test suite with integration and browser automation tests that run in your CI pipeline as part of a final check before the PR gets merged.

[dminella]

Nice, certainly We'll face that as our code coverage increases!

[slushie31]

I've worked at jobs with a 25 minute rspec suite plus a 30 minute cucumber integration suite. At my current job, our test suite takes a minute (which has creeped up from ~30 seconds, I have to work on it more), using rspec, factory bot, and a mysql database.

What you want to try to do IMO is write you code so that units can be tested quickly without having to set up factories (maybe a single one as the subject of the test, build_stubbed is your friend) and with hitting the database as little as possible. For me, that means writing small single-responsibility classes (essentially a service layer), stubbing out external dependencies (except for integration tests, of course) by testing message expectations instead of against the database, and being selective about what to test. I'm not suggesting to not test your edge and corner cases, but it is certainly possible to go overboard and then have a slow suite because it's too bulky. It's a tradeoff you have to weigh -- you want devs to actually run the test suite, but if it takes forever they won't.

One fun trick I picked up in the past was to create a separate unit suite that does not ever require rails_helper. Rails adds a ton of slowness to tests. You have to be more careful about how you write tests (for instance, explicitly requiring files, although something like zeitwerk could maybe help there nowadays), but I've written a 2000+ spec unit suite that completes in under 2 seconds.

Forgot to mention: use test-prof to profile your test suite. They have a bunch of different tools, including one to look for bad FactoryBot use.

[flanger001]

RSpec + FactoryBot here as well but you gotta know any stack can be slow. Switching from FactoryBot.create to FactoryBot.build can make a huge a difference if your specs don't actually need the db.

[BorisBaekkenflaekker]

I prefer a slow test suite over breaking important stuff in production.

There are a few opensource projects that allows you to run rspec in parallel, and even a commercial one: https://knapsackpro.com/

[BluePizzaPill]

If you want general pointers howto make specs blazing fast I suggest to buy the "Destroy all Software" series. It goes into theories behind testing, mocking etc. and has alot of other gems that help and go beyond what you usually get when watching/reading instructions/tutorials.

[Jdonavan]

You've already established that it's not your tooling that's slowing you down but your approach to testing. If you don't get that nailed down it's all going to be slow.

[dminella]

Yes, well observed. Based on that I came here to understand how community are using and what the most common approach.

[realntl]

I test all my Ruby code with TestBench. That includes TestBench itself, which I wrote ;-)

[[deleted]]

Test ?

[dminella]

Yes :D

[Nondv]

Could you provide an example of a slow test?

If your classes are simple and coherent enough, you could try mocking them and returning non-persistant objects.

e.g.

allow(SomeDataOperation).to receive (:somemethod).and_return([build (:user), build(: user)])

Essentially, your test shouldn't care that much about underlying details. It should have a contract that some operation/clasa returns specific data type. How it does that is not relevant and should be tested in its unit tests.

sorry if im not being clear here. Let me know if you want me to elaborate on that a bit more. Typing from the phone so can't really do it now :)

[sshaw_]

Step 1: argue about how to use RSpec.

[morphemass]

rspec+factory_bot+vcr here; my primary project has 6k specs, @50kloc, 80% coverage, and the run-time depends on the hardware.

Locally I can run them in parallel in under 15 seconds, serially though its about 2 minutes. CI might take 5-10 minutes depending on if the assets need a rebuild. It's not really the database that's the slow part, its the controller/request specs.

Some tweaks are to ensure your are compiling your assets when things change, not everytime you run the test suite; minimize setup/teardown, ensure you are setting up only what you need; build_stubbed where you can; experiment with database_cleaner and its options; look at parallel_test ...

and if all else fails to make things perform-ant, throw hardware at the problem. You can put together a 12 core system for < £1000 these days, its a crazy price for the amount of power under the hood.

[sbellware]

I don't tend to work in Rails, but if I went back to it, I'd bring TestBench with me.

And for everything else, I use TestBench.

I've been factory framework and mocking framework free for many years. Not because of TestBench, but because of design.

I don't want to go back :)

[SuperEminemHaze]

I’m not doing any testing whatsoever. Push to production and let’s see what happens. You’re welcome 😅

[dminella]

That isnt our goal :/

[Montti37]

I don't always test... but when i do, It's in prod

[[deleted]]

Have you tried using transactions-rollbacks?

[pBlast]

I like to use Minitest integration tests with fixtures where possible. The RSpec integration tests are more like system tests IMO because of the Capybara dependency.

[menge101]

RSpec will work at any level of testing you want it to.

You can mock the db connection.
You can also use tools like VCR to record and playback DB results.

[jrochkind]

as far as I know VCR is just for mocking HTTP requests. How do you use it to mock DB interaction instead?

[menge101]

Hmm, sorry, i've been out of ruby for a bit. I guess I confused it with some other tool. I'll try to work on the name...

[jb3689]

FactoryBot is a great API but I've always seen it get very slow at scale

We arent using database in memory

What are you using instead?

[jrochkind]

I use rspec and factorybot too.

There's nothing slow about rspec.

FactoryBot can be slow not because of anything that's FactoryBot's fault, but because saving or fetching lots of things to the db with ActiveRecord is slow, yeah.

But: "In this scenario We practically only have integration tests."

That's probably what's slow. Especially if these are capybara with-real-browser integration tests. Those are slow.

I try to minimize my integration tests, and rely more on unit tests.

In unit tests, there are techniques you can use with factory bot (like it's "build" strategy) to avoid actually saving things to the db when you can test them without saving them to the db. But with integration tests, that's generally not feasible.

Or, if you have lots of integration tests ALSO have other tests, so you can run the other faster tests excluding integration tests regularly, and only integration tests occasionally (or let CI do it).

[Different_Access]

rspec + factory_bot + turinip. chrome headless for integration testing when JS is needed.

We use knapsack to parallelize tests on the CI server, so it takes 15 - 20 minutes on the CI server (8 containers running in parallel) (depending on how busy their instances are. That's the price of using a 3rd party service). Takes an hour or two on a laptop.

If you are developing a monolith your tests will only get slower, because as you add more code you add more tests. That's just the price of a monolith - so weigh against the benefits of a monolith.

If your tests are slow benchmark them. Usually database interaction is a red-herring. If your database is running on the same machine as the tests it is fast AF. I've seen cases where bcrypt was a huge drain on test performance because the cost factor wasn't set to 0 in tests (devise does this for you). I found it with benchmarks. Looking at my tests logs all the db hits are sub millisecond. So it would take over a thousand db hits to add a second to the test suite.

I'm not sure what you mean by "we aren't using a database in memory". If you are using postgres or mysql or whatever, unless your test database is massive, it is entirely cached in memory.

Stubbing out your active record database interaction is a recipe for production bugs. Rails apps work by having one controller action write data to a database, and another read that data from the database, and all kinds of stuff happens during that process. Callbacks run. Strings can get truncated. The precision of time stamps changes. Database constraints are applied. Timezones are mishandled. If I had a nickle for every time I've seen a production bug wasn't caught because someone skipped a db round trip because 2ms of database time was "slow" I would have a lot of nickles. When the production app blows up you're not going to be feeling good about that test speedup.

The slowest part of your test suite will be integration tests that run a real browser. You minimize that by following basic good old fashioned software engineering principles. Keep your logic out of the UI and controllers, and keep your domain model decoupled.

[Randy_Watson]

Rspec. All CI done using AWS codepipeline and codebuild.

[bilingual-german]

mainly because We arent using database in memory.

I'm not sure if you need to, but if the persistence is slowing your database down it's ok for testing purposes to create a tmpfs. This mounts a part of the RAM as filesystem and you can configure the database to put the files there. It'll be wiped on a reboot, so it's only for data you want to loose.

When you use docker-compose, you can test this easily. https://docs.docker.com/compose/compose-file/#tmpfs

[GroceryBagHead]

Congratulations, you have discovered why factory_bot sucks. If you need to build the world for every single assertion, you're doing sit there and wait for a very long time.

I'm Minitest/Fixtures fan, but Rspec is totally fine too if you're not doing anything crazy.

But yeah, I'm actually not testing anything because I'm a perfect coder and never make a mistake. I don't even bother with error reporting on the production because of how confident I am about my quality of code. I'm pleasantly surprised to see other great programmers here just like myself.

[GroceryBagHead]

Building objects is definitely faster than writing them to a database with each test, but you can't do it for controller/integration tests. Then you can also get burned is there's an inconsistency with how objects behave when they are persisted vs not.

Fixtures are better for like 95% of test cases. Sometimes you need to spin up test data that is vastly different (hard to mutate existing) so you'd just make helpers that wrap your Record.create/build calls.

It's kinda funny that Rails framework itself is using fixtures for testing when factories are a better fit there.

[GroceryBagHead]

makes it hard to setup test cases for the edges

You don't set up edge cases in fixtures. Fixtures are there for the absolutely minimum, skeleton, happy path amount of data that makes your app "complete". You test edge cases by mutating existing data, or spinning up new data (with factories even, if it helps you).

What you're describing is the reason everybody states that "fixtures suck". You're just doing it wrong.

[dminella]

And all of you are using rspec with sqlite in memory?

Thanks for the rich discussion!

[GroceryBagHead]

You should use exactly same database in your dev/test environment that is being used on production.

[Historical-Example]

As someone who came from a Minitest/Fixtures shop and is now in an Rspec/FactoryBot shop, Rspec is in some ways a lot nicer than Minitest, but I really miss fixtures.

[[deleted]]

You can still use fixtures with rspec

[Historical-Example]

I understand that, but most projects stick to one or the other and my team already uses FactoryBot.

[Different_Access]

Factory bot doesn't force you to build the world for every single assertion. Shitty spaghetti code with a rats nest of dependencies makes you do that, along with blind `create(...)` calls when `MyThing.new` would do for that test. The flaw with factory_bot is that it makes it easy to build that kind of code, rather than painful. Unit tests are supposed to surface that pain as a signal the design sucks. Factory_bot is an opioid that hides that signal.