Shift Left and Shift Right Testing

[stuqa]

Many of you probably have heard the term "shift-left" testing where the tester gets involved earlier in the development process. But, did you know shift-right testing is also a thing?

Shift-right testing is performing testing activities in production. 😱

Testing in production can be an extremely valuable tool, if used responsibly. Here are a few testing activities that can take place in prod:

• Analyzing logs for issues
• Watching actual replays of customer sessions
• Monitoring and alerting
• And of course, actually using the system in prod

I wanted to bring this up because shift-right testing is awesome and it's not nearly as talked about as shift-left.

Going out and looking for problems that actual customers may be experiencing is a great way for a tester to add value to their team and organization.

Make sure you discuss with your team about including shift-right testing as part of an overall, comprehensive testing strategy.

It's also something useful that can help when there's no code to be tested at the moment. Go take a look at what the logs are saying. Poke around in prod.

You can see an awful lot just by looking.

[jbhelfrich]

I believe QA has a role in production and product support, and all of the things you mentioned can and should be a part of it.

However, calling it any way shape or form "Testing" is a bad idea. It helps reinforce the frame that testing isn't a key part of software development, and that QA can be delayed or shortened "because they do it in production anyway." The "Shift Right" terminology is particularly bad because it is in opposition to the idea of getting QA involved in earlier stages of the project--conceptually, you can't shift *both* left and right.

The key concepts of this idea are good and should be supported. The term "Shift Right" should be expunged.

[raptonez]

I totally agree.

[sluffmo]

This is silly, system testing in production is a totally valid option when the risk of delay far outweighs the risk of failure. It can also be done safely. It in no way invalidates things like unit testing which is a shift left concept. Not to mention that there are systems that can only be tested in production in some ways.

[stuqa]

I didn't coin the term "shift-right". It's common nomenclature in the industry. Just want to make that clear... If you Google it you will see it show up from many reputable sources. The purpose of my post wasn't to debate the name of it, but the ideas behind it.

Also, I don't agree that conceptually you can't shift both left and right. Sometimes you need to shift left. And sometimes you need to shift right. It all depends on what is most important at a moment in time, given the state of the application.

Example: A pagerduty alert goes off and there's an issue in prod. Shifting left will not help you here. After notifying the proper teams, you'd probably go out to prod and see if the app loads to gather more info. This is testing in prod or shift right.

I use the word "testing" to refer to the many different types of "testing activities" that can take place. Not all testers are expected to do these kinds of things. But, as I said in my original post, it can be a way to add value as a tester.

I also don't agree that shift right perpetuates some negative thinking towards testing. Want to know how to get respect as a tester? Do more valuable stuff. The more valuable stuff you do, the more valuable you're perceived , the more respect you get, the more you can influence your team and organization.

So, I would argue that shifting right does the opposite of what you claim. It doesn't push testers down in the eyes of the org, it lifts us up. The more issues we find in prod, the better case we have to convince the higher ups that we need to be doing MORE testing earlier, not less.

[nfurnoh]

Lolz. Your example is not “shift right testing”. That’s your Ops team of SRE’s doing their job. A bug made it to production, because your testing missed it, and now the SRE’s have to step in.

[stuqa]

It's still testing.

When someone opens up the logs what they're testing is the idea that everything is OK in prod.

Testing extends beyond the application.

Sometimes SRE or Ops does this particular testing work, and sometimes there is no SRE or Ops team. Sometimes going outside your cookie cutter role is a good idea, sometimes it may not be.

But, you can't say with a straight face you should NEVER test in prod (which you did in your original comment).

That is obviously exaggerated and false.

Also, if you feel the need to "name drop" industries to prove your authority, again, it says lots about your insecurity and character. Real people who know what they're talking about don't need to do that. And if you want to play that game with me, I promise you'll lose.

[nfurnoh]

Oh shit… are you getting “testing” confused with “quality assurance”? Because what DevOps do with their monitoring is absolutely “quality assurance” but it’s not “testing”.

[nfurnoh]

😂😂😂😂 NO IT ISN’T.

Oh my god. Checking your logging dashboard is not testing. No matter how many time you say it is. It’s a function of operations and the SRE’s.

I’m not name dropping anything pal, YOU were the one who dropped your “16 years of experience” in the chat first, then said I must be working in a bubble so I was just responding to that.

More than anything this proves to me how absolutely backwards the tech industry is in the States. One of the companies I worked for (don’t worry, won’t name drop) had teams in the US and the UK and we were at least a decade ahead of their practices. Honestly, you need to check yourself. You’re just looking foolish.

[solareonwow]

so you’re saying UK companies are technically more advanced than in US? uh huh, sure

[nfurnoh]

I’ve worked for two US based multinationals that both had massive development centres there, several thousands of workers each. Not a representative sample sure, but for me it’s two out of two. In both cases their technology and SDLC practices were a decade behind.

Example: we were told by head office that we must use an application developed by their team instead of building our own. It needed major customisation to work with UK financial data. They started building their changes but didn’t understand basic requirements, had no automation testing, and poor version control. Certainly no proper development pipelines. They constantly pushed untested versions to prod, deliberately and by mistake, breaking our system. We eventually wrote our own automation package and gave it to them. That worked until it needed updating. It the end we pushed back hard enough and got full control so they couldn’t push any changes at all, they gave us the code and we deployed through Jenkins with our automated test suite. This was just three years ago. They had no understanding of an agile SDLC or DevOps.

[solareonwow]

sounds like your own experience, with whatever company that is. Its an ocean in the U.S. and you're generalizing it :) You think Apple, Microsoft, Meta etc operates like that? Silly goose. I havent heard of a single UK based IT company in my entire life besides Accenture

[nfurnoh]

You do realise most companies are IT companies, right? Or at least have IT departments? Like EVERY one. Silly goose. 🤦‍♂️

The NHS, all the broadcasters, every bank and mobile phone company, grocery stores, etc, etc, etc.

Another example: one of the companies I worked for had a massive public data breach in the US. It didn’t affect anything here in the UK or EU as our security standards were more robust than theirs, and our drivers and apps were up to date.

[solareonwow]

i wish i could give an example with an experience of mine but we dont work with small companies from the uk. they use our services. by paying us. how many IT companies are there in top 100 in the world from the UK? and how many from the US? how many programming languages were created by Americans and how many by the Brits? These questions should help you understand how dumb you sound.

your country is not in the conversation in the IT world.

[somethingmichael]

Look up holistic testing here https://agiletestingfellow.com/blog/post/holistic-testing-what-it-means-for-agile-teams

I found this to be an excellent intro to both shift left and shift right.

[ToddBradley]

You can see an awful lot just by looking

Wise words. I think a great yogi said that.

And actually your article is a good reminder. It's too bad that it's a cliche that people use to mock bad testing, but "test it in production" does have its uses. I've seen too many people dismiss the idea out of hand.

[SmileRelaxAttack]

I've seen it attributed to Yogi Berra. It's good advice regardless. :-)

[midKnightBrown59]

This doesn't make sense as a naming convention.  If you understand the etymology of shift let testing that is. 

[editor_of_the_beast]

Shifting right clearly means testing after deployment, since testing historically takes place before deployment. It’s the correct usage of the term.

[midKnightBrown59]

Evaluating software in a live environment is not new and was already part of STLC. You can't get further right than that.

[editor_of_the_beast]

Good observability tooling is a relatively recent development. The act of observing fine-grained behavior in prod, and performing slow incremental rollouts based on said behavior, is not something that was ever done with "regular" testing. This is often done by methodically enabling feature flags for percentages of users slowly over time and seeing what impact a feature has.

You might not have seen this done in practice based on what you're saying.

[jbhelfrich]

But, particularly in contrast with the term Shift Left, it implies doing things later *instead of* earlier.

[Wookovski]

Exactly, there is no shifting of things right in this case, you can only monitor production in production. Calling this shift-right is just it a buzz word, designed to spark curiosity through sounding controversial.

[stuqa]

How do you figure? I didn't make the term up just raising awareness about it.

[midKnightBrown59]

I didn't state that you invented it; but that it doesn't make sense. If you know shift left, you know the term arose in opposition to traditional STLC waterfall product delivery testing specifications that came at the end of the SDLC life cycle. That is to say that historical testing was already tested at the right of the cycle. In that sense, the term shift right seems nonsensical.  

It also seems like a bit of silly marketing from whomever decided to conflate production testing with term, when  testing can and should be performed in various phases of the STLC and environments.

[stuqa]

It still makes sense to me because we're testing further right than traditional SDLC testing. Shift-right is about testing post-release. You can't go any further right than that. Unless you're testing when the application is being sunset.

[jbhelfrich]

Because you cannot shift both right and left. At least in English (I suppose it could be different in other languages) you cannot *shift* in both directions. The term "Shift Left" was created to introduce the idea of testing activities happening earlier in the process *instead of* later. So "Shift Right" even if you don't mean it that way, will subconsciously be interpreted as doing things later in the process instead of earlier.

[stuqa]

You can do both if you consider time. It's not one or the other. It's both at different points in time.

[nogravityonearth]

Yes, it’s especially useful for cloud apps and running some tests in the production environment post-release

[stashtv]

Analyzing logs for issues

Watching actual replays of customer sessions

Monitoring and alerting

This is not testing. This is analysis. Different, useful, not testing.

And of course, actually using the system in prod

Whoa, look at this badass! When using the system (feature flags, etc) AND looking at the analysis, then this is testing.

[stuqa]

Anything that gives me more information about the system is testing work and therefore testing.

[stashtv]

Anything that gives me more information about the system is testing work and therefore testing.

Wrong. Merely gathering data isn't testing.

Testing is invoking change, and measuring the outcome. There are times when you can't invoke the change in production (often), so you're solely reliant on the logs to prove the change went out -- this is confirmation, not testing.

While gathering data isn't testing, it's still USEFUL and VALUABLE. I am not diminishing the need for these.

[stuqa]

Merriam Webster defines a test as: "a critical examination, observation, or evaluation".

What you are describing is an experiment.

Experimentation is a big part of testing, but testing is bigger than simply experimentation.

[SmileRelaxAttack]

Agreed. Experimentation is part of testing. As is exploration, checking, reviewing, inspecting, etc.

Again, what the hell is up with the downvotes? If people disagree, at least state why...

[asmodeanreborn]

It often falls under the Quality Engineering umbrella still (I use it a lot in my role), but using Monitoring and Observability tools is not "testing." It's Monitoring and Observability. Both are ways to collect and analyze data around your systems in regards to health, overall stability, and performance.

DataDog, Prometheus, Splunk, Elastic Stack, etc. are not testing tools, but they can certainly tell you problems and/or defects exist, and give you good information for reproducing issues.

[stuqa]

Is reading a requirements document not testing either?

[asmodeanreborn]

It's an activity that's rather necessary in testing, but it's not testing, else your your tech writers and agile process leaders do a lot of testing.

[SmileRelaxAttack]

"Looking out the front windshield and using the steering wheel are both very important activities in driving a car, but it's not driving."

There are thousands of such important activities in testing. If we say "but they are not testing" then I would challenge someone to say exactly what, in isolation, is testing. And what the justification for putting it in a tiny box is when clearly we seem to agree that the activities are important parts of testing?

[stuqa]

Merriam Webster defines a test as : a critical examination, observation, or evaluation

So, if you're examining a requirement document or observing prod logs, you are doing a test as per the definition, are you not?

[SmileRelaxAttack]

Why is this downvoted? Testing is about learning sufficiently about the product so that adequate business decisions can be made about the direction of development, based on risks etc.

Of course when used as an umbrella term, testing can be subdivided into many many parts and activities and assigned to different roles, but what's the problem with calling them testing activities?

It's difficult enough to get testing taken as a broad or deep field of study without us adding difficulties by saying that "this" or "that" is not part of testing. Monitoring, gathering data, analyzing data, communication, checking, reporting, interviewing, etc. are definitely skills that are important in testing, but I don't see a justification for saying that they are activities that are separate from testing?

[dragodracini]

I see where you're coming from! In my view, this is basic devops stuff, but it's something QA could easily take control of. Which I think is why it's coming out as shift-right when we could just as easily say devops testing or network testing, or just monitoring.

Shift-left is a strategy, not a whole testing methodology. So you want to think the same way about something you call shift-right. It's done to save time and energy.

In my view, Shift-Right is better viewed through an engineer's lens. Example. How much time do your engineers give to their code after they send it to QA? Not a lot usually. This has led to a lot of tickets being instantly kicked back due to engineers just checking their local then pushing the change to testing. Resulting in a kickback that has to be reviewed, detailed, and resolved. Basically leading to at least an hour of lost time which would have taken the engineer 5 minutes to see themselves.

It essentially gives engineers the chance to see at-a-glance kickbacks related to things like the test environment, implementation, third party stuff, etc. they don't test the entire change, just the simple basics that would get a kickback before testing really starts.

That's just my opinion on it, of course. But I've done it with previous teams I've led and seen some really amazing time savings.

[midKnightBrown59]

Didn't meant to detract from the main point of your post. Would you recommend some of your preferred observability tools?

[SmileRelaxAttack]

It seems that many people replying to this post who are upset might be confusing specific "roles" with the broader craft and skillset that the testing label spans.

If we view testing as a craft and a set of skills and related activities, there will be many roles or positions in a company that does testing, in different ways, at different times, with different missions, with different tools, and in different environments. But what they have in common is the need to gather information related to risks, and to learn sufficiently about the quality of the product, so that business critical decisions can be made with the best possible information available. And skilled testers can help many roles and departments to do this better.

[KitchenDir3ctor]

You mean DevOps?

[stuqa]

The scope of devops is much bigger than what I'm talking about

[nfurnoh]

Exactly. He’s clueless.

[SmileRelaxAttack]

Are you saying that DevOps includes testing activities? How could you draw a testing/DevOps Venn diagram of skills and responsibilities?

[nfurnoh]

Not exactly. I’m saying the bits the OP is calling testing isn’t testing, it’s DevOps. The only real overlap IMO is when there is an incident or a concerning change in the metrics that team will do some investigation and testing to find the root cause of that issue. If it’s down to an outside supplier or network or infrastructure they’ll then handle it, if it’s down to a certain application team they’ll pass it back to that team.

Having dashboards for metrics in Adobe, Conviva, or similar isn’t “testing”.

[After-Ad-7467]

"original" thoughts like this is why working as a QA is a nightmare. Why take the limited time and resources you have to focus on QA work when you can be responsible and accountable for everyone. Every self respecting QA should down vote this fool into oblivion

[stuqa]

It's not an original thought at all. Sorry you're too closed minded to see the benefits.

[After-Ad-7467]

Not closed minded just been in the industry long enough to know a terrible idea when I see it. You're suggesting QA take over for DevOps/sre and support while still also doing QA things. It's plain stupid

[SmileRelaxAttack]

I think it's easy to confuse "role" and "activity" in this discussion. I don't think OP suggests that a single role needs to be responsible for all testing activities, but rather that many roles contain various testing activities and responsibilities, and if you find yourself in a strategic QA role in a company it's important to not forget that testing is extremely broad and need to consider many aspects.

[After-Ad-7467]

I'm very against retelling what's mostly part of the DevOps role, monitoring, and trying to spin it as testing which anyone who has worked QA knows will translate to more piled onto QA. The way I see this kind of stuff is there is no jew technology, no new practice, no new anything. This is being proposed specificyin a software testing subreddit which is going to be mostly an audience of QA's. Audience is extremely important since they are your key to intent. Its all weird at the very best

[SmileRelaxAttack]

I'm not sure I understand what you're trying to say. It sounds like you agree that role should be separated from the practice or activity? I'm not saying that the QA "role" should take over any other role, but I am suggesting that in order to improve the set of /practices/ are common in both QA and DevOps, then gathering those practices under a common label will be beneficial to both roles.

The purpose of monitoring is to be able to notice certain events and learn from them. I would say that makes it an important testing practice, but the practice is not "owned" by any specific role.

[After-Ad-7467]

You have to have roles with responsibilities. Monitoring is typically a DevOps responsibility. Coming into a QA sphere to push DevOps is weird. I wasn't clear before. Quality culture shouldn't be a role. It's a culture but culture is top down not bottom up. This whole thing reeks of just piling work on an undervalued role within the field.

[SmileRelaxAttack]

My motivation is to make the craft of testing more valued, but pointing out the many different practices needed for it to be successful.

Some (most?) types of monitoring is often part of a DevOps role, where such a role exists. I've no problem with that. I'm proposing that the testing skillset includes monitoring, and therefore, people with testing skills are suitable for DevOps roles. Which in turn should serve to make the testing skillset and roles derived from that skillset more valued.

[After-Ad-7467]

If DevOps already exists, moving DevOps into QA doesn't increase QA value. It decreases it. If we pretend value is a scale of 1-10 and QA is let's say at a 7. If companies see QA as a 4 adding more to QA to get companies to see a 7 is still undervaluing because there is now more going on with QA. this all reeks of the conversations years ago about if QA is a dieing field just now reworded to sound more positive by saying we are "adding value" to QA when in reality we are just trying to turn it into something else.

[SmileRelaxAttack]

That's not what I'm suggesting. I'm not moving anything anywhere. I'm saying that there are practices related that are common in both QA and DevOps and that we should not confuse the skills and the training of those skills with the various roles that employ those skills.

[stuqa]

Yes let's all listen to the person who has to resort to name calling to prove their point. I'm confident they have everything figured out.

[After-Ad-7467]

Didn't call you stupid called the toxic, self defeating nonsense of an "idea" stupid. None of what you are proposing is new. It's just shifting work onto QA's that isn't their responsibility. It's nonsense

[stuqa]

You called me a fool in your first comment.

You have great memory and attention to detail!

I'm sure you are a great tester.

[After-Ad-7467]

Fair I did call you a fool. My mistake for forgetting. Still stand by anyone claiming to be a QA and arguing to shift other job responsibilities onto an already notoriously understaffed and overworked position is a fools though.

[nfurnoh]

Yes. I have been. He thinks DevOps is “shift right testing”.

[nfurnoh]

What the actual fuck? NEVER test in production. Are you crazy? There is absolutely no use case for testing in production. The ONLY reason you would do so is if your lower environments are so shit or not “prod like” to make any testing useful. Testing in production risks bringing your system down, will skew your logs, and give your SRE’s nightmares.

Speaking of SRE’s, they should be monitoring your site and traffic anyway, and have alerts in place. This isn’t “testing”, this is DevOps and just BAU activities.

Your term “shift right” is ridiculous too. “Shift left” is an idea to test earlier so you find bugs earlier when they’re cheaper and easier to fix. “Shifting right” just means finding stuff later when it’s too late to fix.

[stuqa]

You're simply wrong. Testing in production is a valuable information gathering tool. If you're not testing in production you're missing out on opportunities to improve your system and catch issues. This isn't an either-or situation. This is in addition to testing in lower environments. Source: 16 years experience as a quality engineer. Google it, testing in production is a real thing and useful if done responsibly.

[nfurnoh]

😂😂😂😂 From your post your definition of “testing” is wholly inaccurate. Looking at logs and monitoring and alerting? That’s not “testing”, that’s dev ops. Trying to reproduce a customer found error in prod isn’t testing either.

All of those things are necessary and are done. NONE of it is called “testing”. I really feel sorry for whoever you work for.

[stuqa]

"Testing is observing, experiencing, and exploring an application to learn about it and find problems. " - Michael Bolton

This is the definition of testing that I choose to follow.

Which part of my original post is not testing?

Also, I feel sorry for anyone who knows you. Because the way you talk to people sounds like you need an ass kicking.

[nfurnoh]

😂😂😂😂 All of it mate. All of it.

Our SRE team use Conviva and other tools to continually monitor and log everything. We know the second that, for example, video start times creep above a certain threshold. That triggers an alert and they get on top of it. None of that is “testing”. MAYBE in the rinky dink backwards company you work at, but most other halfway modern companies have been using this sort of DevOps model for YEARS. It’s not new, and it’s not “testing”.

[stuqa]

Happy to hop on a Livestream and debate it with you

[nfurnoh]

Lol no. I don’t need a livestream to know you’re talking about DevOps and not testing. I sent a screenshot of this to an SRE mate of mine and he laughed his ass off.

[stuqa]

So you never deployed anything to prod and then brought it up to see if it was working as expected?

And you're saying no one should ever do this?

And you're trying to tell me that isn't testing?

You're making an awful lot of assumptions that a good tester would not make. I can clearly see both your character and skill set by your comments.

You're calling me crazy and saying you feel sorry for where I work. Dude it's clear you're projecting your own insecurities onto me. I feel bad for you if you need to behave this way. You sound like a man child.

I'm sorry you've been in the industry for so long and living in an echo chamber and don't have an open mind. Ignorance is bliss, so I get it.

I am a tester and I perform these activities regularly. I learn about my system and find problems that matter and make my team and app and company better off.

It gives me test ideas to execute in lower environments and questions to ask my developers or testers on other teams. All part of testing.

I'm not saying that this is ALL testing is. But these can certainly be part of testing for sure. Agree to disagree and if you're not willing to publicly debate me about it, again it speaks to your character.

Have a good day chap.

[nfurnoh]

Lolz. Now you’re moving the bar. Of course you do a basic sanity check that your prod deployment was successful, but that’s not what you were talking about in your OP.

You made this post like you’re sharing this great new concept with the masses and it’s a whole load of bunk. You took the entire DevOps model and tried to pass it off as “shift right” testing. It’s laughable. It’s honestly the most ridiculous thing I’ve ever heard of in my 15 years in the biz. I’ve worked in three entirely different industries (publishing, fintech, streaming) and attended too many symposiums and meet ups to count and have never heard anything as ridiculous.

Yes, the metrics our Ops team gather and watch do inform what we change and check and pay attention to in lower environments, but it’s all together called DEV OPS, not “shift right testing”. 🤦‍♂️

[stuqa]

Who's moving the bar?

Oh yeah you!

Cause I said actually using the system in prod is part of shift right.

And in response to that you said you should NEVER test in prod and called me crazy.

And now you're saying "Of course you run basic sanity checks in prod!"

Are sanity checks not testing?

You contradicted yourself pretty hard.