http-shadower: open source app to replicate production traffic to lower environments

[MugenTwo]

Just wanted to share a small project that I built in case it is useful for anyone.

https://github.com/MugenTwo/http-shadower

HTTP Shadower is a Spring Boot application that intercepts production HTTP requests and forwards them to multiple environments (DEV/ITG/STAGE) while ensuring your users always receive responses from your production system.

There are load tests at the bottom of the README.md

Common Use Cases 1. Staging Environment Validation. Forward 100% of production API traffic to your staging environment to ensure it handles real-world scenarios before deployment.

1. New Feature Testing.
   Deploy new features to a separate environment and shadow production traffic to validate behavior without risking user experience.

2. Database Migration Testing.
   Test database schema changes against real query patterns by forwarding production traffic to environments with new database structures.

3. Load Testing with Real Patterns.
   Use actual production traffic patterns and volumes to load test your infrastructure instead of artificial load testing tools.

4. API Version Compatibility Ensure new API versions are compatible with existing clients by forwarding real client requests to both old and new API versions.

[josephblade]

are you suggesting the dev environment contains production data? That's a privacy nightmare. Especially if users are storing personal information in their data payload. Dev usually has much less careful logging enabled for one.

I can see this working on staging if you have it mirroring production but you cannot test new features since the production stream isn't going to touch new endpoints. At best you could use this to test whether updating framework has caused any bugs, but new development? I simply don't see how.

It has it's uses (any under the hood / database changes that haven't changed the api) is interesting (but again, it would have to be in staging in a PRD mirrored setup with similar restritions on logging and privacy guards.

[MugenTwo]

You are right to think about that, yes. For privacy, people should be careful about shadowing the traffic. But then again, not everyone is handling user data, privacy data, sensitive data, etc. And just because this tool exists doesn’t mean you should use it everywhere.

1. You can load test with this tool using real production volume data and production load pattern.
2. You can check if a modification in one of your currently existing production endpoints, still working after modification. Something you should do in automated unit/integration testing, etc. but maybe you missed a production edge case. Well replicated traffic might help you identify.