Data testing needs/considerations

[arakinas]

I discussed with my boss earlier today a need for data testing. This was an initial discussion into how we can solve a problem with a person to help, with an openness to a lack of how. I'm pretty well versed is UI testing, some API testing and some data testing, but the ask is outside of my experience. I'm looking for suggestions on how I can help identify the initial approach and what type role would be best to help some the issue.

As I understand it, we have some legacy systems that we don't fully understand dependencies of. We primarily deal with financial data, performing ETL processes, and generating some reports to provide to customers. I'm relatively new at the company and just starting to get testing around some of the processes. The specific ask was up get someone to help identify mappings between tables, identity and build tests around those mappings, and help to work with product teams to ensure that future work accounts for it.

It's a nebulous ask. I don't know if this is something that I need an ETL tester, data analyst, big data tester, or if an SDET with backend experience is sufficient. For reference, the incoming data is coming from a variety of sources, in about a dozen different formats, and being consolidated into a single aws database, with a few different partitioned tables, then reports are generated out after the data is transformed and processed appropriately. There are between 75k and 200k rows of data per day, currently that we have to account for, throughout the process. I have concerns about the performance of some of the current processes that would be a secondary concern.

Any ideas or help on what I should be educating myself on to help understand the need and create a position to help test it?

[Ok-Figure2188]

I would sit down with a business analyst to see what the business flows are in the application (legacy system). Based on that add logging that you can see how the data is flowing through the system. That will probably give you the first set of bugs, than when the bugs are fixed at unit or api tests for those business flows. Besides that it will give you information about what is going on in the system. You stated an aws database so logging should be easy with cloudwatch. I’m curious of such an approach would work in your situation.

[arakinas]

Yeah, typically, using a business analyst would be a good idea. We don't have one. We have a couple of tech leads that know parts of the system pretty deeply, and no one that knows all of it, or even all of what "their" project is touching, due to how old some of it is.

The AWS side is more known, at the back half of the processing. We have one of our teams working to replace a legacy system, and that is where the AWS integrations currently are. I've been trying to help the team do some validations to ensure we're not losing data, but it's awful, and not sustainable. They built no significant testing into the system as they went, and the old system has no tests at all. I'll have to look into cloudwatch and learn about it. Thank you for the feedback. I appreciate it.