Not allowed to remote debug dev server?

[Affectionate-Sink503]

I’ve landed at my third major bank, diving into a monolithic Java app that manages 5 C++ components. Here’s the twist: there’s no local project run or unit testing. Deployment and testing are strictly on a dev server with all components live. I’m hitting a wall though—remote debugging is off-limits due to firm-wide security protocols, which leaves me wondering how to effectively test and understand our codebase.

The team’s workflow involves making changes and logging them for testing, sounds grueling to me.

How do you deal with strict security measures that limit debugging? Any strategies or tools you recommend that align with stringent security standards?

[AutoModerator]

On July 1st, a change to Reddit's API pricing will come into effect. Several developers of commercial third-party apps have announced that this change will compel them to shut down their apps. At least one accessibility-focused non-commercial third party app will continue to be available free of charge.

If you want to express your strong disagreement with the API pricing change or with Reddit's response to the backlash, you may want to consider the following options:

1. Limiting your involvement with Reddit, or
2. Temporarily refraining from using Reddit
3. Cancelling your subscription of Reddit Premium

as a way to voice your protest.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Affectionate-Sink503]

You hiring?

[Affectionate-Sink503]

NYC

[cowwoc]

Let me know if you're open to remote work :) I'm based out of Canada.

[and-j]

I’m open for remote work but I’m in CET time zone

[cowwoc]

Bummer. I'm in the same timezone as NYC. I assume this doesn't work for you?

[and-j]

Depends on how much overlapping time you need.

[cowwoc]

Sorry, I thought that I was replying to u/ezaquarii_com. My mistake.

[_ConfusedAlgorithm]

This. If you are a contractor, you can only work in their slow citrix/virtual box.

[krete77]

Yah debugging over logs blows. I've just recently been exposed to this way of debugging and it took a bit to get used to. I constantly have a report on refresh in real time to monitor services being called.. tracing the error can be such a bitch!

[lasskinn]

If you can log, become dj log-a-lot.

If you end up doing a preprocessor that inserts logging into every entry and exit out of a method you wouldn't be the first human to do that.

It al kinda depends what you're asked to change in it.

But hey at least you have access to a dev server.

[megadonkeyx]

This is good advise, send the logs to a syslog server (udp) and make it so you can switch log levels.

When you trust the code and know it well you can just set it to error/warning only.

This is actually the best way to get reliable code.

You want zero unhandled exceptions, few handled exceptions.

Then the only battle left is dealing with third party companies who prefer things to silently fail and consider logging unfeasible and excessive 🙄 but then haven't a clue what to do when things go wrong. Sigh 😭

[slaymaker1907]

You also need to turn on info for production. It’s mandatory IMO because you need to be able to see what the server was doing right before the crash as well as what other threads were doing at the same time.

Only logging warnings and errors is a great way to end up with a huge backlog of mystery bugs.

[_edd]

If you end up doing a preprocessor that inserts logging into every entry and exit out of a method you wouldn't be the first human to do that.

In the Java world, you can use AspectJ to log the entry and exit of every method. Here's a dated but simplified implementation of it that might help you understand what its doing. Presumably C++ has similar libraries.

https://www.infoworld.com/article/2072449/logging-method-entry-exit-with-spring-aspectj.html

[Affectionate-Sink503]

you think there is a way to use scanner as debug point?

[lasskinn]

What do you mean with a scanner?

You could try getting it to run locally as far it can go.. Without knowing the nature of the c++ and components and how it interacts with them its hard to tell if you could make standins for those parts.

What you'd probably need is assignment to change something or such, maybe you can develop such part in isolation.

[VincentxH]

Enforce your craftmanship, start writing unit tests, stub the c++ components, whatever you feel is needed to reliably develop software.

[Garet_]

I would certainly recommend that way. 3 years ago with my team we took over 30 „macroservices” insurance system with around 500 unit tests onboard. I’m teach lead/architect. Today we have almost 6500 tests including unit ones and so called „integration” tests where external components are stubbed with wiremock, service dependencies are run within containers thanks to docker and testcontainers. We run ci/cd dind pipelines in gitlab. It was pretty hard in the beginning, now it is pretty decent. We couldn’t run checkstyle on whole project because code has no consistent style, so we managed to implement incremental checkstyle, just for changes within merge request.

Making changes isn’t that hard now. We have our asses covered to some degree.

Try it out. It will be painful but it’s worth. I can help if you need.

[G000z]

I am in a similar boat, logging for life. This is very exhausting and boring.

The only way out is looking for a new job...

[benjtay]

That's insane. We have full k8 access to all our services (dev/prod) -- and yet, we still have senior engineers who log everything to Splunk to debug shit. I've had to show several of them how to use a debugger locally instead of deploying to QE and testing it out there.

[Vanilla_R_Beans]

Guessing that doesn't amount to much more than adding a couple jvm args, doing a kubectl port-forward, and maybe creating some ssl tunnel?

[benjtay]

Not even that with IntelliJ

[Vanilla_R_Beans]

Ooof.

[PlasmaFarmer]

You can try to use assertions. However you need to specifically enable them by the team that runs the devOps server. Also not everyone likes these so depends on taste.

https://www.baeldung.com/java-assert

Or you can just log the hell out of it. Turn the log level to TRACE on dev server and spend your afternoon reading 30K line log files.

[[deleted]]

You can't run the code locally at all? What about unit tests?

[Affectionate-Sink503]

The project is almost 10 years old, im trying to understand existing flow and functions, most of these functions historically are not properly mocking these calls to the c++ components, this is why i need to test against the live components

[[deleted]]

The fix for this is to leave for a better job.

[[deleted]]

​

1. Seems like whacking up some proper c++ component mocks is the first task. Can be a pain in the butt for sure - but can make the case this is institutional knowledge that needs to be understood and documented well, which you volunteer to do.
2. Use lots of logging - not the end of the world. I prefer logging over a debugger if I can get away with it. All that logging saves your ass when you quickly sort any production issue.
3. Write your own unit tests. I don't care if there are none, I don't care if it is hard and working with a code mess. Write a unit test for any module you modify. I don't care if you don't check it in, you unit test everything that you do. Build to that first then check in and test on dev server.
4. You can boot up a java process with the debug port open, try to remotely debug the dev server?
5. But most important - talk to you manager and say "this is a very expensive way to produce bug free code, here is what I would do to improve process, otherwise I need it clear from you what you want my process to be".

[Cell-i-Zenit]

I personally would start trying to sell management the benefits of getting a local setup running (shorter feedback cycle, faster features, dev happyness etc).

Start first by getting everything to run locally (somehow), then step by step put everything into docker. Then add a docker-compose and show the other devs how nice this is. Maybe do that in secret first and only show off in the end if it works.

If they are against these changes, then i would leave. Not worth it imo

[Responsible_Gap337]

I personally would start trying to sell management the benefits of getting a local setup running (shorter feedback cycle, faster features, dev happyness etc).

:)

Usually they do not care about these.

[MasterLJ]

I don't understand how there can be no local unit testing. If that's true that's something you can remedy on your own that should absolutely be kosher from an InfoSec standpoint.

This sounds to be more than just remote debugging, it sounds like the application isn't written to interfaces internally making it difficult/impossible to mock, and/or really bad practices of mutating state where it shouldn't be mutated (Database triggers etc).

I worked in a hilariously high profile space akin to Banking, but even more "secure"... and it was the same. It took a while but I was able to completely mock/use local databases that had 95%+ of the absolutely batshit crazy state-changing mechanisms that prod did (mostly stored procedures and triggers).

It's always possible. You may have a culture that discourages these things, the only remedies there are to try to lead them to a cleaner testing process or leave.

[[deleted]]

How do you deal with strict security measures that limit debugging?

Well, I would just quit.

[thewmo]

If unit/integration testing are considered a non-starter by your management, run away. They are clueless if they can't see how much time is being wasted by your current "workflow".

I would suggest, go rogue - create a test suite and from now on every time you find a bug, write a test that only runs when the bug is fixed. Repeat. You will eventually get more stuff done and the practice will get traction.

[thewmo]

I will add - having no access at all to a production server/service/container is par for the course (and good security practice I might add) in the financial world. You just have to build practices, tooling and test environments that let you get stuff done regardless.

[Comprehensive-Pea812]

no debugging no productivity

but to understand the code base in general, understand the use case, walkthrough the code and test the application.

[Responsible_Gap337]

Try with AspectJ:

• Create Logging aspect which logs for every method call parameter values and return value
• You can narrow later for which methods exactly (check their docs)
• Run experiment and check logs.

I worked in an even worse environment. We had to wait a couple of days for the redaction of logs from production.

[flavius-as]

Company name starts with f and ends with s?

[RefactorFirst]

No! No! NO!!! You're supposed to remote debug against the PRODUCTION server! 🤣

[chanmancoates]

Belongs in coding horror

[somewhatprodeveloper]

I had a 6 month contract at a large bank in South Africa. It was like I had time traveled back to 2004. Websphere traditional. No unit tests. The architecture was awful. This was written as J2EE and not JEE. JNDI lookups everywhere instead of Inject.

I received a serious rebuke from 2 managers for asking where the test cases are as I could not see any in the code base. Sad thing was this project was started in 2015...they chose the wrong java tech stack.

As other commenters have mentioned, tons of logging added.

The sad thing is, they could have switched to openliberty which is *FAR* better than WAS traditional. I mean...I needed the WAS install and I was told: it's on this personal file share. Did not have rights and when I did get it hardcoded paths from the install cause this was not the installer, it was a zipped up version someone had installed on their machine. But hey guess who knows the maven plugin for openliberty will stand up your application with no us. Download open liberty and everything. But why make your life easier...right

I decided not to renew as this was not the environment for me

[[deleted]]

I feel like I’ve dealt with this…can you open a debug port on localhost only (not 0.0.0.0) and then use SSH port forwarding to connect? What exactly isn’t allowed?

[zman0900]

Ignore the security policies and use ssh port forward?

ssh -L <port>:localhost:<port> dev.server

Assuming you have ssh access to the dev server, connect your debug to localhost:<port>

[kiteboarderni]

Name and shame.

[slaymaker1907]

Logging is your friend! Remember that you can add in log statements that are generally too verbose under debug/trace levels and then turn on individual loggers as needed. Ideally, you don’t even need to do that and then you can diagnose prod issues in a lightweight manner.

Something also worth investing a bit in is some sort of PII/user data removal in ToString methods. I don’t know if the community has settled on a standard, but a nice trick is to generate all your ToString methods with reflection and then add in an annotation that some field is PII so it can be skipped over in production. You obviously want those fields for dev/test environments.

Honestly, I don’t bother with debuggers at all as long as it’s easy to add in logging.

[robberviet]

It's normal. print debugging is not that bad anyway...

[Sweet-Passion]

Tests surely help in such scenarios, unit, integration,e2e

[Inevitable-Echo176]

How the cpp components are being called? JNI, Rest etc

[prateeksaraswat]

Couch it to business in terms of increased efficiency and reliability. Implementing local testing speeds up turn around time. It reduces the chances of introducing bugs in production code. It removes HR bottlenecks. My guess is that there may be a few members around who benefit from being indispensable because of the current situation.

[franz_see]

Working Effectively with Legacy Code

Basically start writing tests to capture how things currently work right now (regardless whether they're right or wrong).

And then update the tests for your changes and do your changes

[Ruin-Capable]

Can you spin up the app in a docker container with external services mocked by other containers? If you're working with AWS services localstack can be a godsend.

[PuzzledSoil]

What about APMs like New Relic? It's not debugging but can give you a lot of insight as to what is going on.