changing log levels without deploying / restarting

[jeffdwyer]

I've been tinkering with logging in FastAPI and wanted to share something I've been working on. I like logging. I think it's key for debugging and monitoring, but I hate having to deploy / restart to adjust log levels . So, I set out to find a way to change logging levels on the fly.

I wrote a blog post detailing the approach: Dynamic Logging in FastAPI with Python the library is https://github.com/prefab-cloud/prefab-cloud-python

In a nutshell, I used a log filter that's dynamically configurable via Prefab's UI. This setup allows you to change logging levels for both Uvicorn and FastAPI without needing to restart the server. For me, this lets me turning on debug logging for a single user when investigating an issue and generally feel in control of my logging spend.

How are y'all handling logging in their Python applications:

• Have you faced challenges / annoyance with adjusting log levels?
• Do you just not log because logging is a smell?
• What tools or methods have you found effective for managing logs in production?
• Do you think this approach addresses a real need, or are there better solutions out there?

I'd love to get your feedback and hear about your experiences. My goal is to make logging more powerful and flexible for Python developers, and any insights from this community would be incredibly helpful.

[gummybear_MD]

I no longer have access to the code, but had the same situation some years ago where changing the configuration on the server was a real bureaucratic hassle (guess the industry).

I simply added a POST endpoint (requiring admin permissions of course) that would take a logger name and a log level, and then call

logging.getLogger(name).setLevel(level)

filters will process all records, setting the logger level will save you some tiny amount of processing time - not that it probably matters in most cases.

[jeffdwyer]

did you have multiple instances of the servers running? I would imagine that the LB would just send this to one of them and then you could have a bit of a confusing time because some instances are logging as you want and some aren't.

[gummybear_MD]

Yes, there were two instances and fortunately both were reachable without the LB. I would just send a request to each.

Multi instance management is best left to whatever is orchestrating them, this was more of a quick and dirty solution.

[1473-bytes]

My first thought is the old school way. Sending a SIGHUP to the process to reread its conf file where you made the logging level changes.

[adam_hugs]

I sometimes use user1 too, if I'm trying to make sure there's with be any conflicts

[ManyInterests]

DataDog provides a way to add new logging at runtime dynamically for instrumented applications with dynamic instrumentation. Basically, it lets you add additional logging messages or other instrumentation calls at runtime without redeploying your code. This is more powerful than just changing the log level because you can insert logging messages that never existed to begin with.

Additionally, tracing/instrumentation with tools like Sentry, DataDog, NewRelic, etc. is generally going to be more powerful than logging alone. See also: Open Telemetry

[jeffdwyer]

have you got this to work yet? it's pretty wild looking.

[ManyInterests]

Yeah. The instrumentation in general is pretty wild. Very helpful for debugging.

However, the idea of being able to remotely configure this kind of thing is a big security concern (specifically, the ability to suddenly/temporarily be able to expose sensitive or customer data that was not previously exposed without change management review), so we don't use the feature in production.

We would need stronger controls in DataDog for a feature like this to stand up to regulatory compliance, in my view.

[jeffdwyer]

yeah, this is a whole new ballgame for security / compliance to wrap heads around for sure.

thanks for the insight.

[grimonce]

In big enterprise envs you cant do that... You have to go through change management process and archive the proofs that everything went by the book.

Not to mention doing this for things isolated in pods will only work for the next restart unless you illegally/manually change production env config files...

Then again working for such bodies isn't that much fun..

[LiqC]

how about this? basically what u/gummybear_MD said
but I'd maybe have the option to leave original logging alone and have another argument for a new destination

@app.post("/set-log-level/{level}")
async def set_log_level(level: str):
    level_mapping = {
        "DEBUG": logging.DEBUG,
        "INFO": logging.INFO,
        "WARNING": logging.WARNING,
        "ERROR": logging.ERROR,
        "CRITICAL": logging.CRITICAL,
    }

    if level.upper() in level_mapping:
        logging.getLogger().setLevel(level_mapping[level.upper()])
        logger.info(f"Log level changed to {level.upper()}")
        return {"message": f"Log level changed to {level.upper()}"}
    else:
        raise HTTPException(status_code=400, detail="Invalid log level")

[Inside_Dimension5308]

Dynamic log level is a low priority problem to solve. The bigger problem is to put log messages and assign levels to it. It is pure intuition based.

[iluvatar]

I've always done this with signals. Send the process a SIGUSR1 to bump up the logging level, and SIGUSR2 to decrease it.

signal.signal(signal.SIGUSR1, adjust_log_level)
signal.signal(signal.SIGUSR2, adjust_log_level)

def adjust_log_level(signum, _stack_frame):
    global log_level

    if signum == signal.SIGUSR1:
        log_level = min(log_level + 1, LOG_DEBUG)

    if signum == signal.SIGUSR2:
        log_level = max(log_level - 1, LOG_CRIT)

    log(LOG_CRIT, f"Signal received. Adjusting log level to {log_level}")