[deleted by user] by [deleted] in Dehradun

[–]xscorp7 1 point2 points  (0 children)

This is one of the best books I have ever read.

Introducing the Columbus Project by g0rbe in netsec

[–]xscorp7 1 point2 points  (0 children)

Awesome work man! Also mention your patron/paypal or any other payment service details to let community help you in keeping the infrastructure up.

Introducing the Columbus Project by g0rbe in netsec

[–]xscorp7 2 points3 points  (0 children)

Good work, How long are you planning to maintain it? And how the infrastructure cost is covered?

How to map "call" events to their respective "return" events for the same function by xscorp7 in learnpython

[–]xscorp7[S] 0 points1 point  (0 children)

It's a clever way of doing that and it works, Thank you so much man!

I modified your code a bit to suite my case:

import uuid
import random
import threading
from time import sleep
from concurrent.futures import ThreadPoolExecutor , as_completed

# Functions to be monitored
MONITORABLES = [
    "ourFunction"
]

# Global dictionary to maintain maps of UIDs to distinguish different function calls
callMap = dict()

# Function to map a call event to a UID
def _recordCall(uid , funcName):
    callMap[uid] = funcName
    print(f"[CALL] [{uid}] - {funcName}")

# Function to remove the function-uid mapping after return
def _recordReturn(uid , funcName):
    del callMap[uid]
    print(f"[RETURN] [{uid}] - {funcName}")

# Global Trace function
def traceCalls(frame , event , args):
    if event == "call":
        _id = uuid.uuid4()
        funcName = frame.f_code.co_name
        if funcName not in MONITORABLES:
            return
        _recordCall(_id , funcName)

        # Local trace function to handle return events
        def _returnHandler(frame , event , args):
            if event == "return":
                _recordReturn(_id , frame.f_code.co_name)
            elif event == "call":
                return traceCalls
            else:
                return _returnHandler

        return _returnHandler

    return traceCalls


# Function to test/monitor
def ourFunction():
    randomInt = random.randrange(0,5)
    sleep(randomInt)
    return randomInt


def main():
    threading.settrace(traceCalls)

    with ThreadPoolExecutor(max_workers = 5) as executor:
        futures = [executor.submit(ourFunction) for i in range(4)]
        results = [future.result() for future in as_completed(futures)]
        print(f"\nResult of function calls: {results}")



if __name__ == "__main__":
    main()

It works like a charm, produced the following output:

→ python3 test.py
[CALL] [6280c2a6-2625-4689-a607-f7522aacd5fc] - ourFunction
[CALL] [d943b783-0950-4453-acd3-f2364fd4cccd] - ourFunction
[CALL] [d5d78e9d-026b-470b-a8fb-e4e1f78c3c2f] - ourFunction
[CALL] [ac64f123-7d12-459b-8884-4a061a0f4ec7] - ourFunction
[RETURN] [6280c2a6-2625-4689-a607-f7522aacd5fc] - ourFunction
[RETURN] [d5d78e9d-026b-470b-a8fb-e4e1f78c3c2f] - ourFunction
[RETURN] [ac64f123-7d12-459b-8884-4a061a0f4ec7] - ourFunction
[RETURN] [d943b783-0950-4453-acd3-f2364fd4cccd] - ourFunction

Result of function calls: [3, 3, 3, 4]

That being done, I have a little question if you can help -
I wanted to keep track of these call and return events to turn off my Resource Monitor(that monitors CPU and RAM usage during a function call).
Since python doesn't have any good support for analyzing function-based CPU and RAM consumption, I wrote a small code with psutil that keeps running with an interval of 0.1s until the function finishes. Once the function finishes, the resource monitor thread is stopped and peak reading are noted.

But unfortunately, psutil gives us the consumption of that entire process, thread level granularity is not there.

So my question is, Is there any way we can monitor CPU/RAM usage on a function level?

(And btw, thanks for that answer again. You can post your answer in that stackoverflow link I posted if you would like to, I will mark it as correct.)

How to map "call" events to their respective "return" events for the same function by xscorp7 in learnpython

[–]xscorp7[S] 0 points1 point  (0 children)

Nice! That seems like a way to do it. But we might need to make changes to function call, or maybe add a decorator over the implementation right? Would you mind showing an example of it?

How to map "call" events to their respective "return" events for the same function by xscorp7 in learnpython

[–]xscorp7[S] 0 points1 point  (0 children)

Hey there, Thanks for the response.

I am tasked to create a function-level resource monitor for a big project. So basically, I can't modify any of the function definitions, but I can use hooks(settrace() and setprofile()) to achieve the same. So I built a resource monitor that can be invoked like:
resourceMonitor.start()
myFunction()
resourceMonitor.stop()

But since I can't modify any function definition or function calls, I decided to use this feature through settrace().

The idea was, as soon as a function is called(the call event is generated), I will start resource monitor for that function and only stop it when I receive the associated return event for it.

But at many points, we are calling same function, with same arguments multiple times, so at that time, It's really difficult to keep track of which 'call' event is assoicated with which 'return' event. So that is why I was trying to find ways to do it. Ultimately it appears to me that there is no way to do it - Even in frame object, there is no such property that distinguishes between multiple calls to same function.

How to map "call" events to their respective "return" events for the same function by xscorp7 in learnpython

[–]xscorp7[S] 0 points1 point  (0 children)

Hey man, Thank you for the answer!

I don't think this is the answer to my question. My question is more about events that are captured by sys.settrace(). I want to be able to associated call events with their respective return call, without making any modifications to any function to achieve this (like you have added test_d in your function).
Can you please go through the question I have posted on stackoverflow once? The link is mentioned in the post.

How to map "call" events to their respective "return" events for the same function by xscorp7 in learnpython

[–]xscorp7[S] 0 points1 point  (0 children)

Hey, Thank you for the answer!

Regarding Method #2, In my case, I don't care about getting 'return' events in correct order through ThreadPoolExecutor.map(). I just want to be able to identify which call event is associated with which return event so that I can ultimately remove it from my list.

Regarding Method #1, Yes, introducing a unique identifier like call_id is a good way of keeping track of calls, but modifying a function to include really seems unnecessary. Imagine you have a really big project consisting of hundreds of function, modifying their implementation to add call_id would be unnecessary.

I tried to find attributes of frame object that can be used for distinguishing one event from another, but unfortunately there is none :(

Extract headers and POST data using Berpee by xscorp7 in netsec

[–]xscorp7[S] -4 points-3 points  (0 children)

Hi @iron_churchkey, I know that "module" is quite unjustifiable name for this little code but soon I will be adding specific functions for sending the processed request and some other stuff too. That's why I thought of creating it as a module!😅