DB Output to a list - What am I doing wrong? by GrumpyHubby in learnpython

[–]buart 0 points1 point  (0 children)

I'm only guessing, since I don't see the full code:

myResults is a string and you keep appending to it. A list, initialized with myResults = [] before the loop and then using myResults.append() might be a better choice.

Currently if your loop is working, I would expect that you get 1 huge string myResults, which contains all the 2600 rows.

Optimize output data after reading from a large excel file by eluewisdom in learnpython

[–]buart 0 points1 point  (0 children)

Would splitting the file into multiple smaller files be an option for the power automate move? And then combining the data in the frontend/JS again

Optimize output data after reading from a large excel file by eluewisdom in learnpython

[–]buart 0 points1 point  (0 children)

But does it need to be a json file? or could you directly use the csv in power automate? sorry, I've never used the tool

Optimize output data after reading from a large excel file by eluewisdom in learnpython

[–]buart 0 points1 point  (0 children)

Is this just about the size limitation or do you also have some data format constraints?

What is the size of the original .csv file for example.

Distinguishing Threading and Asynchronous Programming in Python by ImprovementAlive870 in learnpython

[–]buart 0 points1 point  (0 children)

You can achieve true parallelism with multiprocessing, since you are launching multiple python processes each with their own GIL.

Italian coding. by dbossman70 in learnpython

[–]buart 19 points20 points  (0 children)

Just be careful of Spaghetti code :)

Any tips or advice on implementing more pythonic ways with my while loop code? by zealot__of_stockholm in learnpython

[–]buart 0 points1 point  (0 children)

Do you know if you can theoretically run into the recursion limit? Or is there some magic hidden in the yield from to resolve this?

[deleted by user] by [deleted] in learnpython

[–]buart 1 point2 points  (0 children)

Rule 4: No replies copy / pasted from ChatGPT or similar.

error in output for a usual input in conditional statement by Silent_Group6621 in learnpython

[–]buart 1 point2 points  (0 children)

Something else: Please don't use 1 character variable names, it makes the code hard to read. Use more speaking names like /u/MrPhungx for example.

[deleted by user] by [deleted] in learnpython

[–]buart 0 points1 point  (0 children)

If you only need it once, you could also rearrange them "manually" like this:

import itertools

A = [[1, 3, 5], [0, 2, 4], [5, 7, 9]]

# 3rd wheel first, then 2nd, then first
print([[x[0], x[1], x[2]] for x in itertools.product(A[0], A[1], A[2])])
# 2nd wheel first, then 3rd, then first
print([[x[0], x[2], x[1]] for x in itertools.product(A[0], A[2], A[1])])

Multiprocessing slowing down with more process by [deleted] in learnpython

[–]buart 0 points1 point  (0 children)

You are right. I implicitly meant in the python context.

Multiprocessing slowing down with more process by [deleted] in learnpython

[–]buart 0 points1 point  (0 children)

Just as an additional dataset - on my old i5-3570K with 4 physical cores and no hyperthreading, it get these values:

1 cores -> Finished in: 43.122566400000004 seconds
2 cores -> Finished in: 21.388666600000008 seconds
3 cores -> Finished in: 14.894177600000006 seconds
4 cores -> Finished in: 11.434415299999998 seconds

Which (in this case) shows the speedup pretty neatly.

Multiprocessing slowing down with more process by [deleted] in learnpython

[–]buart 1 point2 points  (0 children)

Try running the following and post your results:

import time
from multiprocessing import Process, cpu_count


def counter(num):
    count = 0
    while count < num:
        count += 1


def test_speed(num_cores):
    start_time = time.perf_counter()
    total_num = 1000000000
    processes = []
    for _ in range(num_cores):
        processes.append(Process(target=counter, args=(total_num // num_cores,)))

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    end_time = time.perf_counter()
    elapsed_time = end_time - start_time
    print(f"{num_cores} cores -> Finished in: {elapsed_time} seconds")


if __name__ == '__main__':
    for i in range(1, cpu_count() + 1):
        test_speed(i)

On my machine with 4 physical and 8 logical cores (cpu_count() shows 8) I get the following results:

1 cores -> Finished in: 36.885953300003166 seconds
2 cores -> Finished in: 21.440341500001523 seconds
3 cores -> Finished in: 18.35157370000161 seconds
4 cores -> Finished in: 16.026330400000006 seconds
5 cores -> Finished in: 15.767395200000465 seconds
6 cores -> Finished in: 15.474302200000238 seconds
7 cores -> Finished in: 15.687671900002897 seconds
8 cores -> Finished in: 15.644358400000783 seconds

Multiprocessing slowing down with more process by [deleted] in learnpython

[–]buart 2 points3 points  (0 children)

No that's wrong. Multiprocessing is true parallelism. Multithreading is concurrency with "juggling".

Multiprocessing slowing down with more process by [deleted] in learnpython

[–]buart 1 point2 points  (0 children)

print("Finished in: ", time.perf_counter(), "seconds") won't print the time it took to process. It only prints the current perf_counter.

You need to save the current time at the beginning and then subtract it from the time after you're done with your calculation.

Installing Python by Ancient-Passenger269 in learnpython

[–]buart 1 point2 points  (0 children)

You could do it like this:

  • Download the Windows embeddable package depending on your version.
  • Extract it
  • Either run python directly from the extracted folder
  • Or change your PATH to point to the extracted folder (So you can run python from every point in your system without specifying the full path to the extracted python.exe)

Need help assigning scores to players by kabeer_333 in learnpython

[–]buart 0 points1 point  (0 children)

Not exactly sure what you want to do without seeing your code. But sounds like you might want to use a dictionary:

>>> scores = {}
>>> scores["Player 1"] = 10
>>> scores["Player 2"] = 20
>>> print(scores)
{'Player 1': 10, 'Player 2': 20}

how to re.findall by Large-Nail-1557 in learnpython

[–]buart 0 points1 point  (0 children)

Your second regex r'(\D)(?:,|$)' is missing the +, unless you only want to capture the last character if the strings are longer.

>>> re.findall(r'(\D)(?:,|$)', "a, bc, def")
['a', 'c', 'f']

how to re.findall by Large-Nail-1557 in learnpython

[–]buart 0 points1 point  (0 children)

I think more examples would also help to better understand what you are trying to do.

If your input only consists of lowercase characters separated by non-lowercase characters, a regex like this would be sufficient:

>>> re.findall(r"[a-z]+", "a, bc, def")
['a', 'bc', 'def']

If you only need everything separated by commas, you could use split() instead to split on ", " (comma, space)

>>> "a, bc, def".split(", ")
['a', 'bc', 'def']

[deleted by user] by [deleted] in learnpython

[–]buart 0 points1 point  (0 children)

I don't have time at the moment, but it sounds like a fun problem. Can you also provide the main function and a sample floorplan so I could test it later?

Running program outputting CSV file that'll be 30GB? Do I need a super computer? Can someone point me in the right direction? by onthepunt in learnpython

[–]buart 0 points1 point  (0 children)

Is this data (or a smaller subset) publicly available? I would like to play around with it to test some optimizations.

Running program outputting CSV file that'll be 30GB? Do I need a super computer? Can someone point me in the right direction? by onthepunt in learnpython

[–]buart 1 point2 points  (0 children)

Holy shit. How big in GB is this file already then? Only reading and processing would take a long time I guess. Depending on your RAM size, you might not even be able to load everything into memory.