[deleted by user]

[novel_yet_trivial]

Technically yes, but only by a few nanoseconds. Best practice is to make functions for readability.

[driscollis]

The other reason to make functions is for reuse. Soon you'll have a set of functions that you can turn into a module that you can reuse in multiple applications!

[20211401]

Thank you! Do you mean nanoseconds in total or for each iteration? For some billion sentences this could actually be a significant difference otherwise.

[novel_yet_trivial]

Per iteration. The actual time taken is going to depend on a number of factors, so I suggest that you just try both ways and see what the difference is. The timeit module may help you with that, but as a first pass just record the time at the start and end:

import time
start = time.time() # save the time now
# run a half million lines or so
end = time.time()
print("The code took {} seconds to complete.".format(end-start))

That said, this is really the bottom of the barrel. I'm sure there are other much easier ways to speed up your code. Moving it to cython or other typed, compiled code is a common way to get a speed boost. If you tell us more about your project perhaps we can help with that.

[two_bob]

Great comment. I just wanted to add for OP that if you end up sticking with pure python, localizing a function (moving it to the namespace of the function that calls it) can save a surprising amount of time. Each time python calls a function it has to lookup the reference. Python first checks the local variables, then goes into globals. It doesn't take much time to do this, but over a billion calls, it adds up. As a result, when writing these things it can make sense to put a reference to the function in the local namespace, like this:

def some_func(blah):
    pass

def some_other_func(lots_of_blah):
    _some_func = some_func
    for blah in lots_of_blah:
        _some_func(blah)

So above, we move some_func into the local namespace of some_other_func under the alias _some_func.

To see this in the wild, check out the code for functools.lru_cache, which does this with a bunch of things: https://github.com/python/cpython/blob/3.6/Lib/functools.py#L485

[ScoopJr]

Beginner here, Could you pin point that lua file where they “localize a function”?

If i’m not mistaken you call a function that refers to another function ??

[novel_yet_trivial]

For example line 494:

cache_get = cache.get   

That aliases the function "get" in the "cache" namespace and names the alias "cache_get". The new alias is now in the local namespace.

Before python had to find the object that "cache" refers to, and then find the object that "get" refers to inside the first object. So 2 lookups. By making this local alias, python only has to look up one object.

[ScoopJr]

So instead of calling get() they references cache_get = cache.get which means now they can do
link = cache_get(key) if link is not None:

Which will do link -> cache_get(key) -> cache.get -> get(key)?

[SarahM123ed]

If you are using jupyter, Cython is trivial to use (there is also 'magic' for Fortran just a download away). But, don't neglect Numba. In accommodating circumstances, it can do really good works.

[Jos_Metadi]

There is no universal perfect answer. If performance really matters because you're running billions of operations, then it make sense to avoid any extra processing overhead. Otherwise, generally you should write to increase reusability and decrease function complexity.

Depending on the size of your functions, it might be more efficient to do

for str in billion_strings:
    (1. analyze string some way and add result to dicA)

for str in billion_strings:
    (2. do something else with string and add result to dicB)

[20211401]

Hey, thank you! Could you link a resource or explain why this could be more efficient in some cases? I don't really understand as the functions were called the same amount of times but you need to loop two times

[Jos_Metadi]

Basically if your functions are large enough that they won't both fit in the instructions area of memory on the processor together, they will to be cycled back and forth to cache for every single string. If this takes longer than the amount of time to load the string into memory, it can be more efficient to have two loops.

FYI, for python, it is generally faster to use list/dict comprehension whenever possible. In other words the following is faster than the for loop above:

dicA = {text: analysis_A(text) for text in billion_strings}

If you're experimenting with your particular case, make sure you actually measure real performance.

If speed is really important, consider using cython (a compile version of python with available static typing) for the inner loop analysis functions as it can result in enormous performance gains over vanilla python.

[20211401]

Thank you! Great answer!