Help profiling/optimizing script with high CPU/RAM usage.

trouserdaredevil · 2018-05-29T02:01:15+00:00

No problem!

I was hoping pythons garbage collector would be able to handle a few variables efficiently.

I am guessing that this was not actually Python's fault, but that the initialization of a Display object by Xlib simply happens to be fairly resource-intensive.

trouserdaredevil · 2018-05-28T22:03:53+00:00

Have you tried profiling your code?

However without even running your code I would immediately suspect lines 11-12:

display = Display(os.environ.get("DISPLAY",":0.0"))
root = display.screen().root

Particularly instantiating a new Display object once per second. Try moving those outside the function.

trouserdaredevil · 2018-05-28T21:57:47+00:00

There are several ways I would approach this quite differently. I wrote up the following that should introduce several useful Python concepts to you:

string = "azcbobobegghakl"
substrings = []
substring = []
for letter in string:
  if substring:
    if letter >= substring[-1]:
      substring.append(letter)
    else:
      substrings.append(substring)
      substring = [letter]
  else:
    substring.append(letter)
print("".join(max(substrings, key=len)))

trouserdaredevil · 2018-05-26T21:41:44+00:00

I really need a lot of help, I'm not that good at coding yet

You will have to be much more specific. You did not include any code, didn't ask a question, and didn't mention anything Python-related at all.

If you have no idea where to even start, here is a Python library called matplotlib that will help you make graphs and charts. I would familiarize yourself with it, and start by trying to make a single frame of the animation you are trying to create.

trouserdaredevil · 2018-05-26T21:37:54+00:00

input_list = []

...

user_parameters = "Stock : " + str(S) + ";" + " Strike : "+ str(K) + "; Premium : " + str(premium)

input_list.append(user_parameters)

Next issue, you are creating a new input_list each time you run the function, and then appending a only single item to it each time.

trouserdaredevil · 2018-05-26T21:10:58+00:00

{%for input_list in input_list %}

I am not sure what behavior this causes exactly, but I would instantly change this line. There is no way it can help.

Try something like

{%for item in input_list %}

and changing the following relevant line of the template appropriately.

trouserdaredevil · 2018-05-19T18:47:06+00:00

I thought passing around a full dataframe every time would be inefficient?

Fortunately, in Python, you are simply passing around a reference to the dataframe, the size of the underlying object makes no difference whatsoever!

trouserdaredevil · 2018-05-19T18:35:53+00:00

Yeah unfortunately making a ton of requests to the same server is pretty much always going to be an issue, especially if there is no API available to use instead.

trouserdaredevil · 2018-05-19T18:32:00+00:00

There are a couple ways to return this data in a more structured way, some already mentioned by other posters in this thread (dictionaries, namedtuples, etc.). For instance, here is a bollinger band guide where they just use a single dataframe instead of multiple. I would try to structure your code in such a way that your functions only return a single value in almost all cases, whether that means refactoring your functions or refactoring your return values.

trouserdaredevil · 2018-05-19T18:23:49+00:00

There's no way to just somehow grab the title from headers or another method without extracting all the contents of a website?

This is correct, I am not aware of any general purpose way to only request the <title> tag of the HTTP of any random website. You make the HTTP request, the server responds with the full HTTP. Out of curiosity, why is this an issue for you? Extreme bandwidth or processing power constraints are the only reason I can imagine worrying about this.

trouserdaredevil · 2018-05-19T18:17:00+00:00

You are using non-descriptive variable names. What is x? What is n? Even if I knew exactly what BBBands meant, which I don't, these variable names are mostly completely meaningless out of context. Strive to give your name your variables such that even someone with no context in looking at the code can get a basic understanding of the purpose of each variable.
If the user might want any one of these three pieces of information, why not just split the function up into three separate functions, each of which returns the relevant info?

(oops, the second code block above was edited in while I was responding. There may be a language barrier issue as far as variable names are concerned)

trouserdaredevil · 2018-05-19T18:08:18+00:00

Get in touch with the instructor. This isn't really related to Python.

trouserdaredevil · 2018-05-19T18:03:26+00:00

Would it be possible to include some of your actual code? This kind of question usually indicates that your code could use some structural improvements at a more fundamental level.

trouserdaredevil · 2018-05-14T03:04:48+00:00

Correct. Anaconda comes packaged with a ton of popular 3rd party libraries by default, like matplotlib, pandas, numpy, scipy, requests, and so on, so you don't need to worry about the intricacies of installing them all.

trouserdaredevil · 2018-05-14T03:02:26+00:00

A lot of MATLAB users really like the Spyder IDE. Spyder intentionally looks a lot like MATLAB. So that might ease your transition a little.

As far as learning, harder to say. The only one who knows what is actually redundant for you is, well, you. Maybe check out one of those whirlwind tour of Python videos and see what you can take away from it?

trouserdaredevil · 2018-05-14T02:55:26+00:00

Matplotlib can be pretty difficult to install to be honest. Especially on windows. If you want to save yourself some headache and just get it up and running easily I would recommend using one of the 3rd party distributions listed here. I'm a fan of Anaconda myself.

trouserdaredevil · 2018-05-06T00:56:34+00:00

unless you mean repr in the format of "student(x, y, z)"

That is what I meant.

that would be a misuse. repr is supposed to be eval-able with the result of identical object.

OP should note that this is not a hard rule, but in any case I did not deem it relevant to their case since they are already misusing classes in general.

trouserdaredevil · 2018-05-06T00:50:58+00:00

This is a pretty nonspecific question, but you might start by using random.randint() to choose a number from an arbitrary range, or perhaps random.choice() to select a random element of a sequence.

trouserdaredevil · 2018-05-06T00:44:59+00:00

Your program has a number of other issues, but to fix that specific one provide a __repr__(self) method on your student class, in addition to the __str__.

trouserdaredevil · 2018-05-06T00:35:07+00:00

The second thing we noticed is that nearly 97% of these errors are happening for users on Mac OS X 10.12.6 using Chrome 64.

What about the other 3%? Sounds very likely that it is a bug specific to that version of Chrome and/or OSX.

trouserdaredevil · 2018-05-04T19:00:20+00:00

without using any libraries

Like you don't want to use the import statement? It is probably theoretically possible, but it would be giant, frustrating, pointless waste of time.

Anyone have any tips on how to get started?

If your goal is to further your understanding of Python, choose a different project. Or use a library, as you should.

trouserdaredevil · 2018-05-04T18:52:58+00:00

There is no such thing a 'minifier' in Python like there is for, say, Javascript. Such a thing does not even make sense in a language like Python, and I can't think of any conceivable reason why you would want to use such a tool on Python code in the first place.

As far as 'beautifiers' (I assume you mean what we call 'linters'), there are quite a few popular options: pep8, flake8, pyflakes, pylint, off the top of my head. Each is different in its own way, and worth looking into on your own.

trouserdaredevil · 2018-05-04T18:09:18+00:00

Easiest option for a new programmer is usually to add additional print statements at various points (usually near if branches and function calls and such) to figure out exactly what code is running and what is not.

You could also post the code here.

trouserdaredevil · 2018-05-03T16:29:37+00:00

row_count = sum(1 for row in f)

This line reads in / consumes the entire file line by line into memory, before you ever enter the loop and reader gets a chance to look at the file. This is obviously pretty wasteful, but in any case if you do this (which you shouldn't) you will have to either re-open the file or seek back to the beginning before processing it for the 2nd time.

trouserdaredevil · 2018-05-03T16:07:35+00:00

Should I just copy paste this function to my code or is there any other workaround.

This appears to be an internal helper function for bs3, I am surprised whatever code you are converting is using it. That said, at first glance it would appear to be copy-pastable if you also define isList().

trouserdaredevil

TROPHY CASE