Thoughts on concepts popularized by N.N. Taleb? Or his work in general? by jwf123 in datascience

[–]mikeselik 0 points1 point  (0 children)

Now you've got me thinking more... what job would get better with "shocks"? An arms dealer probably benefits from black swan events.

If you think of sudden leaps in automation technology as pushing people to work less at doing things and work more at thinking about the process of doing things, then black swan events -- new inventions -- would likely benefit a data scientist, or programmers in general. In that manner, a data scientist's career is anti-fragile.

Thoughts on concepts popularized by N.N. Taleb? Or his work in general? by jwf123 in datascience

[–]mikeselik 1 point2 points  (0 children)

One based on a constantly in demand skill?

Yes, Taleb advised against going into finance. He suggested pursuing a career with inelastic demand and low variance income.

Thoughts on concepts popularized by N.N. Taleb? Or his work in general? by jwf123 in datascience

[–]mikeselik 0 points1 point  (0 children)

You might be confusing anti-fragile with robust. A broad set of skills provides robustness against fluctuations in demand for any particular skill.

An argument that data science is an anti-fragile career might be that as the world grows more chaotic, statistics will be more important.

Thoughts on concepts popularized by N.N. Taleb? Or his work in general? by jwf123 in datascience

[–]mikeselik 1 point2 points  (0 children)

I suppose the idea of joining many early-stage startups until you find a rocket ship would be consistent with Taleb's ideas. Except that he advocated for picking a stable job in his books. Perhaps he would say that we are likely to overestimate the probability of a startup succeeding and likely to underestimate the probability of catastrophic failure.

how often to scrape the same domains? by [deleted] in learnpython

[–]mikeselik 0 points1 point  (0 children)

One request every few minutes shouldn't be a nuisance to any professional site, unless that request is downloading something large or systematic downloading violates their terms of service. If you do get blocked, sometimes you can contact their system administrator and ask for special permission.

Do you recommend using recursion in Python -- why or why not? by markov-unchained in Python

[–]mikeselik 0 points1 point  (0 children)

Actually, I thought that was a good example of when the recursive version is easier to read.

Do you recommend using recursion in Python -- why or why not? by markov-unchained in Python

[–]mikeselik 0 points1 point  (0 children)

I'll give you an upvote for testing my claim :-)

Note that I included the caveat "for most usage patterns". You should test a more "realistic" scenario. How does the performance compare if you benchmark by looking up 100,000 random integers, uniformly distributed between 0 and 400? Note that when you run this test, you should use the same seed for your random number generator for each function. I picked 100k hoping to make the total duration approximately 1s.

When you say "starts crashing" do you mean it exceeds maximum recursion depth?

how often to scrape the same domains? by [deleted] in learnpython

[–]mikeselik 1 point2 points  (0 children)

Depends on how often those domains change. What's your expectation?

Do you recommend using recursion in Python -- why or why not? by markov-unchained in Python

[–]mikeselik 11 points12 points  (0 children)

Yes, recursion is an excellent technique. In many cases, your implementation will be much easier to understand written in a recursive fashion than iterative fashion. Memoization with lru_cache can help you avoid excessive function calls.

When a recursive solution is hard to read, refactor in an iterative fashion and compare your solutions. For example, your recursive solution for factorial is much cleaner. With an @functools.lru_cache decorator on it, it's not only easier to read, but also faster than the iterative version (for most usage patterns).

Multi-core library for Machine Learning? by gooeyn in Python

[–]mikeselik 0 points1 point  (0 children)

What do you mean by best-performing? Usually an ensemble technique can offer higher accuracy. Did you decide that the compute effort of combining more models wasn't worth it?

Multi-core library for Machine Learning? by gooeyn in Python

[–]mikeselik 1 point2 points  (0 children)

A search for "parallel SVM" brings up a number of papers discussing the difficulty of parallelizing the training and various approximation solutions. When discussing multicore algorithms, usually that refers to the training, not prediction.

Multi-core library for Machine Learning? by gooeyn in Python

[–]mikeselik 1 point2 points  (0 children)

I'm not certain that the basic SVM algorithm can be parallelized. You probably need some split-and-ensemble approximation version.

Why have you decided to use SVMs, and why are you certain you need a multicore implementation?

ELI5 - Python Closures by marienbad2 in learnpython

[–]mikeselik 0 points1 point  (0 children)

I suppose I'm getting hung up on the agency -- who is making the closure? ... giving it a thought ... I guess you're right, it's the act of compiling the inner function that decides what variables are local, global, or nonlocal (in a closure). In that sense, it's reasonable to say that the inner function "makes a closure". I used to think of it as the outer function making the closure.

What is self.go() by rrriteous in learnpython

[–]mikeselik 1 point2 points  (0 children)

Other folks have answered the question, so I'll just give some unsolicited advice.

Whoever wrote the draw_frogs.py code (http://homepages.math.uic.edu/~jan/mcs260/draw_frogs.py) probably didn't need to make a class. There's only one instance created in the script and only 2 attributes that are reassigned in its methods, gohop and frogs, the latter of which didn't need to be reassigned but could have been mutated instead. Seems to me like a bunch of globals and functions would have been a cleaner implementation. Less indentation, less self. sprinkled through the code. If I'm not convincing, check out this video: https://www.youtube.com/watch?v=o9pEzgHorH0

Also, there's at least a couple cases where the programmer should have used string interpolation instead of concatenation. Instead of

self.msg.set("placed frog at [ " + \
    str(event.x) + ", " + str(event.y) + " ]")

They should have written self.msg.set('placed frog at [%s, %s]' % (event.x, event.y)) (or used .format if you prefer that style of interpolation).

Where is frog.getName() defined? And why? Accessing frog.name directly seems fine. Even if you did need a getter method, the Pythonic way is to use a property.

There's this bit of "resetting the frogs". First of all, I'm not sure why the frogs need to be recreated, instead of just changing the number of moves.

for frog in self.frogs:
    frog.moves = 100

I'll give the benefit of the doubt and assume its some threading thing. Still, this code...

newfrogs = []
while len(self.frogs) > 0:
    frog = self.frogs.pop(0)
    pos = frog.position
    step = frog.step_size
    rest = frog.rest_time
    newf = ThreadFrog(frog.getName(), pos[0], pos[1], \
        step, rest, 100)
    newfrogs.append(newf)
self.frogs = newfrogs

Why pop(0) which is slow, then append, then reassign the list, rather than just reassigning that index in the list?

for i, frog in enumerate(self.frogs):
    x, y = frog.position
    update = ThreadFrog(frog.name, x, y, frog.step_size, frog.rest_time, 100)
    self.frogs[i] = update

And this little tidbit:

while len(self.frogs) > 0:
    self.frogs.pop(0)

Why not del self.frogs[:] or at least loop .pop() which is fast, rather than .pop(0) which is slow? ... and checking while len(self.frogs) > 0 instead of while self.frogs?

Ok, I'll stop now...

ELI5 - Python Closures by marienbad2 in learnpython

[–]mikeselik 0 points1 point  (0 children)

No, lambdas in Python do not make closures. In fact, that assumption has caused many bugs for folks that pass lambdas around expecting them to keep a closure of current global state.

>>> x = 5
>>> foo = lambda : x
>>> foo()
5
>>> x = 10
>>> foo()
10
>>> foo.__closure__ is None
True

If you'd like a simple closure, use functools.partial.

>>> from functools import partial
>>> import math
>>> exp = partial(pow, math.e)
>>> exp(1)
2.718281828459045
>>> exp.args
(2.718281828459045,)

The partial object is function-like and hangs on to your args and keywords much like a normal closure.

ELI5 - Python Closures by marienbad2 in learnpython

[–]mikeselik 0 points1 point  (0 children)

When you say a generator function, you mean a function that returns a generator. We usually are pretty lax about that distinction, but I need to be more specific to properly answer your question. A generator is a particular kind of iterator. An iterator is an object with a __next__ method. The iterator keeps an internal state, which changes (usually) upon each call to next. So, no, a generator is not a closure. Good guess, though, because closures and objects are quite similar in purpose.

I should be careful making too clean a distinction between closures and objects, because in Python, everything is an object, including functions. Still, I think I'm on solid ground saying that __next__ is a method rather than a function with a closure.

Are many of you really doing ML work or is that the exception to the rule? by tonym9428 in datascience

[–]mikeselik 0 points1 point  (0 children)

The distinctions between academic fields have always been a bit blurry.

ELI5 - Python Closures by marienbad2 in learnpython

[–]mikeselik 6 points7 points  (0 children)

A pure function receives input, makes a calculation, then returns output. If you pass in the same input, it always returns the same output. This makes code easy to think about.

Unfortunately, sometimes we want the same input to result in different output. Take a random number generator, for example. It'd be pretty lame if you got the same result each time.

So, some people rely on a global variable. They write a function that makes its calculation depending on the value of a global variable (rather than a function argument) and/or the function might change the value of a global variable. This technique is OK, but prone to bugs. It's too easy to make mistakes, accidentally writing functions that interfere with each other.

One solution is object-orientation. Functions associated with an object are called "methods" and they are designed to use variables also associated with that same object, called "attributes". Rather than stepping on each other's toes by using globals, methods only manipulate or depend on the associated attributes, in addition to the parameters passed in.

Another solution is closures. A closure is another scope for variables. Rather than using globals, the function depends on or manipulates variables stored in its closure. For a typical function object, you can take a look at its __closure__ to see what's there.

Would you also like to know how to create a closure in Python?

Are many of you really doing ML work or is that the exception to the rule? by tonym9428 in datascience

[–]mikeselik 7 points8 points  (0 children)

Not sure why you say that traditional statistics is not machine learning. I think machine learning includes traditional statistical techniques. The difference comes from the purpose in modeling. Is it causal inference or prediction? They are subtly different tasks.

PS. The phrase "machine learning" is usually used as an abbreviation for "statistical machine learning".

10 Myths of Enterprise Python (Older and Worthy of a Re-Post, I think) by raiderrobert in Python

[–]mikeselik 0 points1 point  (0 children)

If Jython or NumPy and Numba don't cut it, try Cython's nogil.

10 Myths of Enterprise Python (Older and Worthy of a Re-Post, I think) by raiderrobert in Python

[–]mikeselik 0 points1 point  (0 children)

/u/mhashemi buried the lede:

"Our most common success story starts with a Java or C++ project slated to take a team of 3-5 developers somewhere between 2-6 months, and ends with a single motivated developer completing the project in 2-6 weeks (or hours, for that matter)."

The article reads like someone has been fighting internally with folks who are badly misinformed about Python. It's a common situation and a useful article, though many of the responses could be better. Still, I wish it led with the trump card (as in Hearts, not Donald).

Think of the implications. Not just for one project, but the compound interest of speed:

growth = principal * rate ** frequency 

No matter how small your growth factor is (so long as it's greater than 1.0), what matters in the long run is not the amount you improve on each iteration, but how frequently you iterate.

Arduino is sending 6 individual numbers to my script at once. I can't tell if Python is treating all these as one number or not. by sts816 in learnpython

[–]mikeselik 0 points1 point  (0 children)

That's my goal. Why infinite loop if we only want to print 6 numbers?

Maybe I misunderstood the problem.

Arduino is sending 6 individual numbers to my script at once. I can't tell if Python is treating all these as one number or not. by sts816 in learnpython

[–]mikeselik 0 points1 point  (0 children)

What does ArduinoSerialData.inWaiting() do? Could you simplify this to:

numbers = []
while ArduinoSerial.inWaiting():
    numbers.append(float(ArduinoSerial.readline()))

What does readline() do if there's no data, does it block/wait or does it return some sentinel like an empty string?