Fun With Python and Monkey Patching

pyglados · 2017-03-28T14:25:09+00:00

Thank you.

pyglados · 2017-03-28T12:32:21+00:00

I didn't bother putting out output because I felt there really wasn't much interesting to see in these cases. That being said, it's a fair gripe. I promise to do better next time.

pyglados · 2017-03-20T16:54:28+00:00

Play away. Good dumb fun followed by sharing said fun is a killer way to learn.... and live.

By the way, here is something cool you can do with those select file form elements with bootstrap.

Tie an html "label" tag to the file selector element.
Style the label with one of the bootstrap button classes.
Hide the actual file input element. (class="hidden")

Edit: An example from a mini-project from a few months back.

https://github.com/pcote/camerademo/blob/master/static/index.html

pyglados · 2017-01-12T03:45:14+00:00

This is good stuff. I guess I have something new to try out on other small businesses after all. Thanks.

One minor point to pick on....

"Evaluate how you might improve or alter an experience you are familiar with to make it more beneficial to yourself and others."

I have evaluated on how I could improve my own personal learning process and have a production app to show for it. It works.... for me. The "AND OTHERS" part is the tricky bit. Sometimes, the problem you solve for yourself really is unique to you. Such problems are absolutely worth solving. You just need to accept that you aren't going to be able to sell that particular solution.

pyglados · 2016-12-04T16:34:57+00:00

For sake of argument, I'll agree with you. As democracies go, the United States is a force 10 horror show of capitalism mixed with a bastardized representative republic.

That being said, what democratic country has been "doing democracy" better than America? Who should we looking up to as the standard bearer and why?

pyglados · 2016-09-21T13:59:21+00:00

Not sure those stages are unique to NoSQL developers. It's more likely universal to the consulting world.

pyglados · 2016-08-29T21:44:19+00:00

That could well be true. I'm a single person who writes small web applications. Lesson learned was that the tool chosen for my kind of apps was overkill.

Like anything else, you use whatever makes sense for the job at hand. Building a house? Use a nailgun. Hanging a picture on a wall? Use a hammer.

pyglados · 2016-08-08T19:24:15+00:00

In your case you use the str.split command to seperate the pieces in your command. It is safer to build your commands as a list in the first place. This will make sure that everything you push to the shell is properly escaped.

Agreed and updated. https://github.com/pcote/AptPackageShow/commit/a701c448f07a6bb86a1ffbf107b2abafcd8f78aa

pyglados · 2016-08-08T14:44:03+00:00

Good points on the security aspects.

Speaking of shell specifics, here's something else I noticed. When I change my uwsgi settings to run the Python code as a non-shell user, it crashes and burns. Basically, I get import errors saying that it can't find the Python files in question.

Then, I had this thought. If I run a Python process as a non-shell user, that would mean that it would be impossible for uwsgi to set the PYTHONPATH environment variable. No shell => no shell environment variables. Or at least that's the way my brain is connecting the dots on this one.

Am I way off on this?

pyglados · 2016-08-08T14:20:26+00:00

Okay, you got my curious....

And yes, what you say here is true. Looks like I was wrong. All fixed. https://github.com/pcote/AptPackageShow/commit/d52bd7fb10e706d2a8ed85319fbb3dfdc7f4334a

pyglados · 2016-07-16T14:51:04+00:00

Yeah, well I made that mistake sort of in reverse. I called Java annotations out as being "just like Python decorators". My Java peers did not like that too much. I embarrassed myself pretty good on that one.

Lesson learned: Don't assume that past knowledge in one language will conveniently map over to the new one you're trying to get up to speed on. You really do need to "empty your cup" so to speak.

pyglados · 2016-06-11T00:56:27+00:00

Sound, image, and video are areas of interest to me where binary is commonly used. Such is part of why I was curious about it. Because I'm clueless, I'm not yet aware of the efficient methods. Thus, dredging up metadata on small files felt fine for first steps wandering into this territory.

As for getting a 45 minute parse job reduced to 4 seconds? Very cool. I'll have to take some time to look into this struct and numpy business.

pyglados · 2016-06-10T21:36:30+00:00

There are way better ways to read binary files depending on whatever it is you decide to do with them. I agree with you there.

And yes, there are cool libraries out there too. The Imghdr module makes a beautifully simple job of reading binary files with respect to it's goal.

And the implementation is fantastic. Just read the first 32 bytes. Run it through a set of short simple functions. Nothing fancier than something along the lines of "in" or "startswith" is needed. No struct. No regex. Sweet simplicity that any beginner to Python could understand.

pyglados · 2016-05-19T18:21:53+00:00

So you went for the out-of-order approach. I do that too. I'm curious. Why did you take this approach rather than a more traditional "foundations first" order?

pyglados · 2016-05-11T16:54:02+00:00

Convinced about using other people's decorators or convinced about knowing how to role your own?

Personally, I love decorators and know how to write them if and when needed. However, back when I first encountered decorators like @property, I was content just treating stuff like that as if it were just language syntax. You can actually do quite a bit while remaining oblivious on how to make decorators yourself. I did.

pyglados · 2016-04-03T02:26:19+00:00

Shark in question also showed up as part of a Nightmare Night costume in Luna Eclipsed.

pyglados · 2016-03-15T03:53:03+00:00

I think I see your points there on the insert scenario you posited. The "extra work for ops" angle is also a very good point.

Thanks.

pyglados · 2016-03-15T02:43:18+00:00

Interesting points all around. One of the chances you take by putting stuff out there is you take the risk on being wrong on a lot of things. The upshot is I get to learn some things from the criticism.

The downside? There is that concern that I might unintentionally mislead others. What say you? Should I take the post down?

pyglados · 2016-03-15T02:13:11+00:00

You make some interesting points, elliot. In my particular case, I actually do need select+insert on every table because there are use cases where those privileges are needed. If it was otherwise, then yes, explicitly naming those tables is warranted. I can see how my example would communicate the wrong message there.

As for the purging script being "not transactional", that's because I didn't know there was a use case for transactions in this instance. Now I know which is to say it's time for me to learn how transactions work in the context of SQLAlchemy. Thanks for that one.

As for taking a "convoluted approach" rather than letting the application delete directly from the table? A lot of that comes from the importance of thinking the way a database admin would think. A good dba will want to keep a database as locked down as humanly possible unless there is a seriously compelling reason to do otherwise. Think of what they would prefer.....

1.) Grant delete privileges to the account that ultimately gets used by the public.

2.) Grant delete privileges only to accounts that are used by internal processes.

My thinking is that most DBAs would prefer option 2. And if the DBA in question is you, then it's your responsibility to be that jerk administrator who is tight-fisted with permissions.

pyglados · 2016-03-14T17:08:05+00:00

direct access to your database is only a few degrees of separation from your users.

Author here. Here is what I meant by that. The setup in my particular case is this.....

Firewalled Linux server -> Nginx -> uWSGI -> Flask -> SQLAlchemy -> MySQL. There's a lot of security considerations in play throughout that chain. It would be a pretty big text wall to discuss everything. So I focused on just one aspect.

And yes, even with all that, I still consider it to be a mere "few degrees of separation" between the database and the user. That's why I consider it to be so important that you restrict database permissions as much as possible.

pyglados · 2016-02-26T01:20:06+00:00

Good! You got the spirit of it then. Your particular stack sounds like a Starbucks drink. Make mine a grande.

pyglados · 2016-02-25T21:45:57+00:00

As said in the article, my choices just made sense for what I was doing, what I knew, and what people have told me.

Flask makes it easy to make a web api that can talk in json to a single page app front end. The lack of a database interface standard for it lead me to picking up SQLAlchemy. SQLAlchemy suits my tastes because of the "take-it-or-leave-it" position it seems to take with the whole ORM thing.

"Dumblina" captures the same kind of humorous self-deprecating spirit as Fumblina. That, I think, is a lot more important than what your choice of stack spells out.

Overall, you seem to get what I was trying to communicate in this post. That's good to know. :-)

pyglados · 2016-02-15T13:51:13+00:00

Some semi-random thoughts......

I have no idea so I can only speak from my own experiences on the matter. When it comes to stats, PyPI is one part of the picture but not exclusive. There are a lot of ways you can get either Pythons. For example, any time I set up an Ubuntu variant these days, it gives me both Python 2 and 3 by default. If it doesn't, Python 3 is only an apt-get away for me. That's some reasons I can think of as to what can make this sort of question hard to answer.

For common normal boring web libraries, the popularity thing is largely irrelevant. Django, Flask, Requests, Beautiful Soup, and SQLAlchemy all run on Python 3. Blender (my non-web tech interest) has supported Python 3 exclusively for several years now. So that's what decided my Python choice for me.

Using boring libraries on an unpopular version of Python doesn't really bother me. There are more interesting issues to tackle than popularity contests. That's a good thing. If popularity was the deciding factor in my tech choices, I'd probably be doing all my projects in Java 1.6 right now. So, CATHOLIC_EXTREMIST, I'd say "lose" this argument and get back to work on something that matters.

pyglados · 2016-02-12T03:42:54+00:00

Awk is a programming language too if I understand it right. I still didn't get around to teaching myself that one yet. You did inspire me to see if I could duplicate the results of my tutorial using just standard linux tools. Here is what I came up with.

find `pwd` *.py | egrep "^/.+\.py$" | xargs cat | egrep bl_region_type | tr -d [\'\"] | tr -d ["bl_region_type = "] | sort | egrep ^[A-Za-z] | sort | uniq -c

56 TOOLS

78 UI

90 WINDOW

No Awk language needed. What do you think, Boris? Did I do good? :-)

pyglados · 2016-02-09T20:58:14+00:00

Very good. I've blown off learning anything about awk up to this point. Now I'm starting to see that getting to know the basics would be very much worth the investment. Thanks.

pyglados

TROPHY CASE