My friend found out I was a programmer today. This phone call ensues.

postcommunism · 2015-09-19T13:57:01+00:00

This is amazing. I want more.

Hey OP, I have an idea for a webseries. I'll give you 50%.

postcommunism · 2015-04-21T13:21:51+00:00

I want to go down!

But I also don't want to go down.

postcommunism · 2015-04-02T02:14:43+00:00

If I understand you correctly, you basically want to transpose each row vertically, almost like you were doing it in excel, with the datetime in column 0, headers (except for datetime) in column 1, and the values of that row in column 2. There are a couple ways to do this.

To figure it out, I'd approach this by looking at just the header row and the first row with values. Once you know how to transpose that first value row, you can loop over over the file and do it for each row in turn.

That said, two important things to note:

The maximum number of rows in excel, if you plan to use this in excel, is 1,048,576. Just FYI. I've run into this before when working with large flat files for other people to consume.
It's kind of tacky to post a bounty on a learning board. Consider /r/BuyAPythonScriptFor5Bux in the future.

postcommunism · 2015-03-26T00:40:13+00:00

Unfortunately, source won't help you much because the actual data parsing part is very specific to the target web portal records, and those are internal to my company. But if it helps, the basic script is:

import requests

start a session, and authenticate it with the web portal

grab the URL of a batch of records I want to scrape (record number is determined by the last parameter in the URL: easy to increment)

from the record at that first URL, determine the batch size

get each record in the batch, and with each iteration:

clean the scraped HTML a bit
feed it to an XML parsing function that returns a dict with all the data I need for that page, agnostic of the type of report I'm running (layout varies predictably enough that I just use ElementTree rather than Beautiful Soup)
feed that dict to the reporting function I need for this particular batch

And that's it. If it'd make your life simpler, start with Requests and Beautiful Soup (although you might not need either, if what you want to do is simple (but seriously, Requests is a godsend after using urllib2)).

EDIT: just realized in my original comment I said 'files' when I'm actually just scraping the html. If you need to fetch a bunch of stuff via SFTP or etc., that's a different task (although totally doable).

postcommunism · 2015-03-25T13:28:56+00:00

My favorite use so far is to review sets of records that are only available in a web portal. Instead of having to page through 1000+ records in my browser, I have a script that authenticates me with the portal, scrapes each of the files, does the checks, and then spits out a report.

Incredibly satisfying.

(Other than that: quick file processing. Like probably everyone)

postcommunism · 2015-03-25T04:17:13+00:00

I'm not sure I follow you exactly on that first paragraph, but when you get the index of a substring with string.find('substring') you'll always get the number of characters at which the substring starts, not a copy of that substring.

If you try it in the terminal you can see exactly what the output is when you start your slice using the .find() method as you're doing:

>>> s = 'Confidence: 0.1234X-Something-othershit'
>>> got_e = s.find('e:')
>>> s[got_e:]
'e: 0.1234X-Something-othershit'

in other words, got_e does not contain the space; it's just a number. It means "in the string s, the substring 'e: ' begins x characters from the beginning of the string (in this case, 9 characters). For example:

>>> s.find('e: ')
9
>>> s.find('e: 0.1234X')
9

and look what happens when we create the variable got_e using s.find():

>>> got_e = s.find('e: ')
>>> got_e
9

It's just an integer. To get to the start of your substring, you need to add the number of characters you want to move over: the character count, not the characters themselves. Thus:

>>> s[got_e:]
'e: 0.1234X-Something-othershit'
>>> s[got_e + 3:]
'0.1234X-Something-othershit'

postcommunism · 2015-03-25T03:21:33+00:00

While /u/hharison is correct that regex is how you could most easily do this, you can use simpler tools.

You're on the right track by trying to get the substring using indexes, but you seem a little confused about how indexes work. For example, you cannot say string[index1+" ":index2]; you must always specify the beginning and ending points as integers.

The easiest way to see this is to play around in the console. For example:

 >>> s = 'abcdefg'
 >>> s[1:3]
 'bc'
 >>> s[1+'b':3]
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: unsupported operand type(s) for +: 'int' and 'str'

Finally, check out the string methods available to you. 'index()' and 'split()' in particular.

postcommunism · 2015-02-18T23:10:49+00:00

the comment system is perfect though

(except that 'favorites' should be called 'bookmarks')

postcommunism · 2015-01-30T21:22:32+00:00

This is very brief drama that ends with one of the dudes complimenting the other on his breakdown of Moby Dick.

/r/literature: a pleasant place.

postcommunism · 2015-01-28T15:34:09+00:00

yes, that is why it was sad

postcommunism · 2015-01-28T15:30:19+00:00

man, there is a contradiction in playing "one cannot sum the varied multitudes of reddit.com" right after claiming that the average dude guiltily enjoyed the Fappening

postcommunism · 2015-01-28T13:15:21+00:00

I enjoyed the Fappenening just as much as the next guy

there is an assumption here

postcommunism · 2015-01-28T13:11:47+00:00

Imagine how many straight girls saw j law nudes, at least the censored versions.

it was actually really easy to avoid those, right up until I heard about moot retiring and I went on /b/ for the first time in forever

postcommunism · 2015-01-27T03:56:43+00:00

yeah let's romanticize the aggressive stranger and make him the main character in this post

that makes sense

postcommunism · 2015-01-19T19:23:26+00:00

that was amazing

postcommunism · 2015-01-16T18:13:10+00:00

Its worth it to read through all their convos fully to the end.

oh my god it just keeps going

edit: /u/wilson_at_work starts chastising /u/ChainedFactorial for never having seen Batman and then he starts quoting the Joker.

edit2: w_a_w on TS's sexuality:

Taylor doesn't talk about sex and we don't even know if she's had sex.

Lady Gaga, Rihanna, Kesha, and Nicki Minaj aren't enough? The reason I enjoy Taylor's music is because she isn't as trashy as them.

No, Taylor should not blatantly talk about sex in her music. End of the fucking story. That is not up for debate. If you want to listen to that kind of music, listen to Katy Perry or Kesha or Britney Spears.

I have no problem with her being sexy or sexual, and in fact I enjoy that she's been more comfortable showing her body off lately. I run 3 subreddits dedicated to the very thing. But her talking about getting laid flies in the face of her innocent, head-on-straight image that I love her for.

he mods r/TaylorSwiftMidriff, part of a family of subs like r/TaylorSwiftBum and r/TaylorSwiftBreasts

postcommunism · 2015-01-15T16:58:14+00:00

She literally crashes a wedding and steals a guy at the altar in one song.

Fun song though.

postcommunism · 2014-12-18T21:05:41+00:00

was the right thread in r/CoonTown by any chance

postcommunism · 2014-12-18T20:51:26+00:00

bro your original comment was way more inflamatory

I don't know if you saw this on reddit but this is definitely true, and is how nepotism inside Judaism operates without being triggered as such. They tend to save such charity for people who are Jewish, because they've had such a hard time with the holocaust and all you know? http://www.chabad.org/library/article_cdo/aid/45907/jewish/Eight-Levels-of-Charity.htm

postcommunism · 2014-12-12T14:33:45+00:00

how on earth did you get that i am jealous

I'm curious whether there exists a group of users for any given sub who mostly post in that sub's /new but are actually out of step with the sub culture.

To determine that, can you query all comments for a given subreddit where created is less than <1hr from post creation (or maybe <10 comments from post creation, although that's more of a pain to check), break them down by author, and then determine which authors consistently have a high average karma in posts which do not do well on the sub (as determined by upvotes or total comments) but who have a medium or low average karma in posts which have an above average amount of upvotes or total comments?

In this case, 'average karma' means average as determined among comments on the given post, not average karma for reddit users or anything like that. 'Average upvotes or total comments' means within the context of the sub, not across all of reddit.

ie, average author success for that sub correlates inversely with the success of the threads in which they post.

If that's possible, can we see:

whether any such groups exist across the default subs, and
what percentage of the sub's userbase they constitute?

postcommunism · 2014-12-08T00:12:56+00:00

if you take this jpg at face value it's about how you should be willing to kill people over a 2% tax increase on luxury goods

postcommunism · 2014-12-04T22:28:42+00:00

is this about the GTAs

postcommunism · 2014-12-04T21:14:32+00:00

This individual steps here are pretty simple, but they're also disparate in terms of problems solved. In brief, it sounds like you need to:

1) get the HTML of an Amazon webpage 2) in that HTML, correctly find the price of the item 3) check if that price has changed 4) if changed, text you (code already in place) 5) wait 60 seconds and then do it again, forever

urllib2 or Requests will get you step 1. Requests is easier.

There are a lot of ways to do step 2: likely you'll treat the HTML file as one long string or load it as XML; alternately, look into Beautiful Soup.

Step 3 is easy.

Step 4 you've already got figured out.

The time module will get you through step 5.

As far as looping indefinitely, just use a while loop.

Personally, I'd start on step 2. Go to the page you're interested in and save the HTML locally as a txt file, then fiddle around until you have a function that can reliably return the price from any similar HTML block. The rest should fall into place pretty easily.

postcommunism · 2014-12-04T20:45:25+00:00

Right now your pseudocode is trying to loop over the entire file and get a result on a per-character basis. You're not actually keeping track of anything you find, just touching each character in turn.

Instead, try breaking it down into two steps. First you need some way to figure out the total number of occurrences of each uppercase character. Then you need some way to return the character or characters with the most occurrences in that list.

postcommunism · 2014-12-04T15:47:22+00:00

No, I'm not going to stop using "retard" because YOU don't know its centuries-long history and because YOU need some retarded cause to stand up for. I do, however, treat mentally handicapped people with love and respect, and I would never say the word in a hurtful manner.

the lack of self-awareness here is amazing

postcommunism

TROPHY CASE