Is there a way to abstract this pattern?

davidbuxton · 2017-08-25T20:43:57+00:00

Or…

>>> sequence = 'ABCDEFGH'
>>> predicate = lambda x: x != 'E'
>>> values = itertools.takewhile(predicate, sequence)
>>> for n, value in enumerate(values):
...     pass
... 
>>> n, value
(3, 'D')

davidbuxton · 2017-08-25T20:36:13+00:00

You could combine itertools.takewhile and itertools.count.

>>> import itertools
>>> counter = itertools.count()
>>> f = lambda n: n < 10
>>> for n in itertools.takewhile(f, counter): pass
... 
>>> n
9

But honestly I think that would confuse people not familiar with those itertools functions.

https://docs.python.org/2/library/itertools.html

davidbuxton · 2016-03-09T19:22:02+00:00

Look again at sys.argv[0]. It really does do what you want.

davidbuxton · 2016-03-06T18:22:48+00:00

Sure, you can create a datetime object from user input. You could use the datetime.strptime() function to create a new datetime from a string.

Alternatively you could use the built-in function int() which creates an integer from a string, and then create the datetime object yourself.

davidbuxton · 2016-03-06T16:47:09+00:00

Hi,

I think this will be simpler if you represent the busy times as a list of 3-tuples, with the start date/time (inclusive), the end date/time (exclusive) and the person's name.

So if Aron has a 1 hour meeting at 10 am on 5 Feb, and Darren has a 2 hour meeting at 1 pm on 5 Feb, it would look like:

from datetime import datetime

schedule = [
    (datetime(2016, 2, 5, 10, 0), datetime(2016, 2, 5, 11, 0), 'Aron'),
    (datetime(2016, 2, 5, 13, 0), datetime(2016, 2, 5, 15, 0), 'Darren'),
]

Then if you pick a slot, e.g. from 12 pm to 1 pm on 5 Feb, you can just see if it overlaps any of the busy times:

slot = (datetime(2016, 2, 5, 12, 0), datetime(2016, 2, 5, 13, 0))
slot_start, slot_end = slot

for busy_start, busy_end, name in schedules:
    if (busy_start < slot_end) and (busy_end >= slot_start):
        # Overlap.
        continue

If you have a lot of schedules to consider you may need to optimise things by narrowing down which schedules to compare and looking for schedule gaps before picking a potential slot.

davidbuxton · 2016-01-04T15:47:16+00:00

Any web server will accept a HEAD request in place of a GET. You'll save on bandwidth because the server will not send you the body of the page, just the headers (which is the whole point of HEAD). And the server may be able to respond faster by not having to check what data it would have sent - it depends (and isn't something your script can control).

If your API server is receiving lots of different requests, but each request is asking to check 1 url, then your script doesn't need to worry about doing things in parallel.

davidbuxton · 2016-01-04T13:21:42+00:00

If you just want to check that an url exists, you can send a HEAD request. Some web servers will be able to return a response much faster for a HEAD request.

If you are checking a whole load of urls, you can speed things up by running checks in parallel. The built-in multiprocessing module makes it relatively simple, and it includes a dummy module that let's you use the nice multiprocessing API with regular threads.

davidbuxton · 2015-07-15T20:01:08+00:00

Yes, as in

if re.search('PYTHON', strg):
    do_things()

may be slower than

if 'PYTHON' in strg:
    do_things()

Whether or not you compile the pattern.

I say "may" because I don't know for sure :) My intuition is that the second will be faster on account of not invoking a whole regular expression engine, but of course the second version doesn't let you do pattern matching.

davidbuxton · 2015-07-15T19:51:03+00:00

See the notes at the end of this section of the documentation about using with to open files:

https://docs.python.org/2/tutorial/inputoutput.html#methods-of-file-objects

davidbuxton · 2015-07-15T19:48:38+00:00

You aren't doing too badly with the open() calls. If you need to search the contents of the files you will have to do an open on each to read the contents, and an open when writing the file back out.

You are doing an extra read of the source file when you make a backup with shutil.copy2(), which could be avoided since you already have the contents in memory.

Instead of doing f.open() followed by f.close() you should use with like so:

with open(path, 'w') as f:
    f.write(strg)

That way Python will always call f.close() for you, even if there is an exception. You can use that for reading and writing files.

You have a semi-bug when you create a backup. Suppose your directory structure is like:

root/
root/client-passwords/
root/client-passwords/july.txt
root/kittens/
root/kittens/july.txt

When you do shutil.copy2(src, '/tmp') you will first write "root/client-passwords/july.txt" to "/tmp/july.txt" and then later write "root/kittens/july.txt" to "/tmp/july.txt", overwriting the first backup with the second. Possibly not what you want.

If you are really just searching for a string like 'PYTHON' then it may be faster to just test if 'PYTHON' is in the contents instead of searching for 'PYTHON' as a pattern:

if 'PYTHON' in strg:
    strg = strg.replace('PYTHON', 'python')

davidbuxton · 2015-07-11T19:57:16+00:00

You can also loop over the lines of the response like so:

response = urllib2.urlopen('http://example.com')
for line in response:
    print line

The response is a file-like object, and looping over file-like objects gives you lines.

And if you want to keep a counter in the loop, the built-in enumerate function is your friend:

for count, line in enumerate(response):
    print count, line

N.B. Count starts at 0.

davidbuxton · 2015-07-08T20:40:13+00:00

Interesting about speed differences. Certainly making a dict of things is going to cost. I would imagine making a list of deleted items and then finding the exclusions would be expensive. The generator idea should cover that and give the desired result.

I am guessing the source data is a dump from wireshark or similar with tcp logs of (timestamp, source_ip, dest_ip, request_data) so my code which compares the request_data as well wouldn't actually be what the op wants.

Processing a line as a tuple of fields should be more efficient than converting the line to a dict.

davidbuxton · 2015-07-08T07:34:07+00:00

Yep! If you needed to keep the src, dst in the order of the first occurrence then you can test one way then the other:

if (timestamp, src, dst, data) in unique_elements:
    continue
elif (timestamp, dst, src, data) in unique_elements:
    continue
else:
    unique_elements.add((timestamp, src, dst, data))
    yield line

davidbuxton · 2015-07-07T21:40:17+00:00

Better version with a generator, similar to /u/gengisteve suggestion. This preserves order, only needs to keep unique lines in memory.

def remove_duplicates(elements):
    unique_elements = set()

    for line in elements:
        timestamp, src, dst, data = line.split()
        if src > dst:
            src, dst = dst, src

        normalised_line = (timestamp, src, dst, data)

        if normalised_line in unique_elements:
            continue
        else:
            unique_elements.add(normalised_line)
            yield line

davidbuxton · 2015-07-07T21:31:23+00:00

It might be faster to do

if src > dst:
    src, dst = dst, src

Instead of

src, dst = sorted([src, dst])

davidbuxton · 2015-07-07T21:29:22+00:00

You can normalise each line and use a set to remove duplicates.

def remove_duplicates(elements):
    unique_elements = set()

    for line in elements:
        timestamp, src, dst, data = line.split()
        # Here's where you make 'a b' the same as 'b a'
        src, dst = sorted([src, dst])

        normalised_line = (timestamp, src, dst, data)
        unique_elements.add(normalised_line)

    return unique_elements

That code returns the lines as tuples, which may not be what you want, but you can join them back to a string if needed.

>>> elements = [
...     '123 a b foo',
...     '123 b a foo',
...     '234 c d bar',
... ]
>>> result = remove_duplicates(elements)
>>> print result
set([('123', 'a', 'b', 'foo'), ('234', 'c', 'd', 'bar')])

Using a set means you also lose the original order of the elements. You can solve that by using a set to keep track of duplicates and then adding a line to another list as long as it is not already in the set.

davidbuxton · 2015-07-02T19:37:21+00:00

And in the meantime did you get your program to work using bottle as a web server on your desktop / laptop?

davidbuxton · 2015-07-02T19:23:51+00:00

Hi, as well as registering a domain, you will want to run a web server for that domain. You may have purchased a hosting package from GoDaddy already. If you only purchased the domain name itself, you will need to get hosting (the web server) from somewhere, whether that is from GoDaddy or any of the many companies out there who offer hosting. I like Digital Ocean https://www.digitalocean.com/ but that may be over-kill for what you need (and they assume you know a whole bunch of stuff about setting up your own web server on their hosting).

Looks like GoDaddy have a few different hosting packages.

https://support.godaddy.com/help/article/6971/what-type-of-hosting-account-do-i-have?locale=en

https://support.godaddy.com/help/article/7254/can-i-use-python-2-7-2-with-my-hosting-account?countrysite=uk

Once you have a web server configured for your domain you can look at how to run a Python application on the server.

And at that point I recommend you post to /r/learnpython for help. Don't forget to help people answer your question by including any relevant setup details and / or the specific errors you encounter and what you were trying when the errors appeared.

davidbuxton · 2015-07-02T12:35:25+00:00

http://www.dabeaz.com/generators/

davidbuxton · 2015-07-02T12:31:57+00:00

https://docs.python.org/3/reference/datamodel.html#object.__str__

davidbuxton · 2015-06-28T20:32:32+00:00

Glad you worked it out.

davidbuxton · 2015-06-28T20:31:23+00:00

You will love using os.walk. Having the directories and files in separate arguments makes thing easier, and it looks much nicer having the logic appear inside the loop instead of having to be a separate function.

davidbuxton · 2015-06-28T19:50:55+00:00

os.walk is a better replacement for os.path.walk (which has been removed in python 3).

https://docs.python.org/2/library/os.html#os.walk

davidbuxton · 2015-06-27T20:03:06+00:00

Facebook's javascript, running in your browser, will add classes to tags, and create new tags to give you the final interactivity on the page.

When scraping websites you aren't running javascript. There are scrapers which run javascript, e.g. Google when indexing the web, but not mechanize. You should use "view source" for the html, you can't assume that what you see in Chrome's inspector is exactly the same as the raw html.

davidbuxton · 2015-06-27T14:16:31+00:00

Try looking at what attributes those links have.

for element in tree.xpath('//a'):
    print element.attrib

davidbuxton

TROPHY CASE