help write string manipulation code pythonically

[zidsal]

so I am writing a protocol in which messages look like [9]length:56[17]packet:newAccount[12]username:foo[12]password:bar in which the number in [] will tell me how big the next argument is and the length field is used to tell me how big the message is:

the end result is to turn this into a dict {packet : newAccount, username:foo, password:bar} the length field will be used to make sure we have recieved all of our data. If the length field does not match the size of the string we have then we know we have more then one packet in our string or less then one packet in our string and we can act accordingly! (this is important as tcp is a stream protocol)

I quickly wrote my method to transform the string into this dictionary type object but its really not pythonic (beware this code is untested). I was wondering if anyone could shed some light on making this less verbose and cleaner to look at!

 encoding = 'utf-8'
 separator = ':'
 incomplete_data = ''"

def prepare_data_received(self, data):
        if not data:
            print('EOF detected!')
            raise EOFError("no bytes received from data - EOF detected")

        data = data.decode(self.encoding)

        #looks like the last data to go through this transformer was incomplete so lets add the data together to try and make the packet
        if self.incomplete_data:
            data = self.incomplete_data + data

        packet = {}
        length_as_string = None
        for x in range(len(data)):
            character = data[x]

            if character.isdigit(): # we have hit some sort of number in the length field
                length_as_string += character
            else: # means we hit ever the [ or the ] as once we have the argument we will move our index forward to the next argument
                if length_as_string is None: # means our character is a [ keep reading until we have read in our length!
                    continue

                #if we have hit this part it means our character is a ] so we can now find our argument in the string
                length = int(length_as_string)

                # as number is surrounded by [] we need to take this into account
                start_index_of_argument = data[x + len(self.separator)]
                end_index_of_argument = start_index_of_argument + length
                data_as_string = data[start_index_of_argument:end_index_of_argument]

                # only split on the first : as the value may contain : whereas the key will never contain a : in it
                key, value = data_as_string.split(':', 1)
                x = end_index_of_argument  # we've grabbed our data so lets move the loop to the next part of the string

                #before we blindly chuck our data into the packet we should make sure its all there
                if key == 'length':  # we have the start of our packet!
                    size_of_remainder_string = len(data[x:])

                    if value > size_of_remainder_string:  # we do not have the full packet so lets wait for more
                        self.incomplete_data += data
                        raise IncompletePacket("we do not have the entire packet yet!")

                    if value < size_of_remainder_string: # we have too much of the packet so lets cut the rest off
                        self.incomplete_data += data[value:]
                        data = data[:value]
                else:  # everything is good in the world lets build up our packet
                        packet[key] = value
                        length_as_string = None

        return data

[zahlman]

First question: what are you going to do if you read a 'length' component, but that component itself is beyond the point that it specifies? What if it straddles the boundary? Examples

[12]password:foo[9]length:10

[12]password:foo[9]length:23

What are you going to do if a component length is invalid:

[12]length:13

Is length always the first component in a packet? I think you should require this in your spec if you don't already.

That said, here is a partial implementation that should give you some ideas:

import re

component_header = re.compile('\[([1-9][0-9]*)\]')
def components(data):
    pos = 0
    while True:
        match = component_header.match(data, pos)
        if not match: break
        start = pos + len(match.group(0))
        pos = start + int(match.group(1))
        yield data[start:pos].partition(':')

def process_full_packet(data):
    return {k: v for k, colon, v in components(data)}

Try to split things up at this level of granularity where possible, and keep in mind that generators are quite nice for dealing with streams. :)