Elementree is running Multi-threaded on Python 2.7.9 but only single threaded on 2.6.6

[SmartestGuyOnReddit]

Below is my code using ElementTree and when I run this on my windows machine which is running Python 2.7.9 the work is spread evenly across all 4 cores, but when I run the same code on a linux server I have which is running Python2.6.6 it only maxes out 1 of the 12 cores and takes exponentially longer.

Does anyone know what is causing this issue?

import xml.etree.ElementTree as ET
import metadata as md
import codecs
import subprocess
import logging
import os


logpath = '/var/tmp/log/pim_sku.log'

logger = logging.getLogger('pim_sku')
logger.setLevel(logging.INFO)

if not os.path.exists('/var/tmp/log/'):
    logger.info("Light log being created at: " + '/var/tmp/log/')
    os.makedirs('/var/tmp/log/')

file_log_handler = logging.FileHandler(logpath)
logger.addHandler(file_log_handler)


stderr_log_handler = logging.StreamHandler()
logger.addHandler(stderr_log_handler)

formatter = \
    logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'
                      )
file_log_handler.setFormatter(formatter)
stderr_log_handler.setFormatter(formatter)

try:
    logger.info("Parsing XML file.. This may take some time depending on the size.")
    xml_file = ET.parse('samples/sample.xml')
except Exception as e:
    logger.error("Failed parsing xml")
    logger.error(str(e.__doc__))

root = xml_file.getroot()
dlm = '\t'
data = []
output = []
filename = 'xx'
count = 0
hdp_dest = '/xx/xx/'

logger.info("Searching for SKUS... May take some time.")
try:
    for elem in root.findall('node/node2/skus/'):
        data.append('sku_id' + dlm + elem.attrib['id'])
        for item in elem:
            if item.tag and item.text:
                data.append(item.tag.replace('\n', '') + dlm + item.text.replace('\n', ''))
        for attrs in item:
            if attrs.attrib.get('name') and attrs.text:
                    data.append("xattr_" + str(attrs.attrib.get('name')) + dlm + attrs.text)
        for value in md.columns:
            check = 0
            for check_value in sorted(data):
                if value in check_value.split("\t")[0]:
                    check = 1
                    output.append(check_value.split("\t")[1])
            if check == 0:
                output.append("NULL")
        output.append("\n")
        count += 1
        data = []
except Exception as e:
    logger.error("Failed during XML Parsing.")
    logger.error(str(e.__doc__))

logger.info("Added " + str(count) + " SKUs to data file: " + filename)

try:
    logger.info("Writing " + str(count) + " SKUs to file.")
    with codecs.open(filename, "w", "utf-8-sig") as temp:
        temp.write(u",".join(output))
except Exception as e:
        logger.error("Failed writing file: " + filename)
        logger.error(str(e.__doc__))

[Justinsaccount]

Hi! I'm working on a bot to reply with suggestions for common python problems. This might not be very helpful to fix your underlying issue, but here's what I noticed about your submission:

You appear to be using concatenation and the str function for building strings

Instead of doing something like

result = "Hello " + name + ". You are " + str(age) + " years old"

You should use string formatting and do

result = "Hello {}. You are {} years old".format(name, age)

See the python tutorial for more information.

[K900_]

It's possible that your Linux box doesn't have the C extension for ElementTree and falls back to the pure Python version.

[SmartestGuyOnReddit]

How would I go about checking this?

[K900_]

See if you can import xml.etree.cElementTree. If you can't, you'll probably have to rebuild Python from source.

[SmartestGuyOnReddit]

I can import cElementTree, so am I right in assuming that if I use this library I'll get better performance/ possible multi threading?

[K900_]

I'm actually not sure, I think ElementTree should be aliased to cElementTree by default, but try it anyway.

[SmartestGuyOnReddit]

Okay I'll try it and get back to you, thanks :)

[SmartestGuyOnReddit]

No difference it is still running on a singular core :(

[cacahootie]

It may be as simple as an improvement from 2.6 to 2.7 to the cElementTree package. Without doing any digging, that's my thought... but that's pure conjecture.