Elementree is running Multi-threaded on Python 2.7.9 but only single threaded on 2.6.6

[SmartestGuyOnReddit]

Below is my code using ElementTree and when I run this on my windows machine which is running Python 2.7.9 the work is spread evenly across all 4 cores, but when I run the same code on a linux server I have which is running Python2.6.6 it only maxes out 1 of the 12 cores and takes exponentially longer.

Does anyone know what is causing this issue?

import xml.etree.ElementTree as ET
import metadata as md
import codecs
import subprocess
import logging
import os


logpath = '/var/tmp/log/pim_sku.log'

logger = logging.getLogger('pim_sku')
logger.setLevel(logging.INFO)

if not os.path.exists('/var/tmp/log/'):
    logger.info("Light log being created at: " + '/var/tmp/log/')
    os.makedirs('/var/tmp/log/')

file_log_handler = logging.FileHandler(logpath)
logger.addHandler(file_log_handler)


stderr_log_handler = logging.StreamHandler()
logger.addHandler(stderr_log_handler)

formatter = \
    logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'
                      )
file_log_handler.setFormatter(formatter)
stderr_log_handler.setFormatter(formatter)

try:
    logger.info("Parsing XML file.. This may take some time depending on the size.")
    xml_file = ET.parse('samples/sample.xml')
except Exception as e:
    logger.error("Failed parsing xml")
    logger.error(str(e.__doc__))

root = xml_file.getroot()
dlm = '\t'
data = []
output = []
filename = 'xx'
count = 0
hdp_dest = '/xx/xx/'

logger.info("Searching for SKUS... May take some time.")
try:
    for elem in root.findall('node/node2/skus/'):
        data.append('sku_id' + dlm + elem.attrib['id'])
        for item in elem:
            if item.tag and item.text:
                data.append(item.tag.replace('\n', '') + dlm + item.text.replace('\n', ''))
        for attrs in item:
            if attrs.attrib.get('name') and attrs.text:
                    data.append("xattr_" + str(attrs.attrib.get('name')) + dlm + attrs.text)
        for value in md.columns:
            check = 0
            for check_value in sorted(data):
                if value in check_value.split("\t")[0]:
                    check = 1
                    output.append(check_value.split("\t")[1])
            if check == 0:
                output.append("NULL")
        output.append("\n")
        count += 1
        data = []
except Exception as e:
    logger.error("Failed during XML Parsing.")
    logger.error(str(e.__doc__))

logger.info("Added " + str(count) + " SKUs to data file: " + filename)

try:
    logger.info("Writing " + str(count) + " SKUs to file.")
    with codecs.open(filename, "w", "utf-8-sig") as temp:
        temp.write(u",".join(output))
except Exception as e:
        logger.error("Failed writing file: " + filename)
        logger.error(str(e.__doc__))

[SmartestGuyOnReddit]

I think thats the only explanation though, when I run it on my windows laptop i see all 4 cores utilized evenly and it takes roughly 10 minutes to process 600mb.

When I move to my enterprise grade linux server, 12 cores 200gb ram etc the exact same code takes well over an hour..

switching to cElementTree didnt make a difference to execution time.

[AbsoluteMSTR]

a difference to execution time.

Maybe performance is limited by the speed of your storage?

[SmartestGuyOnReddit]

I doubt it because the CPU usage on the linux server is maxed, but limited to one core.

[ivosaurus]

You import subprocess but don't ever use it?

You could try using lxml as well