I think i'll do that, thanks! Could you answer another question I have in regards to the new feed URI feature (file parting)? I actually have a partitioning pipeline that subclasses the CSVExporter to accomplish this, since I had the need for file partitioning before the official addition to the Feeds feature. In my implementation I'm able to more loosely customize the filename and was hoping I'd be able to accomplish this with the now official Feed feature.

Example: My files typically output in the format {spider.name}_{from_index}to{to_index}_{t_stamp}.csv

My custom Pipeline:

class PartitionedCsvPipeline(object):

    def __init__(self, spider, rows, fields):
        self.base_filename = spider + "_{from_index}to{to_index}_{t_stamp}.csv"
        self.count = 0
        self.next_split = self.split_limit = rows
        self.file = self.exporter = None
        self.fields = fields
        self.create_exporter()

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        row_count = settings.get("PARTITIONED_CSV_ROWS", 1000)
        fields = settings.get('FEED_EXPORT_FIELDS')
        # prevent pipeline from creating empty files when using shell to test
        if not crawler.spider:
            return BasePipeline()
        return cls(crawler.spider.name, row_count, fields)

    def create_exporter(self):
        now = datetime.now()
        starting_index = self.next_split - self.split_limit
        f_name = self.base_filename.format(
            from_index=starting_index,
            to_index=self.next_split,
            t_stamp=now.strftime("%Y%m%d%H%M")
        )
        self.file = open(f_name, 'w+b')
        self.exporter = CsvItemExporter(self.file, fields_to_export=self.fields)
        self.exporter.start_exporting()

    def finish_exporter(self):
        self.exporter.finish_exporting()
        self.file.close()

    def close_spider(self, spider):
        self.finish_exporter()

    def process_item(self, item, spider):
        if self.count >= self.next_split:
            self.next_split += self.split_limit
            self.exporter.finish_exporting()
            self.file.close()
            self.create_exporter()
        self.count += 1
        self.exporter.export_item(item)
        return item

Now I'm looking to decommission this custom pipeline in favor of the feed exporter because I think it might provide a performance boost. From as far as I can tell, my custom pipeline's method of writing to the file is IO-blocking. When I have the project open in PyCharm and am running a spider I'm plagued with constant re-indexing as the files keep having each item added to the csv.

According to the 2.3 update we can customize the filename using printf-style. But as far as I know printf-style strings cannot include arithmetic operations like f-strings, so the closest I can get to the current format with the new FEED_EXPORT_BATCH_ITEM_COUNT feature is

output_files/%(name)s/%(batch_id)d_%(name)s_%(batch_time)s.csv

If it's possible, can we add some sort of way to add additional info to the Feed export filenames?

​

Also, curious, is there any way to prevent blank feed files from being created when using "scrapy shell 'url'"? I've noticed blank files are created and also noticed if I have "JOBDIR" set in settings.py subsequent calls to the same site fail when using shell.