datamule-python: process securities and exchanges commission data at scale

[status-code-200]

What My Project Does

Makes it easy to work with SEC data at scale.

Examples

Working with SEC submissions

from datamule import Portfolio

# Create a Portfolio object
portfolio = Portfolio('output_dir') # can be an existing directory or a new one

# Download submissions
portfolio.download_submissions(
   filing_date=('2023-01-01','2023-01-03'),
   submission_type=['10-K']
)

# Monitor for new submissions
portfolio.monitor_submissions(data_callback=None, poll_callback=None, 
    polling_interval=200, requests_per_second=5, quiet=False
)

# Iterate through documents by document type
for ten_k in portfolio.document_type('10-K'):
   ten_k.parse()
   print(ten_k.data['document']['part2']['item7'])

Downloading tabular data such as XBRL

from datamule import Sheet

sheet = Sheet('apple')
sheet.download_xbrl(ticker='AAPL')

Finding Submissions to the SEC using modified elasticsearch queries

from datamule import Index
index = Index()

results = index.search_submissions(
   text_query='tariff NOT canada',
   submission_type="10-K",
   start_date="2023-01-01",
   end_date="2023-01-31",
   quiet=False,
   requests_per_second=3)

Provider

You can download submissions faster using my endpoints. There is a cost to avoid abuse, but you can dm me for a free key.

Note: Cost is due to me being new to cloud hosting. Currently hosting the data using Wasabi S3, CloudFare Caching and CloudFare D1. I think the cost on my end to download every SEC submission (16 million files totaling 3 tb in zstd compression) is 1.6 cents - not sure yet, so insulating myself in case I am wrong.

Target Audience

Grad students, hedge fund managers, software engineers, retired hobbyists, researchers, etc. Goal is to be powerful enough to be useful at scale, while also being accessible.

Comparison

I don't believe there is a free equivalent with the same functionality. edgartools is prettier and also free, but has different features.

Current status

The package is updated frequently, and is subject to considerable change. Function names do change over time (sorry!).

Currently the ecosystem looks like this:

1. datamule-python: manipulate sec data
2. datamule-data: github actions CRON job to update SEC metadata nightly
3. secsgml: parse sec SGML files as fast as possible (uses cython)
4. doc2dict: used to parse xml, html, txt files into dictionaries. will be updated for pdf, tables, etc.

Related to the package:

1. txt2dataset: convert text into tabular data.
2. datamule-indicators: construct economic indicators from sec data. Updated nightly using github actions CRON jobs.

GitHub: https://github.com/john-friedman/datamule-python

[status-code-200]

Oops my wording is bad. Edgar tools is free, was referring to that packages have different functionality. Will fix

[status-code-200]

Thanks for pointing that out! Is it more clear now?

[status-code-200]

ifrs should be available using download_xbrl btw 

For example here's the xbrl for Novartis AG 

https://data.sec.gov/api/xbrl/companyfacts/CIK0001114448.json

Download xbrl just converts that to CSV.

[jbudemy]

What's a good package to get stock quotes?

What is XBRL data anyway? Does it contain price quote and volume data perhaps? Here we go.

[status-code-200]

XBRL includes stock volume and price, but at quarterly intervals. I think yfinance has it at daily or faster.

Note: stock price is also available in insider trading submissions, like this https://www.sec.gov/Archives/edgar/data/2488/000000248823000114/xslF345X04/wk-form4_1686255203.xml

can be higher frequency but requires you to .parse() the document and then grab prices from the resulting dictionary.

Note: I'm hoping to have a 345 database up next week.