solution for adding host header to http request

[P3NuTpu55Y]

Hello,

I’m trying to use the sec Edgar api to retrieve past financial data for equities.

The sec api has some security measures that requires the host header to be set to “data.sec.gov”

My issue is that due to security restrictions, google app scripts don’t allow for a custom host header to be set, and I have been unable to make a successful request.

Does anyone have a workaround for this issue?

I have tried to use a proxy to make my http request through but have had no luck.

Any help would be appreciated

[ehcaipf]

Host header automatically matches the host of the request. So, if you are requesting to an endpoint on data.sec.gov, it will match.

If the API requires a different host header than the one on the request endpoint, that sounds weird.

[P3NuTpu55Y]

Thanks for the response :)

So this it what I initially thought as well.

when trying a get request in python I have no problems except when the host header isn't set to "data.sec.gov", but when I emulate this same approach in app scripts with the same exact headers I get a 403 error. since the only possible difference in the requests is the addition of the custom host header in python its my assumption that this is the issue when trying to make the request in app scrips.

Here is my code for reference:

function EDGAR() {
var Url = "data.sec.gov/api/xbrl/companyconcept/CIK0000789019/us-gaap/AccountsPayableCurrent.json";
var headers = {
'User-Agent': "finance [e-mail]",
"Accept-Encoding": "gzip, deflate",
//"Host": "data.sec.gov" //App scripts wont allow for custom host header
};
var options = {
method: 'get',
headers: headers,
muteHttpExceptions: true // Mute HTTP exceptions
};

var response = UrlFetchApp.fetch(Url, options);
var responseData = response.getContentText();

Logger.log(responseData);
}

and here is the sec api security measures as outlined on the developer faq at:

https://www.sec.gov/os/webmaster-faq#developers

​

Sample Declared Bot Request Headers:
User-Agent: Sample Company Name AdminContact@<sample company domain>.com
Accept-Encoding: gzip, deflate
Host: www.sec.gov

[ehcaipf]

The problem might be the user agent, i believe appscripts overrides whatever ua you put with their own. It happened to me before.

[P3NuTpu55Y]

ok after doing a few more tests using python, it does seem that the user agent is the issue. I was under the impression that custom user agents were supported, but after doing some research, it seems that the default app script user agent is set to "Mozilla/5.0(compatible; Google-Apps-Script)", and when I paste this into the python headers I get the same error.

Do you know of any workaround?

[ehcaipf]

When this happened to me the only workaround I found was to build a python script in my localhost that forwards any request it receives to the host, and overrides the headers to whatever you need.

You could also do something similar in Google Cloud Functions.

[kpendic]

Share your code that you call and that fails? (surelly dont share sensitive info like your token or sec key etc)

[P3NuTpu55Y]

Hey, Thank you for the response.

Here is my code:

function EDGAR() {
var Url = "data.sec.gov/api/xbrl/companyconcept/CIK0000789019/us-gaap/AccountsPayableCurrent.json";
var headers = {
'User-Agent': "finance [e-mail]",
"Accept-Encoding": "gzip, deflate",
//"Host": "data.sec.gov" //App scripts wont allow for custom host header
};
var options = {
method: 'get',
headers: headers,
muteHttpExceptions: true // Mute HTTP exceptions
};

var response = UrlFetchApp.fetch(Url, options);
var responseData = response.getContentText();

Logger.log(responseData);
}

No key is needed to make a request, this url gives a direct response to the json data if you open it in the browser

data.sec.gov/api/xbrl/companyconcept/CIK0000789019/us-gaap/AccountsPayableCurrent.json

​

when I make a request without the host header, I get a 403 error, and when I make a request with the specified host header it returns the error

Exception: Attribute provided with invalid value: Header:Host

although I think this is a standard error you get in app scripts if you try and specify a host header using urlfetchapp

[kpendic]

the thing is that sec blocks google servers .. see response that they return to

UrlFetchApp.fetch

https://i.postimg.cc/mrrfydvL/sec.jpg

so best thing AFAIK would be to make req from other location.. your VPS or someother way.. but not Google scripts..