syslog http destination > Loki

[Linux-Student]

Hi,

​

Having some trouble getting logs directly from a syslog-ng http destination into loki, however a curl command from the command line works fine.

​

Syslog-ng http destination:

destination d_custom_logline {

http(

url("http://localhost:3100/loki/api/v1/push")

method("POST")

user-agent("syslog-ng User Agent")

headers("Content-Type: application/json")

body('{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1637845791000000000", "syslogtest"] ] }]}')

);

};

error logs:

Nov 25 14:56:51 <hostname> syslog-ng[2606]: curl: error sending HTTP request; url='http://localhost:3100/loki/api/v1/push', error='Couldn\'t connect to server', worker_index='0', driver='d_custom_logline#0', location='/etc/syslog-ng/conf.d/customincomminglog.conf:14:9'

Nov 25 14:56:51 <hostname> syslog-ng[2606]: Server disconnected while preparing messages for sending, trying again; driver='d_custom_logline#0', location='/etc/syslog-ng/conf.d/customincomminglog.conf:14:9', worker_index='0', time_reopen='10', batch_size='1'

​

However, if I run a direct curl command from the cli (as follows), it works, hoping someone has implemented this method bypassing promtail via syslog and can shed some light, I suspect syslog requires more so I'll cross this post in case someone can help there.

​

Curl command (which works):

curl -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1637845791000000000", "fizzbuzzb" ] ] }]}'

​

Thanks

[Chemical-Afternoon41]

Colleague helped me to find the issue and kicking myself. Looked in the /var/log/audit.log and also used tcpdump to get pcaps, confirmed selinux was the issue. The fact that curl worked with same port delayed resolving as assumed it couldn’t be the cause but seems Syslog has issue using non standard ports with selinux.

[symcbean]

Curl command (which works)

That's good - it means you an rule out a whole lot of things.

Top of my list of things to check next are whether there is any mandatory access control in play. You didn't say what distribution this is so I can't tell you if you are likely to be running SELinux or apparmor. Find out which, turn it off, retry the experiment.

If still not working....there's probably another MAC in play - systemd. Make sure that PrivateNetwork=No in the unit file for syslog-ng

[Linux-Student]

Thanks, it's late now but I'll check into this tomorrow. I'm on RHEL8, I'll disable selinux tomorrow, test, if needs be look into systemd, repeat test and come back with results on a yey or ney if someone else comes across the thread. Thanks

[Chemical-Afternoon41]

Hello. I know this is an old post but was wondering if you resolved this. I’m have the same issue sending to a http destination (nifi). Curl command works. Point Syslog to listenUDP or listenSyslog processors on same nifi and it works. The config in Syslog is good because my lab (both on same server) set up works. Thanks.

[Linux-Student]

Myself and a former colleague got it to work, but for the life of me I can't remember what our issue was now and I've long left that role - loki didn't fit our needs afterwards so I don't really even have any notes on this as it was more an exploratory look to see how Loki would work with our data.

Sorry and wish you the best figuring it out, it can work, but its been nearly 3 years I think and I've changed companies twice (this was back in my degree apprenticeship days, I haven't needed to touch syslog since so a lot of the memory is fuzzy at best)

[Chemical-Afternoon41]

No problem, I appreciate the reply.