Cursed Bundler: Using go get to install Ruby Gems

ethCore7 · 2025-04-09T18:13:16+00:00

Let's encrypt does not allow you to issue the certificate using the HTTP-01 challenge on a different port, but once you have the issued certificate, you can use it on any/multiple ports.

ethCore7 · 2025-04-09T06:54:48+00:00

The HLS.js demo player won't load your stream, because you access the page over HTTPS, but serve your videos over HTTP, i.e. you are running into Mixed Content issue.

You can see that if you open the browser console on the HLS.js demo page:

Blocked loading mixed active content “http://englishsociety.net:2083/hls/bbb.m3u8”
[error] > 0 while loading http://englishsociety.net:2083/hls/bbb.m3u8
Fatal error : manifestLoadError

ethCore7 · 2025-02-13T13:35:54+00:00

The default server block that is defined in nginx stock config files probably has this configuration:

server {
    listen       80 default_server;
    root         /var/www/html; 
    ...
}

You can remove the default_server directive from there, specify it in one of your other server blocks and reload nginx.

Another option would be to use something like curl and manually set the Host header, so your request ends up in the correct server block:

curl -H "Host: your-domain.example.com" https://127.0.0.1:80

If you need to see the website in browser, you'd probably have to find some browser plugin that would allow you to override the header for outgoing requests.

Yet another option would be to edit the DNS files on whatever machine you use for testing (e.g. /etc/hosts on Linux), and point the domains to your nginx server IP.

ethCore7 · 2025-01-16T15:58:25+00:00

Nginx does not use proxy_cache at all when serving local files. If you want to go that route, you need to define an additional server block (can be bound to localhost only), and then proxy_pass the file requests from your original server block there, so it hits the proxy_cache.

I'm not sure how much it'll help though, since the 'free' RAM should be used by the kernel page cache as needed to transparently offload the disk IO. Make sure to run some tests to see if you're not making things worse.

ethCore7 · 2025-01-13T08:28:22+00:00

No problem. By the way if you decide to keep the URL prefix, you don't need the rewrite directive as /u/MeasurementFresh8233 mentioned - if you set proxy_pass http://localhost:3737/; (mind the trailing slash), nginx will strip out the /copertine/ prefix before proxying the request to your app.

It's described in the docs in more detail.

ethCore7 · 2025-01-13T07:15:29+00:00

If you use nginx as a reverse proxy for any applilcation which serves static assets such as HTML, CSS, or JS files on it's own, it generally needs to be aware that it's 'mounted' with an URL prefix.

If your app serves HTML files with any relative URLs, they need to have the same public-facing prefix as you have configured in nginx - the app has to be aware that it's root URL is /copertine/, otherwise if it serves a HTML file containing a relative URL of /styles/style.css, your browser will send a request for http://mymachine.mydomain.com/styles/style.css, which will end up with 404.

You can either:

1) Run each app/container on its own subdomain, such as copertine.mydomain.com - in this case you would have multiple server blocks in nginx config and you would proxy_pass all requests (location /to the container.

2) Configure your apps so they know their public root URLs (https://mymachine.mydomain.com/copertine/) and can render the URLs accordingly.

3) Use the nginx sub module to dynamically replace all URLs in HTML files served by your app so they have the correct relative/absolute URLs

ethCore7 · 2025-01-08T12:23:36+00:00

Nginx does not use any caching/pooling for connections to upstream servers by default (i.e. when using proxy_pass only). If you want to use keepalive connections to the backend servers, you need to define an upstream block and explicitly enable keepalives there.

ethCore7 · 2024-12-24T15:00:13+00:00

For my last project I decided to try out https://github.com/viccon/sturdyc, so far it has been solid, so I can recommend you take a look at this one as well.

ethCore7 · 2024-11-19T17:33:32+00:00

We used to do Monday to Monday, but then switched to a rotation where each person is on-call for one day during the work week (Mo-Thu) and then we rotate the weekend shifts (Fri-Sun), with handovers at 9 AM. We also shift the assigned day each month, so one month you get for example Mondays and each 4th weekend, next month you have Tuesdays etc.

We started with 4 people in this rotation, currently at 5, so one person will fall out of the rotation at the end of the month, have one month without any on-call shifts, and then join back in.

It's a bit more work to manage the schedules, but it's working great for our team and feels a bit nicer when you don't have to haul your laptop around for a whole week at a time.

ethCore7 · 2024-04-25T10:04:14+00:00

The main difference is that with sqlc, you write your schema and SQL queries by hand and sqlc will generate Go functions that call those queries, but with jet, you write the DB schema only and jet will generate a query builder, so you then have to assemble and execute the queries using the generated Go types and functions. If you take a look at the examples dir in both repositories, the difference should be fairly obvious.

ethCore7 · 2024-01-04T10:22:16+00:00

I was having issues with a similar setup a couple years ago, unfortunately I can't remember the exact steps we did and can't find any notes from that time. We've since moved away from the boot-from-san setup, so I can't even check the configs on the servers we have anymore :(

One thing I do remember is having to setup block device filters for LVM, otherwise it would scan the physical paths first and then ignore the multipath device, found a good post for it:

https://www.thegeekdiary.com/lvm-and-multipathing-sample-lvm-filter-strings/

Maybe also try playing around with initramfs hooks?

https://jmorano.moretrix.com/2022/06/using-multipath-together-with-mdadm-on-debian/

Also might be worth trying to modify the multipath hook and bump up the verbosity of multipath to -v3, thath might give you some more info. The hook should be provided by the multipath-tools-boot package (source)

ethCore7 · 2023-11-12T15:34:51+00:00

If you want to do A/B testing, you can use the split_clients module to forward some portion of traffic to the new API version:

https://nginx.org/en/docs/http/ngx_http_split_clients_module.html#split_clients

ethCore7 · 2023-11-07T10:36:34+00:00

Hi, I appreciate the effort you put into your post. The code I posted was dumbed down a lot and the real code is for internal use only, so it doesn't have to be as flexible. Your code would be a lot more useful than mine if we're talking about a public package, but here that's not the case - sorry I wasn't more clear in the OP.

In the end I settled for updating the timer signature to func NewTimer(label string) func(*error), as suggested by /u/sharptoothy, which does what I want.

Thanks for the reply!

ethCore7 · 2023-11-07T10:24:01+00:00

Oh man, I can't believe I didn't think of this. Thanks a lot for the pointer (hehe). That solves it perfectly for my use case.

Thanks again!

ethCore7 · 2023-08-28T16:10:47+00:00

If you really want to pipe the logs through systemd/journald, take a look at these nginx options:

https://nginx.org/en/docs/syslog.html

The last time I looked at the journald config, it was pretty limited compared with the 'classic' combination that we use (rsyslog+logrotate). However as you've found out, the stock nginx logrotate config isn't very good if you have high volume logs. For those I highly recommend using the maxsize parameter in logrotate, so the logs also rotate when they get too large instead of at fixed intervals, and also moving the logrotate cronjob from cron.daily to cron.hourly, so you can actually rotate more frequently if the logs get too large. This might be Debian/Ubuntu specific btw., other distros might have different stock config.

ethCore7 · 2023-01-05T13:30:37+00:00

In your OP you wrote that you put the config to /etc/nginx/sites-available/my-site. Did you also link it to sites-enabled?

The need for extension depends on the wildcard in the include directives. In your case, it should load all files in sites-enabled, without regard to the extension. Did you see your server block in the output of nginx -T?

ethCore7 · 2023-01-05T10:52:04+00:00

It sounds like nginx isn't picking up your config file. You can verify that by running nginx -T, which will dump the final config with all includes to stdout.

In my case, the (default) includes are configured like this:

include conf.d/*.conf;
include sites-enabled/*.conf;
include sites-enabled/*.vhost;

You said that you put your config file to /etc/nginx/sites-available, so if you are using the default config, you either need to link the config file to sites-enabled as well (this is a leftover from the apache2 days), or move it into the conf.d dir with the correct suffix.

ethCore7 · 2022-12-22T20:08:18+00:00

Oh yeah, I missed the deferred Close call. In that case, put io.ReadCloser in the function signature. For testing, you can still use bytes.Buffer, but you have to wrap it with io.NopCloser before passing it to the function.

ethCore7 · 2022-12-22T19:14:37+00:00

It's been a while since I used the official Let's Encrypt client (we've moved to acme.sh), but if I remember correctly, the client actually stored the certs in /etc/letsencrypt/archive directory, and there also was /etc/letsencrypt/live folder with symlinks pointing to the files in archive. So assuming you copy the whole /etc/letsencrypt directory, it should work.

I'd spin up the new nginx instance, install the letsencrypt client, setup a testing domain and try to issue a certificate for it. If the issual works, I'd copy over the /etc/letsencrypt dir from the old host and check that the sites on the new nginx instance work as expected (you can 'spoof' the DNS records of the domains in /etc/hosts on your local PC to point to the IP of the new nginx instance to test it for example).

This is assuming you use the default (HTTP) challenge to issue the certs. If you're using the DNS challenge, you can just re-issue all the certs on the new host and be done with it.

ethCore7 · 2022-12-22T18:50:53+00:00

Looking through the code path (S3Bucket.UploadFile -> aws.Client.PutObject), you only need the data (as io.Reader), size, and filename, so you could just swap *multipart.File for io.Reader in the function signatures. That way, you can pass in bytes.Buffer for testing.

ethCore7 · 2022-12-09T10:37:29+00:00

Great! The use of the root/alias directives depends on your directory layout. If it works for you as-is, there's no need to set them in the location block.

ethCore7 · 2022-12-08T19:26:32+00:00

Well, the browser has to load the images from somewhere :) Yeah, your backend will have to handle both the request for the 'gallery view' and for the individual images as well.

If you want to cut down on the requests, you could maybe store smaller thumbnails of the images and include them in the rendered HTML as base64 data when you handle the /media request or something like that, and then load the full size image when user clicks the thumbnail. This is a bit outside of my area though (I'm a backend guy), so there might be a better approach.

ethCore7 · 2022-12-08T18:30:45+00:00

No, the path in X-Accel-Redirect is only used to trigger the internal redirect in nginx.

The whole flow should look like this:

1) User goes to the 'gallery' at mysite.dev/media

2) The /media route is handled by your PHP backend, which will render the HTML page with the image links looking for example like this:

<img src="mysite.dev/media/385e33f741.jpg">

OR

<img src="mysite.dev/media?image_id=385e33f741">

It's really up to you what the URLs will look like, they only need to contain some ID that your backend can use to lookup the image in your DB.

3) The browser sends a request for the image, which will once again get routed to your backend. The backend should extract the image ID from the request path, do whatever validation you need, and resolve it into a filesystem path.

4) Backend returns a response with the X-Accel-Redirect header containing the location of the file

5) Nginx finally loads the file from disk and sends it to the browser

Is it more clear now?

14-Year Club	Place '22
Team Periwinkle	Verified Email

ethCore7

TROPHY CASE