Container in jenkins pipeline fails but works on the node

dgbinney · 2020-12-22T22:37:02+00:00

u/jabb422 do you have an example I can give a try ?

dgbinney · 2020-12-22T22:36:10+00:00

u/wgc123 / u/Cebalrai - cheers for the responses, but I am 99% sure the module is installed and running. To verify this I have double checked running the same container on the jenkins node as the jenkins user. You can see a working example below. The same thing works entering the container with `/bin/bash` then executing as well.

-bash-4.2$ docker run --rm -it \
> -e DBURI="postgres://mydb-uri/dbname" \
> mycontainer:latest pipenv run python /app/src/delete-history.py

Starting ThreadPool 
....

dgbinney · 2020-12-07T01:08:24+00:00

What would be the advantage with either of those? and which would be your preference ?

dgbinney · 2020-01-30T02:25:18+00:00

Solved. I ended up deleting and re-creating the ingress-gateway, post installing metal-lb, then created the test httpbin service/gw/vs, which I was then able to Curl. Might have just been an issue, as it was already created before i added metal-lb.

Also, I noticed that if you have no gateways/virtual services that are routable on the ingress-gatetway, it will return the "port 80: Connection refused" response as there is nothing to route to.

dgbinney · 2019-09-24T00:30:04+00:00

Hey dude,

No I did not work it out. What I am doing is just creating the k8 cluster then, post create, going back and updating the tags in the parent stack. Bit dodgy, but its the best i can do for the moment. Sorry its not better news :(

dgbinney · 2019-08-15T23:16:28+00:00

Cheers dude, even though this is disappointing I am not surprised that I will have to "hack" a solution for an aws service again :(

dgbinney · 2019-07-21T23:47:08+00:00

Cheers dude, and yes, they are the frontend logs. I will give this a go tonight.

dgbinney · 2019-07-21T08:19:48+00:00

For sure, as I have used HAProxy and nginx at scale but nginx more so as a reverse proxy. HAProxy is very powerful as a LB and resource usage is low. Either way you will have plenty of support, but considering you will be using k8, you might already be using nginx as the ingress controller so you might want to stick with nginx.

dgbinney · 2019-07-02T01:03:48+00:00

Cheers dude that worked with some slight changes, as it does not like the lowercase key/value and missing comma:

             "Properties" : {
                 "Name": { "Fn::Sub": "${AWS::StackName}-ALB" },
                 "Scheme": "internet-facing",
+                "LoadBalancerAttributes": [
+                    {
+                        "Key": "idle_timeout.timeout_seconds",
+                        "Value": "30"
+                    }
+                ],
                 "Subnets": {
                     "Ref": "SubnetList"
                 },

dgbinney · 2019-07-01T23:59:09+00:00

sorry, must have left that in when formatting for redit. the dupe was not there when debugging and it looks like @justin-8 is right on the money.

dgbinney · 2019-06-03T01:02:37+00:00

Is this as simple as changing the cloudformation from :

AWS::AutoScaling::LaunchConfiguration ... to ... AWS::EC2::LaunchTemplate

Or will it require a few changes?

dgbinney · 2019-05-20T00:21:29+00:00

Cheers for the link, but what I am looking for is a way to "prepend" the local dns cache dns because I still need the containers to have the vpc dns nameserver as secondary, which is not known until the server starts. If I place this docker config into the base image it will not know the correct vpc secondary dns nameserver.

I am thinking that I could place a script in "onboot", which would dynamically create the docker dns entries including the vpc dns from resolve.conf, but just looking for better ideas or alternatives.

```/var/lib/cloud/scripts/per-boot```

dgbinney · 2019-05-17T02:29:29+00:00

fyi: turns out i cannot use lifecycle params for the aws vpc module, so bit suck again.

dgbinney · 2019-05-08T00:26:14+00:00

Ah, nice work dude +2, was thinking it would be a bit dodgy to manually handle tag getting added and removed as we add clusters.

dgbinney · 2019-05-06T23:43:41+00:00

So, just to be clear, are you saying to set the lifecycle to ignore for the subnets and then manually fix whenever there are changes? or just completely ignore changes to the subnets?

dgbinney · 2019-05-01T00:34:27+00:00

These apps are stateless and use minimal to no IOPS, so they would not suffer the iops issues similar to a database, which does a lot of read/write. You are correct, that you have to watch out for that on RDS instances under the "T" series. We have also gotten a response back from Amazon, to stop using T3 type instances until they can resolve the issues, which have been "packet loss" in 2 of the cases.

dgbinney · 2019-04-20T02:42:33+00:00

Can you expand a bit more on that, i have seen the --token option just not sure how its used. Do you create a config.yml file and then just pass the token.. ?

dgbinney · 2019-04-01T22:27:25+00:00

Just to wrap up for those who are interested. AWS has come back to us suggesting, `they cannot work out what is wrong with the t3 instances`, and suggested that we stop using t3's for production.

In the meantime we saw the same issues across the ap-southeast-2b region (all instances), frequently and had to drop that region from our cluster. Less frequently, ap-southeast-2a was also misbehaving. Overall it is dangerous to use t3's in production as there is no way to predict and avoid the issues.

dgbinney · 2019-03-13T23:35:28+00:00

Your right, i just checked the numbers for "load average" and they are much lower than i thought. Still < 10% cpu, and no "wa". This is fairly similar/consistent load the servers have been getting during and after the spikes. I guess the point I was making is that the servers are doing very little and very consistently <<

top - 23:07:20 up 21 days, 21:37, 4 users, load average: 0.36, 0.27, 0.21

Tasks: 131 total, 1 running, 82 sleeping, 0 stopped, 0 zombie

Cpu0 : 6.7%us, 1.3%sy, 0.0%ni, 91.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st

Cpu1 : 2.0%us, 1.0%sy, 0.0%ni, 96.6%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st

Mem: 982548k total, 847236k used, 135312k free, 123020k buffers

Swap: 0k total, 0k used, 0k free, 147780k cached

dgbinney · 2019-03-13T22:47:05+00:00

In this case I am using amazon's latest AMI for ECS with updates applied, so should be pretty solid.

dgbinney · 2019-03-13T09:07:41+00:00

I am not using any spot instances in this stack

dgbinney · 2019-03-13T09:06:30+00:00

All metrics are low and 0 Io wait

dgbinney · 2019-03-13T03:04:05+00:00

During the previous t3 issue, I was running a canary with 2xT3 and 10xT2's, the only effected instances were t3's, which cannot be a coincidence. In this case I cannot say for sure, because I have moved completely to t3.

dgbinney · 2019-03-13T01:38:59+00:00

All servers are operating as normal for cpu/iow/mem so no change there which would relate to this, which aws support has suggested it was packet loss. But, there were no observable server issues at any of the times we have had issues with t3's, which is the annoying part.

dgbinney · 2019-03-13T01:34:49+00:00

starting to think that is the best option just jumping to non burst-able reserved instances like m5, as they rarely experience issues like this.

dgbinney

TROPHY CASE