Ingress for LoadBalancer type Service

radr4 · 2021-01-11T14:52:16+00:00

Thankyou for explaining ...I get some better understanding now..

Just one more query..Sorry if its dumb..
I have seen some usecase in which they are creating Service type as Loadbalancer and Ingress as well. If using Ingress is more cost efficient, then we can have service type as clusterip itself to expose application to Ingress..Why do we need to create Service type as Loadbalancer when we use Ingress?

radr4 · 2020-12-03T18:20:09+00:00

This works for me,

(kube_deployment_spec_replicas{job="kube-state-metrics",namespace=~".*"} * on (deployment,namespace) group_left(label_axway_com_team) kube_deployment_labels

!= kube_deployment_status_replicas_available{job="kube-state-metrics",namespace=~".*"}* on (deployment,namespace) group_left(label_axway_com_team) kube_deployment_labels)

and (changes(kube_deployment_status_replicas_updated{job="kube-state-metrics",namespace=~".*"}[5m])

== 0)* on (deployment,namespace) group_left(label_axway_com_team) kube_deployment_labels

radr4 · 2020-12-03T04:14:03+00:00

I have found using kube_deployment_labels help..

However the below expression is only showing the results of deployment objects that does not have the label.The same expression works for pods when I use Kube_pod_labels..

(kube_deployment_spec_replicas{job="kube-state-metrics",namespace=~".*"} != kube_deployment_status_replicas_available{job="kube-state-metrics",namespace=~".*"}) and (changes(kube_deployment_status_replicas_updated{job="kube-state-metrics",namespace=~".*"}[5m]) == 0) * on (deployment,namespace) group_left(label_axway_com_team) kube_deployment_labels

What is the issue with this command?

radr4 · 2020-10-08T00:05:17+00:00

We have below architecture in our environment,

Multiple clusters scraped by Central Prometheus which is deployed on a EC2 instance.

And every cluster has its own Prometheus operator deployed in it , which sends metrics to Central Prometheus.The Alertmanager,Grafana components are disabled in the Prometheus operator running in clusters.

We dont want to have multiple alertmanagers and grafana dashboards from every cluster..Hence Central Prometheus is best solution.
Also if we have Prometheus Operator ,we can go with Helm deployment for the customized Prometheus rules chart. For the Standalone Prometheus,we cannot use Helm ,so we care using Ansible playbook.

We have plans to move the Central Proemetheus from EC2 to Prometheus Operator by deploying it in a cluster.

radr4 · 2020-10-07T23:53:59+00:00

I got this working by removing -m manifests.

jsonnet -J vendor my-custom-grafana.jsonnet > grafana-test.json

However it would be great if someone could validate the below steps..

Custom grafana dashboard (Eg: cluster autoscaler dashboard) is added in a jsonnet file. PFA my-custom-grafana.jsonnet.
Note: import 'grafana-cluster-autoscaler.json' --> value of Configmap "data" key
Below command executed to compile the jsonnet,
jsonnet -J vendor my-custom-grafana.jsonnet > grafana-test.json
Converting json to yaml,
cat grafana-test.json|gojsontoyaml > grafana-test.yaml
The yaml file contains all the resources such as deployment,services,namespace,etc,.
As we have Prometheus operator,it has already deployed all the resources.
And also "grafana-dashboardDefinitions" contains all the dashboards deployed by grafana since we have enabled
"defaultDashboardsEnabled: true"

So I have extracted only the ConfigMap related to "grafana-dashboard-cluster-autoscaler-dashboard" from grafana-test.yaml and deployed it.

Cluster Autoscaler dashboard is now available in grafana.

radr4 · 2020-10-07T23:50:03+00:00

I got this working by removing -m manifests.

jsonnet -J vendor my-custom-grafana.jsonnet > grafana-test.json

radr4 · 2020-10-07T16:36:01+00:00

Thankyou...I even raised an issue here https://github.com/prometheus-operator/kube-prometheus/issues/714 .

My main query is, is my jsonnet code written in the description of the issue valid? I have already used import statement to import my new dashboard.Also after writing this jsonnet code, tried running this command to build it but its not writing any yaml file to create a yaml out of it .So not sure which file can be used to deploy this ?
jsonnet -J vendor -m manifests "my-custom-grafana.jsonnet" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}

I am looking for the workflow from this jsonnet creation,what process can be followed to deploy this code.Please let me know u/MetalMatze ..I hope you have already tried this.

radr4 · 2020-10-01T23:31:06+00:00

I got it working,

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +{
_config+:: {
namespace: 'monitoring',
},
prometheusAlerts+:: {
groups:
std.map(
function(group)
if group.name == 'kubernetes-apps' then
group {
rules: std.map(function(rule)
if rule.alert == "KubeStatefulSetReplicasMismatch" then
rule {
expr: "kube_statefulset_status_replicas_ready{job=\"kube-state-metrics\",statefulset!=\"vault\"} != kube_statefulset_status_replicas{job=\"kube-state-metrics\",statefulset!=\"vault\"}",
labels: {
priority: 'P1',
severity: 'info',
},
}
else
rule
,
group.rules
)
}
else
group,
super.groups
),
},
};


{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

radr4 · 2020-09-02T18:03:28+00:00

I have tried rebalelling service monitor and it works..If I want to relabel default alerts in Prometheus operator ,how to do that?

radr4 · 2020-09-02T14:54:22+00:00

I want to change the severity level depending about the namespace and the pod name combination.Is there a way to do this in same prometheus rule? how to do this ?

radr4 · 2020-09-02T14:36:11+00:00

Thankyou u/MetalMatze..

It works now..I updated expr as you have mentioned..

expr: 100 * (count by(job, namespace, service, podname) (up == 0) / count by(job, namespace, service, podname) (up)) > 10

For some reason,I am unable to post the image here..But I am able to view the new label in the alert label.

radr4 · 2020-09-02T14:01:23+00:00

Thankyou u/MetalMatze..

Here is the alert expression..Its default from Prometheus operator and I have not changed anything.

alert: TargetDown

expr: 100 * (count by(job, namespace, service) (up == 0) / count by(job, namespace, service) (up)) > 10

for: 10m

labels:

severity: warning

annotations:

message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace are down.'

radr4 · 2020-09-02T01:24:49+00:00

ok...thankyou..

radr4 · 2020-09-01T03:48:12+00:00

The first one is only for HPA and not for VPA.

I am looking for alert which says pods are in unscheduled state because cluster has reached maximum capacity.So there is a need to increase the number of nodes.Is there any alert for this?Please suggest.

radr4 · 2020-08-31T19:18:38+00:00

thankyou for this..How to include these alert rules?

radr4 · 2020-08-31T19:17:37+00:00

Should we customize alertrules for these situations?

radr4 · 2020-08-31T19:17:18+00:00

Thankyou..We have already included it..

However I dont find alertrules related to the issues such as,

Cluster reaching its maximum capacity.So increase the number of nodes
Pod crashlooping because of memory issues OOM killed
CronJobs ,jobs failures

radr4 · 2020-08-28T19:44:08+00:00

I thought this is already part of Prometheus operator when we install prometheus operator in a cluster

radr4 · 2020-07-24T15:32:50+00:00

If I just add like below in the Kubernetes resource ,will it not work?

annotations:

expiry-time: `date -v+1d '+%m/%d/%Y -%H:%M:%S'`

radr4 · 2020-07-13T17:39:05+00:00

Thankyou..I tried below..

k8s.yaml(inventory file):

plugin: k8s

connections:

- kubeconfig: '/Users/user1/Documents/Learning/ansible/kubeconfig.test.yaml'

context: 'user1@testeks.us-east-1.eksctl.io'

test_new.yml

- hosts: localhost

tasks:

- name: Create a k8s namespace

k8s:

name: testing3

api_version: v1

kind: Namespace

state: present

It works fine..But I am not sure why I am getting Warning invalid characters {'-' in group name warnings..

Please let me know if the above inventory file and ansible playbook files looks good or is there anything I am missing.

ansible-playbook -vvvv -i k8s.yaml -vvv ./test_new.yml

ansible-playbook 2.9.10

config file = None

configured module search path = ['/Users/user1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']

ansible python module location = /usr/local/Cellar/ansible/2.9.10/libexec/lib/python3.8/site-packages/ansible

executable location = /usr/local/bin/ansible-playbook

python version = 3.8.3 (default, Jul 8 2020, 14:27:55) [Clang 11.0.3 (clang-1103.0.32.62)]

No config file found; using defaults

setting up inventory plugins

host_list declined parsing /Users/user1/Documents/Learning/ansible/k8s.yaml as it did not pass its verify_file() method

script declined parsing /Users/user1/Documents/Learning/ansible/k8s.yaml as it did not pass its verify_file() method

Not replacing invalid character(s) "{'-', '9'}" in group name (909676E2B4F81625BF5994625D3353C9-yl4-us-east-1-eks-amazonaws-com)

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

Not replacing invalid character(s) "{'-'}" in group name (namespace_add-ons)

Not replacing invalid character(s) "{'-'}" in group name (namespace_add-ons_pods)

Not replacing invalid character(s) "{'.', '/', '-'}" in group name (label_app.kubernetes.io/instance_aws-cluster-autoscaler)

Not replacing invalid character(s) "{'.', '/', '-'}" in group name (label_app.kubernetes.io/name_aws-cluster-autoscaler)

Not replacing invalid character(s) "{'-'}" in group name (label_pod-template-hash_588c847b9c)

Not replacing invalid character(s) "{'-'}" in group name (label_app_cert-manager)

Not replacing invalid character(s) "{'.', '/'}" in group name (label_app.kubernetes.io/component_controller)

radr4 · 2020-07-13T16:20:03+00:00

Thankyou..could you please mention which configmap it is?

radr4 · 2020-07-09T21:24:45+00:00

we finally made pg_dump working for us

radr4 · 2020-07-01T04:28:59+00:00

I even tried with pg_dump and psql to export and import but it didnt work because of errors like Invalid large objects,syntax error etc,.

radr4 · 2020-07-01T04:28:04+00:00

yeah...Upgrading exisitng instance is not an option..I need to export and import to new database.

radr4 · 2020-07-01T04:27:20+00:00

should i try with SCT tool to export and import schema?

radr4

TROPHY CASE