| *** kranthikirang has quit IRC | 00:04 | |
| *** sdake has quit IRC | 00:19 | |
| *** sdake has joined #openstack-helm | 00:27 | |
| jayahn | evrardjp: portdirect: sorry for not being there at irc meeting. Over the past week, it has been crazy busy here, in addition, I got sick recently (I am not saying it was due to heavy drinking on last friday) | 00:30 |
|---|---|---|
| jayahn | I am reading through meeting logs. | 00:30 |
| *** kranthikirang has joined #openstack-helm | 00:36 | |
| openstackgerrit | Merged openstack/openstack-helm master: Keystone: remove stale pids on startup if present https://review.openstack.org/633842 | 00:38 |
| *** kranthikirang has quit IRC | 00:40 | |
| openstackgerrit | Arun Kant proposed openstack/openstack-helm master: Updating keystone-api to work with opensuse based image https://review.openstack.org/633847 | 00:52 |
| openstackgerrit | Arun Kant proposed openstack/openstack-helm master: Updating keystone-api to work with opensuse based image https://review.openstack.org/633847 | 00:54 |
| *** sthussey has quit IRC | 01:01 | |
| *** sdake has quit IRC | 01:19 | |
| *** sdake has joined #openstack-helm | 01:23 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add pre-fixes to the Selenium jobs and remove "|| true" https://review.openstack.org/633826 | 01:49 |
| *** jamesgu_ has quit IRC | 01:52 | |
| *** jamesgu has quit IRC | 01:52 | |
| *** cfriesen has quit IRC | 01:59 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: HTK: Support tls secrets on non-fqdn overridden hosts in ingress https://review.openstack.org/633836 | 02:06 |
| *** kranthikirang has joined #openstack-helm | 02:24 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add pre-fixes to the Selenium jobs and remove "|| true" https://review.openstack.org/633826 | 02:25 |
| *** kranthikirang has quit IRC | 02:28 | |
| *** sdake has quit IRC | 03:00 | |
| openstackgerrit | Merged openstack/openstack-helm master: Fix wrong chart name and env-variable's name in deployment-script https://review.openstack.org/633440 | 03:09 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [ingress] explicitly specify the Prometheus scrape port https://review.openstack.org/633651 | 04:12 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [Prometheus] Fix filesystem space checks https://review.openstack.org/633659 | 04:12 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [Prometheus] Relax disk IO constraints https://review.openstack.org/633619 | 04:12 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [tiller] Disable monitoring by default, enable in gate https://review.openstack.org/633624 | 04:12 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Fluentd: Remove unused liveness port https://review.openstack.org/633832 | 04:12 |
| *** kranthikirang has joined #openstack-helm | 04:12 | |
| *** jamesgu_ has joined #openstack-helm | 04:14 | |
| *** jamesgu has joined #openstack-helm | 04:14 | |
| *** kranthikirang has quit IRC | 04:17 | |
| *** cfriesen has joined #openstack-helm | 04:41 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: HTK: Add support for tls secrets on cluster hosts https://review.openstack.org/633868 | 05:09 |
| *** kranthikirang has joined #openstack-helm | 06:01 | |
| *** kranthikirang has quit IRC | 06:05 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-infra master: [Calico] Configuration robustness improvements https://review.openstack.org/625325 | 06:33 |
| *** aojea has joined #openstack-helm | 07:12 | |
| *** nmimi has joined #openstack-helm | 07:27 | |
| *** jamesgu_ has quit IRC | 07:32 | |
| *** jamesgu has quit IRC | 07:32 | |
| *** kranthikirang has joined #openstack-helm | 07:49 | |
| *** gkadam has joined #openstack-helm | 07:51 | |
| *** kranthikirang has quit IRC | 07:54 | |
| *** georgk has joined #openstack-helm | 08:14 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-infra master: [Prometheus] Tweak K8SApiServerLatency to ignore DELETECOLLECTION https://review.openstack.org/633888 | 08:41 |
| *** gkadam has quit IRC | 08:42 | |
| *** roman_g has joined #openstack-helm | 08:43 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-infra master: [alertmanager] default to 1 replica, multinode gate uses 3 https://review.openstack.org/633889 | 08:44 |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-infra master: HTK: Add support for tls secrets on cluster hosts https://review.openstack.org/633868 | 08:45 |
| *** georgk has quit IRC | 08:46 | |
| openstackgerrit | Deokjin Kim proposed openstack/openstack-helm-infra master: Fix calling wrong variable name in gnocchi https://review.openstack.org/633891 | 08:56 |
| *** georgk has joined #openstack-helm | 08:59 | |
| *** georgk has quit IRC | 09:08 | |
| *** roman_g has quit IRC | 09:31 | |
| *** cfriesen has quit IRC | 09:37 | |
| *** kranthikirang has joined #openstack-helm | 09:37 | |
| *** kranthikirang has quit IRC | 09:41 | |
| *** georgk has joined #openstack-helm | 09:43 | |
| *** georgk has quit IRC | 10:05 | |
| *** georgk has joined #openstack-helm | 10:39 | |
| *** georgk has quit IRC | 10:39 | |
| *** georgk has joined #openstack-helm | 10:56 | |
| *** georgk has quit IRC | 10:58 | |
| *** georgk has joined #openstack-helm | 11:01 | |
| *** kranthikirang has joined #openstack-helm | 11:18 | |
| *** kranthikirang has quit IRC | 11:22 | |
| *** kranthikirang has joined #openstack-helm | 11:29 | |
| *** georgk has quit IRC | 11:32 | |
| *** kranthikirang has quit IRC | 11:33 | |
| *** georgk has joined #openstack-helm | 11:50 | |
| *** kranthikirang has joined #openstack-helm | 11:55 | |
| *** kranthikirang has quit IRC | 11:59 | |
| *** sdake has joined #openstack-helm | 12:02 | |
| *** gkadam has joined #openstack-helm | 12:08 | |
| *** gkadam is now known as gkadam-bmgr | 12:19 | |
| *** georgk has quit IRC | 12:24 | |
| *** georgk has joined #openstack-helm | 12:25 | |
| *** georgk has quit IRC | 12:26 | |
| evrardjp | there are a few [openstack-helm] topics on the ML | 12:27 |
| *** sdake has quit IRC | 12:43 | |
| *** sdake has joined #openstack-helm | 12:45 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [CEPH] Uplift docker images to latest Mimic release https://review.openstack.org/633577 | 12:47 |
| *** georgk has joined #openstack-helm | 13:03 | |
| *** kranthikirang has joined #openstack-helm | 13:43 | |
| openstackgerrit | Tony Sorrentino proposed openstack/openstack-helm-infra master: MariaDB Backup Script https://review.openstack.org/633821 | 13:47 |
| *** kranthikirang has quit IRC | 13:47 | |
| *** howell has joined #openstack-helm | 14:11 | |
| *** sdake has quit IRC | 14:20 | |
| srwilkers | o/ | 14:32 |
| srwilkers | thanks evrardjp. i thought i had subscribed to the new mailing list when it was announced, but doesn't seem to be the case | 14:34 |
| *** sdake has joined #openstack-helm | 14:35 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: WIP/DNM: Armada check to test password rotation https://review.openstack.org/632558 | 14:38 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Add armada experimental job to osh-infra https://review.openstack.org/633067 | 14:40 |
| *** sdake has quit IRC | 14:49 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: Armada Check: Add release uuid value to charts https://review.openstack.org/633967 | 14:52 |
| *** sdake has joined #openstack-helm | 14:54 | |
| *** cfriesen has joined #openstack-helm | 14:55 | |
| *** bocob has joined #openstack-helm | 15:04 | |
| bocob | hi.. anyone can help me with a router namespace issue? VM can ping router interface, but would not ping interface that connects to the external network, so it would not get out to internet. the host can ping the router interface that connects to the external network. so I think there is so issue at the router NS... anyone see this before? | 15:08 |
| *** sdake has quit IRC | 15:11 | |
| *** sdake has joined #openstack-helm | 15:13 | |
| *** kranthikirang has joined #openstack-helm | 15:15 | |
| *** georgk has quit IRC | 15:22 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [WIP] [CEPH] Switch to using ceph-volume https://review.openstack.org/633981 | 15:45 |
| openstackgerrit | Tony Sorrentino proposed openstack/openstack-helm-infra master: MariaDB Backup Script https://review.openstack.org/633821 | 15:51 |
| *** gkadam-bmgr has quit IRC | 15:57 | |
| evrardjp | how can I debug a very slow deployment? It seems wait for pods is waiting for a job to be marked as 1/1 in completions (kubectl get jobs -n openstack) . | 16:08 |
| evrardjp | but kubectl describe job <jobname> -n openstack says completions: 1 pod statuses: 1 running/ 0 succeded / 0 failed | 16:09 |
| evrardjp | I am obviously missing something | 16:09 |
| *** sdake has quit IRC | 16:14 | |
| *** jamesgu has joined #openstack-helm | 16:17 | |
| *** jamesgu_ has joined #openstack-helm | 16:17 | |
| *** kranthikirang has quit IRC | 16:19 | |
| *** todin_ has joined #openstack-helm | 16:21 | |
| *** sdake has joined #openstack-helm | 16:21 | |
| cfriesen | has anyone else run into problems with the "mariadb-ingress-*" pods getting stuck in "Terminating" status? | 16:25 |
| evrardjp | cfriesen: nope | 16:27 |
| portdirect | evrardjp / cfriesen : both of these issues are likley to be as a result of the k8s cluster/container runtime you are using | 16:27 |
| portdirect | evrardjp: have you checked that clocks are co-ordinated? this is a common cause of issues liek the one you are seeing | 16:27 |
| evrardjp | mmm | 16:27 |
| evrardjp | you mean between the deploy node and k8s cluster? | 16:27 |
| evrardjp | "deploy" | 16:28 |
| portdirect | within the cluster | 16:28 |
| evrardjp | I don't have access to worker nodes | 16:28 |
| evrardjp | fun | 16:28 |
| evrardjp | can I try something else? :p | 16:28 |
| evrardjp | (I mean I can't ssh into them, if you know a command I can run to easily show that using k8s api, feel free to share) | 16:29 |
| portdirect | simplest way - (being lazy) | 16:30 |
| portdirect | are you running ceph on that cluster - with mons spread acros nodes? | 16:30 |
| evrardjp | nope | 16:31 |
| evrardjp | ses is on a different location | 16:31 |
| cfriesen | portdirect: we're just using normal docker. If I run "docker ps -a | awk /mariadb/'{print $1}'" then it shows three containers. Trying to get a bash shell in one hangs, in the other it says the container isn't running, and in the third it complains "executing setns process caused 'exit status 21'" | 16:31 |
| portdirect | if so then `ceph -s` will at least give you some insight into how things are across them | 16:31 |
| portdirect | cfriesen: that sounds like an unhappy docker daemon for sure. | 16:31 |
| evrardjp | portdirect: agree with you on cfriesen issue | 16:32 |
| portdirect | the one thats not running, that'll be a red herring - you cant exec into a non running one | 16:32 |
| cfriesen | portdirect: for the third case, I'm seeing a zombie "pause" process which is the child of docker-containerd-shim | 16:32 |
| evrardjp | reboot the node? :p | 16:33 |
| portdirect | how did you bring up k8s? | 16:33 |
| cfriesen | automated install stuff, via puppet. pretty standard stuff I think. all the other resources cleaned up fine except for these mariadb-ingress pods | 16:34 |
| cfriesen | I've been using this cluster all day yesterday testing various helm charts and armada manifests | 16:35 |
| evrardjp | portdirect: ok I checked, all the nodes are in sync timewise. | 16:38 |
| *** roman_g has joined #openstack-helm | 16:38 | |
| evrardjp | so it's not a time issue | 16:38 |
| evrardjp | any other idea on the discrepencies between the two kubectl calls? | 16:38 |
| portdirect | not offhand | 16:43 |
| portdirect | without access to the cluster it would be hard to determine root cause | 16:44 |
| portdirect | are you a amemeber of the k8s slack? | 16:44 |
| *** sdake has quit IRC | 16:51 | |
| *** georgk has joined #openstack-helm | 16:52 | |
| cfriesen | portdirect: just to confirm, the way openstack-helm is set up the expectation is that the "pause" container will handle reaping children from the "real" container? Presumably the docker-containerd-shim process should then handle reaping the "pause" process if it exits? | 16:54 |
| *** georgk has quit IRC | 16:56 | |
| *** sdake has joined #openstack-helm | 17:00 | |
| portdirect | cfriesen: for some pods yes | 17:01 |
| portdirect | docker is odd in this respect in k8s - as its the only container runtime where this needs to be explicitly set | 17:01 |
| portdirect | all others that im aware of do this by default | 17:01 |
| cfriesen | what container runtime do you folks use? | 17:03 |
| evrardjp | good question | 17:13 |
| portdirect | personally i use cri-o with runc backing it in my home lab | 17:16 |
| portdirect | airship uses docker with containerd -> runc behind it | 17:16 |
| portdirect | and the gates use docker/containerd/runc - as this is the most common k8s config atm | 17:17 |
| portdirect | its one of the reasons we dont handle k8s deployment in osh though - as its just as diverse in the world of k8s as openstack is | 17:18 |
| srwilkers | evrardjp: with respect to: but kubectl describe job <jobname> -n openstack says completions: 1 pod statuses: 1 running/ 0 succeded / 0 failed | 17:26 |
| srwilkers | that indicates the jobs still executing, so i wouldn't suspect the pods waiting on that job to move beyond `init` until that completes | 17:26 |
| evrardjp | yes? | 17:26 |
| srwilkers | sorry - late to the party | 17:26 |
| evrardjp | Would you mind having a look at that with me and logs when I have finished my meeting? | 17:27 |
| srwilkers | sure | 17:27 |
| evrardjp | confused about what you just said :) | 17:27 |
| *** aojea has quit IRC | 17:33 | |
| portdirect | evrardjp: what steves asking if if you can get logs from the job thats showing as still running | 17:34 |
| portdirect | `kubectl describe -n openstack job <foo>` would be the 1st step there | 17:34 |
| openstackgerrit | chinasubbareddy mallavarapu proposed openstack/openstack-helm master: OSH: Add ingress netpol for ceph-rgw pods https://review.openstack.org/633045 | 17:42 |
| openstackgerrit | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: ceph-rgw: Add network policy for ceph-rgw pods https://review.openstack.org/632567 | 17:42 |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [WIP] [CEPH] Switch to using ceph-volume https://review.openstack.org/633981 | 18:03 |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [WIP] [CEPH] Switch to using ceph-volume https://review.openstack.org/633981 | 18:03 |
| *** kranthikirang has joined #openstack-helm | 18:03 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [WIP] [CEPH] Switch to using ceph-volume https://review.openstack.org/633981 | 18:06 |
| *** kranthikirang has quit IRC | 18:07 | |
| *** georgk has joined #openstack-helm | 18:35 | |
| *** kranthikirang has joined #openstack-helm | 18:39 | |
| *** georgk has quit IRC | 18:40 | |
| *** kranthikirang has quit IRC | 18:44 | |
| *** kranthikirang has joined #openstack-helm | 19:06 | |
| *** georgk has joined #openstack-helm | 19:08 | |
| *** georgk has quit IRC | 19:12 | |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm-infra master: This adds the ability to specify custom resource dependencies https://review.openstack.org/634037 | 19:31 |
| *** sdake has quit IRC | 20:06 | |
| *** sdake has joined #openstack-helm | 20:07 | |
| *** georgk has joined #openstack-helm | 20:14 | |
| openstackgerrit | Kim Ha-gun proposed openstack/openstack-helm master: Octavia: add octavia chart and deployment scripts https://review.openstack.org/630821 | 20:14 |
| georgk | portdirect: hello! I just wanted to give you an update that this work is still ongoing: https://review.openstack.org/#/c/626894/ | 20:20 |
| georgk | portdirect: I am spending most time currently getting this up and running in Airship, hence no real progress on the chart | 20:21 |
| *** sdake has quit IRC | 20:39 | |
| *** sdake has joined #openstack-helm | 20:43 | |
| *** sdake has quit IRC | 20:44 | |
| *** sdake has joined #openstack-helm | 20:47 | |
| *** sdake has quit IRC | 21:01 | |
| *** sdake has joined #openstack-helm | 21:02 | |
| *** sdake has quit IRC | 21:38 | |
| *** sdake has joined #openstack-helm | 21:50 | |
| *** openstackgerrit has quit IRC | 21:50 | |
| *** ianw_pto is now known as ianw | 21:59 | |
| *** openstackgerrit has joined #openstack-helm | 22:04 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Fluentd: remove unused configuration section https://review.openstack.org/634054 | 22:04 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [alertmanager] default to 1 replica, multinode gate uses 3 https://review.openstack.org/633889 | 22:16 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: [Calico] Configuration robustness improvements https://review.openstack.org/625325 | 22:16 |
| *** howell has quit IRC | 22:25 | |
| *** bocob has quit IRC | 22:28 | |
| *** jamesgu_ has quit IRC | 22:40 | |
| *** jamesgu has quit IRC | 22:40 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm-infra master: Fix calling wrong variable name in gnocchi https://review.openstack.org/633891 | 22:54 |
| *** sdake has quit IRC | 23:01 | |
| *** georgk has quit IRC | 23:04 | |
| openstackgerrit | Merged openstack/openstack-helm master: Armada Check: Add release uuid value to charts https://review.openstack.org/633967 | 23:04 |
| *** jamesgu_ has joined #openstack-helm | 23:11 | |
| *** jamesgu has joined #openstack-helm | 23:11 | |
| *** sdake has joined #openstack-helm | 23:29 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Fluentd: remove unused configuration section https://review.openstack.org/634054 | 23:30 |
| *** sdake has quit IRC | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!