| *** ivve has quit IRC | 00:16 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: fix: Deploy enable_service last (rocky only) https://review.opendev.org/696884 | 00:43 |
|---|---|---|
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora: Set rp_filter=1 for calico https://review.opendev.org/696916 | 00:44 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s: Clear cni configuration https://review.opendev.org/696915 | 00:44 |
| openstackgerrit | Simon Merrick proposed openstack/magnum-ui master: Add ui for resizing clusters https://review.opendev.org/696814 | 00:47 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s: Clear cni configuration https://review.opendev.org/696915 | 00:59 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora: Set rp_filter=1 for calico https://review.opendev.org/696916 | 00:59 |
| *** yolanda has quit IRC | 01:25 | |
| *** xinliang has joined #openstack-containers | 01:39 | |
| *** ykarel has joined #openstack-containers | 02:24 | |
| *** ykarel_ has joined #openstack-containers | 02:52 | |
| *** ykarel has quit IRC | 02:54 | |
| openstackgerrit | Simon Merrick proposed openstack/magnum-ui master: Add rolling upgrade ui https://review.opendev.org/697000 | 03:10 |
| *** ricolin has joined #openstack-containers | 03:17 | |
| openstackgerrit | Simon Merrick proposed openstack/magnum-ui master: Add rolling upgrade ui https://review.opendev.org/697000 | 03:23 |
| *** xinliang has quit IRC | 03:50 | |
| *** ykarel_ has quit IRC | 04:04 | |
| *** udesale has joined #openstack-containers | 04:13 | |
| *** ykarel_ has joined #openstack-containers | 04:21 | |
| *** ykarel__ has joined #openstack-containers | 04:22 | |
| *** ykarel_ has quit IRC | 04:25 | |
| *** ykarel__ is now known as ykarel | 04:48 | |
| *** ricolin has quit IRC | 05:09 | |
| *** udesale has quit IRC | 05:15 | |
| *** udesale has joined #openstack-containers | 05:16 | |
| *** udesale has quit IRC | 05:42 | |
| *** udesale has joined #openstack-containers | 05:51 | |
| *** elenalindq has joined #openstack-containers | 06:10 | |
| *** sapd1_x has joined #openstack-containers | 06:23 | |
| *** ykarel has quit IRC | 07:05 | |
| *** ykarel has joined #openstack-containers | 07:06 | |
| *** yolanda has joined #openstack-containers | 07:22 | |
| *** sapd1_x has quit IRC | 07:23 | |
| *** ykarel is now known as ykarel|lunch | 07:47 | |
| *** trident has quit IRC | 07:51 | |
| *** johanssone has quit IRC | 07:51 | |
| *** trident has joined #openstack-containers | 07:51 | |
| *** johanssone has joined #openstack-containers | 07:52 | |
| *** udesale has quit IRC | 08:43 | |
| *** udesale has joined #openstack-containers | 08:44 | |
| *** ykarel|lunch is now known as ykarel | 08:47 | |
| *** ivve has joined #openstack-containers | 08:51 | |
| *** dasp has quit IRC | 09:42 | |
| *** dasp has joined #openstack-containers | 09:43 | |
| trident | brtknr: With the latest patch set v1.14.8 deploys fine with calico plugin. With v1.15.5 I still get a bunch of pending pods. But that's perhaps expected? | 10:05 |
| brtknr | trident: cool, but 1.15.5 works with flannel? | 10:06 |
| *** rcernin has quit IRC | 10:06 | |
| *** pcaruana has joined #openstack-containers | 10:08 | |
| trident | brtknr: Yes. | 10:08 |
| brtknr | trident: interesting | 10:09 |
| brtknr | it could be a missing backport | 10:09 |
| brtknr | it could be another missing backport | 10:09 |
| brtknr | but I cant think of which one it could be | 10:11 |
| brtknr | Any reason why the pods are in pending? | 10:11 |
| brtknr | which pods are pending? | 10:11 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Prometheus metrics from traefik https://review.opendev.org/697044 | 10:13 |
| trident | brtknr: Might be a error in my test env. One moment. I'll try respinning in a few minutes. It's a bit of a mess with all the patches on top of each other. | 10:13 |
| brtknr | :D | 10:13 |
| brtknr | trident: you can simply do `git review -d 696916` to open a new branch with only the relevant patches :) | 10:14 |
| trident | brtknr: There, in a better state now. But the issue with 1.15.5 persists. | 10:27 |
| trident | brtknr: calico-kube-controllers, coredns, heapster, kube-dns-autoscaler, kubernetes-dashboard all hangs in Pending. Nodes are in state NotReady: "KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" | 10:28 |
| brtknr | is openstack cloud controller manager running? | 10:29 |
| trident | brtknr: Nope. | 10:29 |
| brtknr | is there a daemonset for it? | 10:29 |
| trident | brtknr: How do I check that? | 10:30 |
| brtknr | kubectl get ds -A | 10:30 |
| brtknr | trident: | 10:30 |
| trident | brtknr: kube-system openstack-cloud-controller-manager 0 0 0 0 0 node-role.kubernetes.io/master= 32m | 10:31 |
| brtknr | ok good so its there | 10:31 |
| brtknr | mind sharing the output of kubectl get all -A | 10:31 |
| brtknr | is there anything in failing stating? | 10:32 |
| brtknr | is there anything in failing state? | 10:32 |
| brtknr | is the master node labelled as master? | 10:33 |
| *** lpetrut has joined #openstack-containers | 10:34 | |
| trident | brtknr: Nothing in failing state in kubectl get all -A - only Pending | 10:35 |
| trident | brtknr: master node has the label "node-role.kubernetes.io/master=" | 10:36 |
| brtknr | hmm curious | 10:36 |
| trident | It does have a few taints though: "CriticalAddonsOnly=True:NoSchedule", "dedicated=master:NoSchedule", "node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule" and "node.kubernetes.io/not-ready:NoSchedule" | 10:37 |
| brtknr | trident: is there aynthing in Running state at all? | 10:37 |
| brtknr | any of the mopods | 10:38 |
| brtknr | any of the pods | 10:38 |
| trident | brtknr: calico-kube-controllers has Warning "FailedScheduling 56s (x29 over 41m) default-scheduler 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate." | 10:39 |
| trident | And so does the other pods. | 10:40 |
| brtknr | trident: is there aynthing in Running state at all? | 10:40 |
| trident | brtknr: Doesn't seem like it... | 10:41 |
| *** udesale has quit IRC | 10:44 | |
| brtknr | trident: I know what the issue is I think | 10:49 |
| trident | brtknr: Oh, cool! | 10:50 |
| brtknr | trident: however, its an extremely messy backport | 10:53 |
| brtknr | its this patch you need: https://review.opendev.org/#/c/694032/4 | 10:54 |
| brtknr | so I think I'm not going to bother pursuing it | 10:54 |
| *** udesale has joined #openstack-containers | 10:58 | |
| trident | brtknr: I'll check to see how bad it is to apply it. Just need to get some lunch first. Will get back to you with the result! | 11:01 |
| brtknr | trident: feel free to try, i dont think it will work personally | 11:02 |
| brtknr | trident: without some more backports | 11:03 |
| brtknr | as the way these pods are deployed has also changed | 11:04 |
| brtknr | the way that calico/flannel services are deployed has changed, they run as containers whereas in rocky, i think they run as systemd service | 11:05 |
| brtknr | trident: are you saying you dont have calico-node service running? | 11:10 |
| brtknr | trident: this might work but no guarentees, I omitted the changes for flannel as that runs as systemd service in rocky | 11:21 |
| brtknr | and as you say, its working fine | 11:21 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/697060 | 11:21 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/697060 | 11:24 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum-ui master: Use python binary chosen by devstack https://review.opendev.org/697064 | 11:40 |
| *** trident has quit IRC | 11:40 | |
| *** trident has joined #openstack-containers | 11:42 | |
| *** lpetrut has quit IRC | 11:53 | |
| trident | brtknr: Beautiful! https://review.opendev.org/697060 seem to have done the trick! | 11:54 |
| brtknr | trident: cool! | 11:55 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/697060 | 11:59 |
| trident | brtknr: Then we just need a few +2:s and +W:s. Anything I can do to speed that up? Comments on any of the changes for example? | 12:14 |
| brtknr | trident: that would help | 12:15 |
| brtknr | tomorrow is our weekly meeting | 12:15 |
| trident | brtknr: Again, thanks for working with us on this. It makes it so much easier not having to have locally applied patches... | 12:15 |
| brtknr | at 9AM UTC | 12:15 |
| brtknr | you can bring this up in the meeting and highlight its importance to you | 12:15 |
| trident | brtknr: Yeah, I intend to join it! Thanks! | 12:16 |
| brtknr | rocky release is fraught with other issues but helps to have the main deployment issues ironed out | 12:16 |
| brtknr | other issues are mainly related to outdated helm charts etc. | 12:17 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum-ui master: Use python binary chosen by devstack https://review.opendev.org/697064 | 12:36 |
| *** udesale has quit IRC | 12:46 | |
| *** udesale has joined #openstack-containers | 12:47 | |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Install traefik with helm https://review.opendev.org/697071 | 12:48 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Install traefik with helm https://review.opendev.org/697071 | 12:53 |
| *** dasp has quit IRC | 13:12 | |
| *** dasp has joined #openstack-containers | 13:12 | |
| *** mgariepy has quit IRC | 13:12 | |
| *** mgariepy has joined #openstack-containers | 13:15 | |
| *** lpetrut has joined #openstack-containers | 13:40 | |
| *** dave-mccowan has joined #openstack-containers | 14:02 | |
| *** ricolin has joined #openstack-containers | 14:03 | |
| *** trident has quit IRC | 14:24 | |
| *** trident has joined #openstack-containers | 14:25 | |
| *** jhesketh has quit IRC | 14:27 | |
| *** jhesketh has joined #openstack-containers | 14:28 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** dave-mccowan has quit IRC | 14:42 | |
| *** ykarel is now known as ykarel|afk | 14:46 | |
| *** munimeha1 has joined #openstack-containers | 14:53 | |
| brtknr | dioguerra: what are your thoughts on Helm3? | 14:54 |
| *** pcaruana has joined #openstack-containers | 15:04 | |
| *** udesale has quit IRC | 15:10 | |
| *** udesale has joined #openstack-containers | 15:10 | |
| dioguerra | brtknr: never used it | 15:18 |
| brtknr | dioguerra: ive been playing around with it today... life is a lot easier without tiller | 15:18 |
| dioguerra | brtknr: about this: https://review.opendev.org/#/c/697071/2/magnum/drivers/common/templates/kubernetes/helm/traefik.sh will not be able to do this yet... | 15:18 |
| dioguerra | we need the DaemonSet | 15:19 |
| brtknr | does the chart not deploy a daemonset? | 15:20 |
| brtknr | dioguerra: i was just thinking why we implement ingress in the current way where a node needs to be labelled to act as the ingress node | 15:22 |
| *** jmlowe has quit IRC | 15:52 | |
| *** elenalindq has quit IRC | 16:02 | |
| *** ricolin has quit IRC | 16:05 | |
| *** jmlowe has joined #openstack-containers | 16:09 | |
| *** ivve has quit IRC | 16:15 | |
| *** elenalindq has joined #openstack-containers | 16:17 | |
| *** udesale has quit IRC | 16:17 | |
| *** dave-mccowan has joined #openstack-containers | 16:19 | |
| *** ricolin has joined #openstack-containers | 16:26 | |
| *** lpetrut has quit IRC | 16:38 | |
| *** ricolin has quit IRC | 16:43 | |
| *** ricolin has joined #openstack-containers | 16:52 | |
| *** ricolin has quit IRC | 16:57 | |
| *** ykarel|afk has quit IRC | 17:03 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: [k8s] Fix rolling upgrade with podman https://review.opendev.org/696204 | 17:15 |
| *** ykarel|afk has joined #openstack-containers | 17:22 | |
| *** ykarel|afk is now known as ykarel|away | 17:30 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: [k8s] Allow cluster upgrade without specifying kube_tag https://review.opendev.org/697143 | 17:38 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: [fix] Allow cluster upgrade without specifying kube_tag https://review.opendev.org/697143 | 17:40 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: [fix] Allow cluster OS upgrade without specifying kube_tag https://review.opendev.org/697143 | 17:42 |
| *** jhesketh has quit IRC | 17:42 | |
| *** jhesketh has joined #openstack-containers | 17:44 | |
| *** jmlowe has quit IRC | 18:15 | |
| *** jmlowe has joined #openstack-containers | 18:15 | |
| *** jmlowe has quit IRC | 18:24 | |
| *** jmlowe has joined #openstack-containers | 18:40 | |
| *** ykarel|away has quit IRC | 18:59 | |
| *** gmann is now known as gmann_afk | 19:00 | |
| *** ykarel|away has joined #openstack-containers | 19:12 | |
| *** ivve has joined #openstack-containers | 19:28 | |
| *** ykarel|away has quit IRC | 19:39 | |
| *** ramishra has quit IRC | 19:45 | |
| *** gmann_afk is now known as gmann | 20:25 | |
| *** vesper11 has quit IRC | 20:49 | |
| *** vesper11 has joined #openstack-containers | 20:51 | |
| *** elenalindq has quit IRC | 21:06 | |
| *** pcaruana has quit IRC | 22:06 | |
| *** munimeha1 has quit IRC | 22:43 | |
| *** rcernin has joined #openstack-containers | 22:57 | |
| *** ivve has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!