| brtknr | jakeyip: its slightly incorrect too, atm stein only supports upto 1.14.6 | 00:18 |
|---|---|---|
| brtknr | 1.14.7 and 1.14.8 do not seem compatible with stein for whatever reason | 00:18 |
| jakeyip | yeah doesn't work for me too. I tried 1.14.6 and that works | 00:19 |
| brtknr | flwang: ^^ | 00:19 |
| jakeyip | error in `pod/kube-flannel-ds-amd64-`, is that what is happening for you? | 00:20 |
| jakeyip | `kube-system pod/kube-flannel-ds-amd64-gptmd 0/1 CrashLoopBackOff 10 27m` | 00:21 |
| brtknr | yep same thing | 00:21 |
| jakeyip | @brtknr: you fixing up the pep8 errors? it's your commit but I can help if you like | 00:22 |
| brtknr | I am going to bed now, will take a look tomrrow, nearly 1am here | 00:23 |
| jakeyip | ok I'll do it | 00:23 |
| brtknr | thankss :) | 00:23 |
| jakeyip | good night :) | 00:23 |
| brtknr | have a good day! will you come to the meeting at 9am utc tomorrow? | 00:24 |
| jakeyip | yeah sure :) | 00:24 |
| brtknr | sweet! | 00:26 |
| openstackgerrit | Jake Yip proposed openstack/magnum master: Add compatibility matrix for kube_tag https://review.opendev.org/685675 | 00:40 |
| *** goldyfruit has joined #openstack-containers | 01:12 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support TimeoutStartSec for k8s systemd services https://review.opendev.org/690445 | 01:14 |
| *** udesale has joined #openstack-containers | 03:42 | |
| *** dave-mccowan has quit IRC | 04:02 | |
| *** dave-mccowan has joined #openstack-containers | 04:12 | |
| *** ykarel|away has joined #openstack-containers | 04:35 | |
| *** goldyfruit has quit IRC | 05:00 | |
| *** dave-mccowan has quit IRC | 05:05 | |
| *** ykarel|away is now known as ykarel | 05:47 | |
| *** trident has quit IRC | 07:57 | |
| *** trident has joined #openstack-containers | 08:06 | |
| *** ivve has joined #openstack-containers | 08:19 | |
| *** flwang1 has joined #openstack-containers | 08:31 | |
| *** ykarel is now known as ykarel|lunch | 08:37 | |
| brtknr | Morning flwang, strigazi, jakeyip, Meeting in 15 minutes? | 08:43 |
| flwang1 | brtknr: yes | 08:45 |
| jakeyip | sure | 08:46 |
| flwang1 | brtknr: can you please revisit this https://review.opendev.org/#/c/690445/ ? | 08:46 |
| flwang1 | brtknr: the podman doesn't work for me without this patch | 08:47 |
| brtknr | Sure thing | 09:00 |
| strigazi | meeting? | 09:01 |
| brtknr | 0/ | 09:01 |
| flwang1 | #startmeeting Magnum | 09:01 |
| openstack | Meeting started Wed Nov 13 09:01:48 2019 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:01 |
| *** openstack changes topic to " (Meeting topic: Magnum)" | 09:01 | |
| openstack | The meeting name has been set to 'magnum' | 09:01 |
| flwang1 | #topic roll call | 09:02 |
| *** openstack changes topic to "roll call (Meeting topic: Magnum)" | 09:02 | |
| flwang1 | o/ | 09:02 |
| strigazi | o/ | 09:02 |
| jakeyip | o/ | 09:02 |
| brtknr | o/ | 09:02 |
| flwang1 | strigazi: long time no see | 09:02 |
| flwang1 | thank you for joining, guys | 09:02 |
| strigazi | last week I was in the summit. I could not join. | 09:03 |
| flwang1 | before we go through the agenda on https://etherpad.openstack.org/p/magnum-weekly-meeting, anything you guys want to start first? | 09:03 |
| flwang1 | strigazi: anything you can share from the summit? | 09:03 |
| brtknr | How’s the summit? | 09:04 |
| strigazi | I'm reading the etherpad, give me a moment | 09:04 |
| strigazi | The summit was 1800 attendees | 09:06 |
| strigazi | From my experience, an english speaking conference in China is not going to be attractive. I'm sure the conference would be more popular in chinese. | 09:07 |
| flwang1 | strigazi: i can imagine | 09:07 |
| strigazi | I was not able to attend the PTG, there was an issue with my flight and I had to leave earlier. (Strike in germany) | 09:08 |
| strigazi | I didn't see many new projects/products in the summit. | 09:09 |
| strigazi | s/many/any/ | 09:09 |
| flwang1 | sigh... | 09:09 |
| brtknr | Stig, the only person who attended from StackHPC shared similar feelings | 09:10 |
| flwang1 | strigazi: do you know where will be the next summit? | 09:11 |
| flwang1 | North America? | 09:11 |
| brtknr | I believe its in Vancouver | 09:11 |
| strigazi | IMO, the TC should focus on stabilizing the core-projects. No new crazy changes | 09:11 |
| strigazi | yes, vancouver | 09:12 |
| jakeyip | that'll be the third time in vancouver | 09:12 |
| flwang1 | strigazi: I'd like to see TC take more responsibility on T instead of others | 09:12 |
| flwang1 | i'd like to go the next one | 09:12 |
| brtknr | What OpenStack needs is fewer projects that work well | 09:12 |
| flwang1 | i have missed the the other two in vancouver | 09:13 |
| jakeyip | long long flight for us flwang1 :) | 09:13 |
| strigazi | Keystone, glance, nova, neutron should work. The rest is debatable. | 09:13 |
| flwang1 | jakeyip: i know, my friend :) | 09:13 |
| brtknr | Equally long time to get there from the UK I think | 09:13 |
| flwang1 | brtknr: which city are you based in UK? | 09:14 |
| brtknr | Bristol, west coast... not that UK is very wide to begin with | 09:14 |
| flwang1 | jakeyip: are you in Sydney or Melbourne? | 09:14 |
| jakeyip | Melbourne. The Core team is in Melbourne. | 09:14 |
| jakeyip | Nectar Cloud Core Services team, to clear up any confusion with the 'core' work | 09:15 |
| jakeyip | word | 09:15 |
| flwang1 | jakeyip: :) | 09:15 |
| flwang1 | strigazi: anything else you want to share? | 09:15 |
| strigazi | no, there was nothing else | 09:15 |
| jakeyip | was there much interest in magnum? :) | 09:16 |
| brtknr | that was going to be my question | 09:16 |
| strigazi | I don't think any of our contributors was there. Mohamed was there | 09:16 |
| strigazi | We didn't have a Project Update. | 09:17 |
| strigazi | So I can not tell what was the interest. | 09:17 |
| flwang1 | fair enough | 09:18 |
| strigazi | Finally, | 09:19 |
| strigazi | Manila and other teams will have additional on-line TPGs | 09:19 |
| strigazi | s/TPG/PTG/ | 09:19 |
| brtknr | Are we other teams? | 09:20 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support TimeoutStartSec for k8s systemd services https://review.opendev.org/690445 | 09:20 |
| strigazi | We should be | 09:20 |
| brtknr | Do we have a date/time | 09:21 |
| strigazi | No | 09:21 |
| flwang1 | strigazi: brtknr: if you guys all think we should have a dedicated PTG, then we can plan it | 09:21 |
| flwang1 | before the Xmas holiday | 09:22 |
| brtknr | I think it would be useful to have some kind of planning meeting, even if we dont call it a PTG | 09:22 |
| strigazi | Let's decide next week? Ricardo is not here. I prefer that he is available before we (cern) can commit to something. | 09:22 |
| flwang1 | strigazi: works for me | 09:22 |
| brtknr | sounds good | 09:22 |
| flwang1 | when we say PTG, how long the session we need? | 09:22 |
| flwang1 | given we're a world wide team, the TZ is still a problem for us | 09:23 |
| strigazi | Two two-hour sessions? | 09:23 |
| strigazi | In different days? | 09:23 |
| flwang1 | then can we split it into 2 days? | 09:23 |
| strigazi | I don't think we need more | 09:23 |
| strigazi | yes, exactly. | 09:23 |
| flwang1 | 4 hours is enough i think | 09:23 |
| strigazi | yeap | 09:23 |
| flwang1 | next Wed and Thu? | 09:24 |
| brtknr | Would a meet/hangout be option? | 09:24 |
| brtknr | Would meet/hangout be an option? | 09:24 |
| strigazi | I can not say for sure now, I need to talk to others here | 09:24 |
| flwang1 | brtknr: yep | 09:24 |
| brtknr | or would it be iRC only? | 09:25 |
| flwang1 | or etherpad | 09:25 |
| flwang1 | all good for me, i prefer to start with meet/hangout to say hi for each other | 09:26 |
| flwang1 | then we can stay on etherpad | 09:26 |
| flwang1 | and use the voice call for necessary cases | 09:26 |
| strigazi | ok | 09:27 |
| flwang1 | strigazi: did you see my email about master resize? | 09:28 |
| brtknr | okay shall we move to a topic on the agenda from roll call? | 09:28 |
| flwang1 | the main thing i'd like to do in U release is the master resize and containerized master nodes | 09:28 |
| flwang1 | sure | 09:29 |
| flwang1 | #topic stable/stein 8.2.0 | 09:29 |
| *** openstack changes topic to "stable/stein 8.2.0 (Meeting topic: Magnum)" | 09:29 | |
| flwang1 | brtknr: would you like to give us an updates? | 09:30 |
| brtknr | Yes, so we recently noticed that the dns autoscaler is broken in stein as the docker repo has been removed completely | 09:31 |
| brtknr | also fa27 as also been removed so CI jobs are failing | 09:31 |
| jakeyip | same here... | 09:31 |
| brtknr | stein 8.2.0 incorporates these changes | 09:31 |
| brtknr | for us at stackhpc, we also need to support multiple NICS on a cluster and I have backported changes from master to enable this | 09:32 |
| brtknr | lastly, i'd like to also incorporate changes to support 1.14.7,1.14.8 in stein, possibly also 1.15.x, but havent managed to get to the bottom of why 1.14.7 and 1.14.8 clusters fail successfuly to spawn calico and flannel services in kube-system namespace | 09:33 |
| brtknr | does any of it seem controversial? | 09:33 |
| flwang1 | brtknr: i think that's why strigazi replaced the atomic system container with podman | 09:34 |
| flwang1 | strigazi: do you know the root cause why the 1.15.x doesn't work on atomic system container? | 09:34 |
| brtknr | i think podman was to support 1.16.x | 09:34 |
| flwang1 | brtknr: without podman, the max version of v1.15.x working for me is v1.15.3 | 09:35 |
| flwang1 | after cherry-pick the podman patch, v1.15.5 works for me | 09:36 |
| flwang1 | i'm curious the root cause | 09:36 |
| strigazi | I haven't tried, I don't know. we are using 1.15.3 with atomic. | 09:37 |
| jakeyip | wondering if it is efficient to spend time figuring out why stein won't work with 1.14.7+? I think users would like to see 1.15 / 1.16 support more | 09:37 |
| jakeyip | for us I think we will support at least one good version in stein and figure out how to get to train ASAP | 09:38 |
| jakeyip | so many nice new features | 09:38 |
| strigazi | 1.14.x should work in atomic. | 09:38 |
| strigazi | where x any version | 09:38 |
| strigazi | I will try and let you know | 09:38 |
| brtknr | strigazi: we have multiple sites where 1.14.7 and 1.14.8 are consistently failing to spawn with upstream stable/stein | 09:39 |
| strigazi | #action strigazi to try latest 1.14.x with atomic | 09:39 |
| brtknr | as in, calico and flannel pods fail to start | 09:39 |
| strigazi | brtknr: what is the failure? why they don't start? what is the error? | 09:39 |
| jakeyip | same thing I am seeing (flannel crashing) | 09:39 |
| brtknr | but please try with upstream stable/stein, not a modified branch | 09:39 |
| brtknr | yes, lots of CrashLoopBacks | 09:40 |
| strigazi | yes, but why? it can't read its token? | 09:40 |
| strigazi | can you do logs? | 09:40 |
| brtknr | it caues everything else to stay in pending state | 09:40 |
| brtknr | cant read logs, says IP not assigned | 09:40 |
| jakeyip | logs is broken for us I still haven't figured out why? does it work for you? | 09:40 |
| strigazi | ssh to node, docker logs | 09:41 |
| jakeyip | same here brtknr. (I feel like I'm saying that a lot this meeting) | 09:41 |
| strigazi | also k get nodes? | 09:41 |
| strigazi | do you see an IP? | 09:41 |
| strigazi | if k8s doesn't node ips (i.e. the occm hasn't given one) | 09:42 |
| strigazi | logs won't work | 09:42 |
| brtknr | occm has a daemonset but doesnt spwan a pod | 09:43 |
| brtknr | occm has a daemonset but doesnt spwan the pod | 09:43 |
| flwang1 | i think it maybe related to the occm | 09:43 |
| brtknr | when i do k get nodes, no IP | 09:44 |
| strigazi | that is why logs don't work | 09:44 |
| jakeyip | I've got a failing cluster on hand, where should I dump the output? | 09:44 |
| strigazi | paste.openstack.org | 09:44 |
| strigazi | in fedora: fpaste <file> | 09:44 |
| brtknr | jakeyip: you can also do | nc seashells.io 1337 | 09:45 |
| brtknr | much easier :) | 09:45 |
| jakeyip | https://seashells.io/v/2MkCnqdw | 09:45 |
| jakeyip | brtknr: exactly what I was looking for :P | 09:45 |
| strigazi | brtknr: well not easier than fedora | 09:46 |
| brtknr | https://seashells.io/p/2MkCnqdw for plaintext | 09:46 |
| strigazi | 5 chars vs 21 | 09:46 |
| strigazi | 6 chars :) | 09:46 |
| brtknr | nc seashells.io 1337 is platform agnistic :P | 09:46 |
| strigazi | and not community managed | 09:47 |
| jakeyip | ok can we concentrate on the error message please :P | 09:47 |
| brtknr | tbh i didnt know about fpaste, good to know... | 09:47 |
| strigazi | jakeyip: ssh to master, docker ps | grep flannel | 09:48 |
| strigazi | docker logs <flannel container> | 09:48 |
| strigazi | brtknr: jakeyip flwang1 before continueing with debugging, anything else for the meeting? | 09:49 |
| jakeyip | blank when I run logs | 09:49 |
| jakeyip | http://paste.openstack.org/raw/786024/ | 09:49 |
| brtknr | are we happy with the shopping list for stein-8.2.0 | 09:49 |
| strigazi | To sync via email for the online planning/PTG | 09:49 |
| *** ykarel|lunch is now known as ykarel | 09:49 | |
| brtknr | anything else people want to ad | 09:49 |
| strigazi | jakeyip: docker ps -a | grep flannel | grep -v pause | 09:50 |
| jakeyip | oh I have a minor bug. I upgraded to stein and my public templates were all not visible to the users anymore | 09:50 |
| flwang1 | strigazi: i'd like to know the master resize work you mentioned before | 09:50 |
| jakeyip | turns out the new column in DB 'hidden' had the values set to 'NULL' instead of 1 or 0 | 09:50 |
| strigazi | flwang1: We did some work on adding/dropping members of the clusters. That's it. | 09:51 |
| strigazi | flwang1: We did some work on adding/dropping members from the etcd clusters. That's it. | 09:51 |
| jakeyip | I told brtknr it's minor and don't need to bother fixing it. But since there's going to be a new stein version not sure if we should fix this. | 09:51 |
| flwang1 | jakeyip: it can be fixed by update the existing cluster's 'hidden' field | 09:51 |
| flwang1 | strigazi: where can i see the code? | 09:51 |
| jakeyip | flwang1: yes all fixed but just to bring it up because it's a breaking behaviour | 09:52 |
| brtknr | jakeyip: if you can locate the commit, please cherry-pick it to stein-8.1.1... i agree 8.2.0 implies there are new features but its mostly bug fixes | 09:53 |
| strigazi | we haven't pushed it. But we need to decide first on VMs vm k8s cluster for master nodes. | 09:53 |
| strigazi | flwang1: ^^ | 09:53 |
| strigazi | flwang1: You have fork that runs the control in k8s, the work I mentioned is irrelevant to that use case. | 09:54 |
| strigazi | flwang1: You have fork that runs the control-place in k8s, the work I mentioned is irrelevant to that use case. | 09:54 |
| strigazi | flwang1: It makes sense only if the master nodes are in dedicated VMs and run etcd | 09:54 |
| flwang1 | strigazi: ok | 09:55 |
| flwang1 | i will think about it again | 09:55 |
| jakeyip | http://paste.openstack.org/show/786025/ | 09:55 |
| flwang1 | thanks for sharing that | 09:55 |
| strigazi | jakeyip: so flannel, calico etc, can't read the token to talk to the k8s api. | 09:57 |
| strigazi | jakeyip: I couldn't make it work without podman. | 09:58 |
| flwang1 | https://stackoverflow.com/questions/46178684/flannel-fails-in-kubernetes-cluster-due-to-failure-of-subnet-manager | 09:58 |
| strigazi | jakeyip: but this was for 1.16.x | 09:58 |
| brtknr | looks like they have backported the same changes to 1.14.7 and 1.14.8 | 09:58 |
| flwang1 | it sounds like we need another mount for the kubelet atomic system container | 09:58 |
| strigazi | we have /var/lib/kubelet already. | 09:59 |
| flwang1 | :( | 09:59 |
| strigazi | looks like they have backported the same changes to 1.14.7 and 1.14.8 What this means?? | 10:00 |
| strigazi | which changes? | 10:00 |
| flwang1 | i think brtknr doesn't know the changes, he just guess there are some changes :) | 10:00 |
| strigazi | oh, ok :) | 10:01 |
| brtknr | yes, its a guess | 10:01 |
| flwang1 | should we call this meeting done ? | 10:01 |
| strigazi | + | 10:01 |
| flwang1 | i'm going to leave | 10:01 |
| strigazi | +1 | 10:01 |
| brtknr | goodnight! | 10:01 |
| flwang1 | thank you guys | 10:01 |
| flwang1 | #endmeeting | 10:01 |
| *** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting" | 10:01 | |
| openstack | Meeting ended Wed Nov 13 10:01:27 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-11-13-09.01.html | 10:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-11-13-09.01.txt | 10:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-11-13-09.01.log.html | 10:01 |
| flwang1 | i will send email to you guys to follow up the online PTG | 10:02 |
| strigazi | +1 | 10:02 |
| brtknr | https://seashells.io/p/xaudCkCB | 10:03 |
| strigazi | this is fine | 10:03 |
| brtknr | this is what i see in install-cni pod | 10:03 |
| jakeyip | strigazi: does CERN runs its own registry ? | 10:03 |
| strigazi | Done configuring CNI. Sleep=true | 10:03 |
| strigazi | jakeyip: yes | 10:04 |
| strigazi | jakeyip: gitlab | 10:04 |
| strigazi | jakeyip: gitlab on prem, storage cephs rados-gw | 10:04 |
| strigazi | cephs rados-gw -> s3 API | 10:04 |
| jakeyip | might need to set up one thinking how to make it HA | 10:05 |
| brtknr | the k8s_POD_calico-node-8x8tw_kube-system_c436a68e-05fa-11ea-afbb-fa163e226df0_0 container doesnt appear to have any logs | 10:05 |
| jakeyip | images are in ceph using S3 API? | 10:06 |
| jakeyip | is there any databases for the metadata e.g. tags and such | 10:06 |
| strigazi | jakeyip: I don't know, I'm an end user | 10:06 |
| jakeyip | ah I see | 10:07 |
| strigazi | jakeyip: if are looking deploying one. gitlab is not implementing the registry v2 api. Which is not good. | 10:07 |
| strigazi | brtknr: I'm deploying a 1.14.8 cluster I'll let you know. | 10:08 |
| brtknr | strigazi: on devstack? | 10:08 |
| strigazi | Is there a story where I can put info? | 10:08 |
| strigazi | brtknr: in our production | 10:08 |
| brtknr | I'll create one now | 10:08 |
| jakeyip | strigazi: I see. thanks that's useful information | 10:09 |
| brtknr | strigazi: https://storyboard.openstack.org/#!/story/2006846 | 10:16 |
| brtknr | jakeyip: do you get the same errors with 1.14.8? | 10:22 |
| brtknr | i saw you were using 1.14.7 | 10:22 |
| jakeyip | I think so but I don't remember exactly. I can spin something up | 10:23 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/694032 | 10:23 |
| *** ianychoi has quit IRC | 10:24 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/694032 | 10:24 |
| brtknr | with regards to 8.2.0 or 8.1.1, it would be nice to also backport the PSP change to support 1.15.x | 10:28 |
| brtknr | I've tested 1.15.3 on devstack with the patch and it works | 10:28 |
| brtknr | strigazi: with the above branch | 10:29 |
| strigazi | 1.14.8 doesn't work for me either. | 10:45 |
| *** udesale has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack/magnum master: Support TimeoutStartSec for k8s systemd services https://review.opendev.org/690445 | 11:20 |
| brtknr | strigazi: \o/ i am not completely delusional then :) | 11:36 |
| *** henriqueof1 has quit IRC | 12:04 | |
| *** rcernin has quit IRC | 12:54 | |
| *** ianychoi has joined #openstack-containers | 13:00 | |
| *** ykarel is now known as ykarel|afk | 13:39 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/694032 | 13:39 |
| brtknr | strigazi: Any joy? Is this failing for the same reason 1.16.x was failing? | 14:00 |
| strigazi | brtknr: yes | 14:05 |
| *** bline has quit IRC | 14:07 | |
| *** bline has joined #openstack-containers | 14:12 | |
| *** lixingxing has joined #openstack-containers | 14:36 | |
| *** ykarel|afk has quit IRC | 14:43 | |
| *** ykarel has joined #openstack-containers | 14:49 | |
| *** lpetrut has joined #openstack-containers | 14:58 | |
| *** lixingxing has quit IRC | 15:37 | |
| *** lpetrut has quit IRC | 16:08 | |
| *** ykarel is now known as ykarel|away | 16:16 | |
| *** ivve has quit IRC | 16:26 | |
| *** goldyfruit has joined #openstack-containers | 16:55 | |
| *** mgariepy has quit IRC | 16:56 | |
| *** ykarel|away has quit IRC | 17:51 | |
| *** flwang1 has quit IRC | 19:36 | |
| *** rcernin has joined #openstack-containers | 22:32 | |
| *** goldyfruit has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!