| *** goldyfruit_ has joined #openstack-containers | 01:07 | |
| *** goldyfruit_ has quit IRC | 01:58 | |
| *** goldyfruit_ has joined #openstack-containers | 03:28 | |
| *** ricolin has joined #openstack-containers | 03:43 | |
| *** goldyfruit_ has quit IRC | 05:00 | |
| *** sapd1_ has joined #openstack-containers | 05:02 | |
| *** sapd1 has quit IRC | 05:04 | |
| *** pcaruana has joined #openstack-containers | 05:25 | |
| *** iokiwi has quit IRC | 05:29 | |
| *** iokiwi has joined #openstack-containers | 05:30 | |
| *** jmlowe has quit IRC | 05:46 | |
| *** elenalindq has joined #openstack-containers | 06:52 | |
| *** elenalindq has quit IRC | 06:52 | |
| *** elenalindq has joined #openstack-containers | 06:53 | |
| *** arbrandes has quit IRC | 07:00 | |
| *** arbrandes has joined #openstack-containers | 07:01 | |
| *** udesale has joined #openstack-containers | 07:25 | |
| brtknr | jakeyip: podman is already available on train | 08:26 |
|---|---|---|
| *** ivve has joined #openstack-containers | 08:36 | |
| *** dim79 has joined #openstack-containers | 08:38 | |
| strigazi | meeting? | 08:53 |
| strigazi | jakeyip: flwang brtknr | 08:53 |
| jakeyip | o/ | 08:54 |
| brtknr | o. | 09:03 |
| brtknr | o/ | 09:03 |
| brtknr | where are you flwang ? | 09:05 |
| brtknr | strigazi: are you any closer to figuring out the loss of support for newer versions of kube_tag? | 09:06 |
| brtknr | looks like the tokens do not get placed in the path mounted inside the container | 09:07 |
| brtknr | strigazi: did flwang confirm he was attending? | 09:09 |
| strigazi | brtknr: I didn't have time to look into it. This was the reason I moved things to podman. | 09:10 |
| strigazi | brtknr: plus the fact that atomic doesn't exist in fedora coreos | 09:10 |
| *** namrata has joined #openstack-containers | 09:10 | |
| strigazi | brtknr: he hasn't confirmed | 09:11 |
| brtknr | i think then we need to support podman in stein if we cannot figure out another fix | 09:11 |
| brtknr | i spent quite a lot of time looking into this and havent worked out the cause yet | 09:11 |
| strigazi | brtknr: what we do with the meeting? | 09:12 |
| brtknr | I suppose there isn't enough topics on the agenda to warrant a full meeting | 09:14 |
| brtknr | Is there anything you'd like to discuss | 09:14 |
| brtknr | Also anything you'd like to discuss jakeyip ? | 09:14 |
| strigazi | python2, is there an objection to it? it is an ussuri goal anyway | 09:15 |
| jakeyip | no | 09:15 |
| jakeyip | I am quite interested in namrata question on versions of kube that has the CVE fixed | 09:17 |
| namrata | Looking for answer if anybody has | 09:18 |
| jakeyip | I know 1.14.6 works on Stein. What's a good solution to get those fixed versions running? podman with Train? | 09:18 |
| brtknr | strigazi: nno objection to python2 | 09:19 |
| strigazi | which CVE? | 09:19 |
| brtknr | although flwang deosnt seem to think we should make it non-voting yet... | 09:19 |
| brtknr | strigazi: billion laughs? | 09:19 |
| jakeyip | for context https://github.com/kubernetes/kubernetes/issues/83253 | 09:20 |
| strigazi | more ddos | 09:21 |
| strigazi | as mentioned, train+podman addresses the issue | 09:21 |
| strigazi | I'll have a look today for stein | 09:22 |
| namrata | strigazi so rocky will not have this issue solved | 09:22 |
| namrata | ? | 09:22 |
| strigazi | it will | 09:22 |
| strigazi | if we fix it for stein, it will be the same fix for rocky. | 09:22 |
| jakeyip | awesome, thanks strigazi | 09:24 |
| namrata | okay so stein also has same issue | 09:24 |
| namrata | thanks for the information strigazi | 09:24 |
| elenalindq | strigazi, thank you for looking into namrata's issue. Please let us know if we can help in anyway, it is easy for us to recreate the problem and we can provide more logs. | 09:28 |
| jakeyip | hi elenalindq namrata wondering where you are from? I'm from Nectar Research Cloud, Australia | 09:30 |
| elenalindq | jakeyip: I'm located in Stockholm, working for City Network | 09:30 |
| namrata | jakeyip I am located in Mumbai, India working remotely for City Network | 09:31 |
| jakeyip | nice to meet you | 09:31 |
| brtknr | elenalindq: namrata: as mentioned before, it will be best if you upgrade magnum component of your openstack to train, rather than waiting for the backports because the fixes are already in the 9.1.0 release | 09:32 |
| namrata | jakeyip likewise | 09:32 |
| elenalindq | jakeyip: dito :) | 09:32 |
| namrata | brtknr yeah but as rocky is maintained release and some of our regions are in rocky and we dont know yet when are we planning to upgrade so we need to find soution for this first | 09:34 |
| namrata | and obviously later we can plan to upgrade our regions | 09:34 |
| namrata | thanks brtknr | 09:34 |
| *** namrata has quit IRC | 09:45 | |
| *** rcernin has quit IRC | 09:46 | |
| dioguerra | flwang: CERN is running 2 in prod | 10:31 |
| *** ricolin has quit IRC | 10:32 | |
| brtknr | dioguerra: running 2? | 10:50 |
| *** namrata has joined #openstack-containers | 10:58 | |
| *** PrinzElvis has quit IRC | 11:08 | |
| brtknr | strigazi: is use_podman with atomic working for you? | 11:14 |
| brtknr | dioguerra: can you also test if you are there? | 11:17 |
| *** lpetrut has joined #openstack-containers | 11:26 | |
| *** namrata has quit IRC | 11:35 | |
| *** PrinzElvis has joined #openstack-containers | 12:15 | |
| *** dave-mccowan has quit IRC | 12:17 | |
| *** pcaruana has quit IRC | 12:24 | |
| strigazi | brtknr: I'll test | 12:27 |
| strigazi | brtknr flwang we didn't choose to run python2. We are using CentOS and the rpms from RDO. RDO is using python2. I imagine for ussuri RDO will switch to python3 if OpenStack drops python2 | 12:30 |
| strigazi | brtknr: flwang from RDO: 13:33 < strigazi> hello, for ussuri, will rdo support python3? | 12:38 |
| strigazi | brtknr: flwang from RDO again: 13:37 < amoralej> no, the plan is to release ussuri in centos8/python3 only | 12:38 |
| strigazi | brtknr: flwang https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8 | 12:39 |
| strigazi | brtknr: flwang according to that discussion, we (CERN), for ussuri, will use centos8 and python3. | 12:40 |
| *** pcaruana has joined #openstack-containers | 12:50 | |
| *** mgariepy has joined #openstack-containers | 12:54 | |
| *** goldyfruit_ has joined #openstack-containers | 12:59 | |
| *** ricolin has joined #openstack-containers | 13:07 | |
| *** namrata has joined #openstack-containers | 13:16 | |
| *** goldyfruit_ has quit IRC | 13:16 | |
| brtknr | strigazi: any joy? | 13:20 |
| brtknr | with use_podman label? | 13:20 |
| *** dave-mccowan has joined #openstack-containers | 13:21 | |
| *** openstackgerrit has joined #openstack-containers | 13:35 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use configured heat-container-agent tag https://review.opendev.org/695196 | 13:35 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Use configured heat-container-agent tag https://review.opendev.org/695196 | 14:18 |
| *** munimeha1 has joined #openstack-containers | 14:32 | |
| *** namrata has quit IRC | 14:36 | |
| *** PrinzElvis has quit IRC | 14:38 | |
| dim79 | strigazi - is there any other repository for openstackmagnum images dockerfiles? current dockerfiles in magnum do not look like were used for 1.15 at least | 14:42 |
| *** goldyfruit_ has joined #openstack-containers | 14:43 | |
| strigazi | dim79: openstack/magnum/dockerfiles | 14:45 |
| strigazi | brtknr: my devstack failed me misserably | 14:45 |
| strigazi | brtknr: I'm restacking | 14:45 |
| brtknr | strigazi: thanks | 14:46 |
| brtknr | strigazi: just discovered a bug that affects a few of our sites re Train: https://storyboard.openstack.org/#!/story/2006917 | 14:46 |
| brtknr | because we dont enable Cinder everywhere | 14:46 |
| dim79 | strigazi: in openstack/magnum/dockerfiles kubelet has KUBE_ALLOW_PRIV command line argument, which is removed from binary starting from kubelet 1.15 | 14:49 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP add containerd support https://review.opendev.org/695210 | 14:49 |
| dim79 | I mean: dockerfiles/kubernetes-kubelet/launch.sh, ARGS="... $KUBE_ALLOW_PRIV ...", KUBE_ALLOW_PRIV comes from /etc/kubernetes/config and has a value "--allow-privileged=...", which is removed: | 14:55 |
| dim79 | sudo runc exec kubelet /hyperkube kubelet --allow-privileged=true | 14:56 |
| dim79 | ... | 14:56 |
| dim79 | F1120 14:32:21.493933 11040 server.go:156] unknown flag: --allow-privileged | 14:56 |
| dim79 | thus kubelet fails to start till I remove or comment out KUBE_ALLOW_PRIV in /etc/kubernetes/config | 14:57 |
| dim79 | at the same time, I see some steps regarding KUBE_ALLOW_PRIV in 1.15 image at https://hub.docker.com/layers/openstackmagnum/kubernetes-kubelet/v1.15.5/images/sha256-fdb1b13e54f48487b4439b265d14e118a1ee1f57ab5a53c4be8f524fce1fe257 | 14:58 |
| dim79 | and wonder - where do they come from | 14:58 |
| brtknr | strigazi: nice! containerd will let us swap runtimeclass | 15:00 |
| dim79 | strigazi - found the answer, please disregard | 15:01 |
| brtknr | dim79: are you trying to debug why 1.15 is not running on atomic? | 15:01 |
| brtknr | dim79: are you trying to debug why 1.15.5 is not running on atomic? | 15:01 |
| dim79 | brtknr: yep | 15:02 |
| dim79 | 1.15.6 | 15:02 |
| dim79 | just found that I need to update my dockerfiles and set ADD_KUBE_ALLOW_PRIV=false in 'docker build' | 15:03 |
| *** johnthetubaguy has joined #openstack-containers | 15:09 | |
| brtknr | and does that work? | 15:19 |
| *** udesale has quit IRC | 15:20 | |
| brtknr | dim79: where do you work btw? | 15:20 |
| *** udesale has joined #openstack-containers | 15:20 | |
| dim79 | brtknr: Virtuozzo | 15:21 |
| dim79 | brtknr: so far - nodes are in ready state (after kubelet's args manual fix), system pods are running | 15:21 |
| dim79 | brtknr: my plan is to rebuild images with cherry-picked strigazi commits and than go deeper | 15:24 |
| brtknr | which commit are you cherry-picking? | 15:25 |
| dim79 | fe0f0efa7237 c9262419fd4 - to stein | 15:26 |
| *** jmlowe has joined #openstack-containers | 16:00 | |
| *** ivve has quit IRC | 16:09 | |
| *** jmlowe has quit IRC | 16:10 | |
| *** udesale has quit IRC | 16:14 | |
| *** dim79 has quit IRC | 16:18 | |
| *** ricolin has quit IRC | 16:24 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Only query Cinder API if volume size > 0 https://review.opendev.org/695239 | 16:35 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Only query Cinder API if volume size > 0 https://review.opendev.org/695239 | 16:40 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Use configured heat-container-agent tag https://review.opendev.org/695196 | 16:42 |
| *** goldyfruit___ has joined #openstack-containers | 16:53 | |
| *** goldyfruit___ is now known as goldyfruit | 16:54 | |
| *** goldyfruit_ has quit IRC | 16:56 | |
| brtknr | dim79 is that working for oyu | 17:03 |
| *** ramishra has quit IRC | 17:23 | |
| *** goldyfruit has quit IRC | 17:27 | |
| *** goldyfruit has joined #openstack-containers | 17:30 | |
| *** goldyfruit has quit IRC | 17:31 | |
| *** dim79 has joined #openstack-containers | 18:06 | |
| *** colby_ has quit IRC | 18:11 | |
| *** jmlowe has joined #openstack-containers | 18:17 | |
| *** dim79 has quit IRC | 18:34 | |
| *** munimeha1 has quit IRC | 19:29 | |
| *** lpetrut has quit IRC | 19:32 | |
| *** lpetrut has joined #openstack-containers | 19:32 | |
| *** jmlowe has quit IRC | 20:43 | |
| *** elenalindq has quit IRC | 21:19 | |
| *** pcaruana has quit IRC | 21:42 | |
| *** dave-mccowan has quit IRC | 22:29 | |
| *** rcernin has joined #openstack-containers | 22:31 | |
| flwang | brtknr: around? | 22:48 |
| brtknr | flwang hi | 23:00 |
| brtknr | Please review the bug fixes above | 23:01 |
| brtknr | Did you have anything to discuss | 23:02 |
| brtknr | also python2 is being a pain in the CI | 23:02 |
| flwang | brtknr: sorry i missed yesterday meeting | 23:03 |
| flwang | I understand the py2 pain and I'm trying to fix it | 23:03 |
| flwang | i'm reluctant to totally drop the gate since i know some of the users are still using py2 | 23:04 |
| flwang | i prefer to make it non vote instead of dropping it | 23:04 |
| flwang | i ping you since i'd like to understand the current status of the v1.14.6/v1.15.4+ support | 23:04 |
| flwang | is there any progress on this? | 23:04 |
| brtknr | Not yet, dim79 above seemed to have made some progress, I can’t even get cluster to deploy with use_podman label atm on atomic | 23:07 |
| brtknr | I am not sure if dim79 actually checked the status of the nodes in kube system namespxe | 23:08 |
| brtknr | I’m happy with non voting for the time being but not sure if it is likely to pass again | 23:09 |
| brtknr | but happy for you to have a go | 23:09 |
| brtknr | flwang | 23:10 |
| brtknr | Status of the pods* | 23:12 |
| brtknr | the nodes will report Ready and cluster will report Healthy even when it’s not | 23:13 |
| brtknr | please check use_podman label for me on master branch, it’s not working for me | 23:13 |
| flwang | the use_podman label works for me | 23:41 |
| flwang | i mean i cherrypicked the necessary commits to stein and it works for me | 23:41 |
| flwang | i have to cherry pick 11 commits | 23:42 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: DO NOT MERGE https://review.opendev.org/695322 | 23:44 |
| flwang | brtknr: ^ | 23:44 |
| brtknr | flwang: What about master? | 23:48 |
| brtknr | its not working for me on master. What params are you using for cluster and cluster template? | 23:49 |
| brtknr | and what’s the DNM above? Is that for python2 support? | 23:50 |
| flwang | the above it for py2 | 23:53 |
| flwang | brtknr: that my lables {u'cloud_provider_enabled': u'true', u'kube_tag': u'v1.16.3', u'etcd_tag': u'3.2.26', u'use_podman': u'true', u'heat_container_agent_tag': u'stein-dev', u'etcd_volume_size': u'5'} | 23:54 |
| flwang | i guess you're not setting the etcd tag | 23:54 |
| flwang | brb | 23:55 |
| brtknr | Why is the etcd tag essential? | 23:57 |
| brtknr | Shouldn’t we make it the default if that’s the cass | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!