| *** sapd1 has joined #openstack-containers | 00:39 | |
| flwang | brtknr: we can't make it as default because we're using the use_podman label | 01:02 |
|---|---|---|
| flwang | which means user may use atomic system container instead of podman to start the etcd | 01:03 |
| *** sapd1 has quit IRC | 01:06 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: DO NOT MERGE https://review.opendev.org/695322 | 01:29 |
| *** dave-mccowan has joined #openstack-containers | 01:35 | |
| *** dave-mccowan has quit IRC | 01:42 | |
| *** ricolin has joined #openstack-containers | 01:55 | |
| *** udesale has joined #openstack-containers | 03:53 | |
| *** ramishra has joined #openstack-containers | 04:37 | |
| *** elenalindq has joined #openstack-containers | 06:07 | |
| *** elenalindq has quit IRC | 06:08 | |
| *** elenalindq has joined #openstack-containers | 06:08 | |
| *** udesale has quit IRC | 06:15 | |
| *** udesale has joined #openstack-containers | 06:16 | |
| *** lpetrut has quit IRC | 06:21 | |
| *** udesale has quit IRC | 06:39 | |
| *** udesale has joined #openstack-containers | 06:39 | |
| *** rcernin has quit IRC | 07:12 | |
| brtknr | flwang: ah okay that makes sense now | 07:15 |
| *** pcaruana has joined #openstack-containers | 07:27 | |
| openstackgerrit | Namrata proposed openstack/magnum stable/rocky: Fix proportional autoscaler image https://review.opendev.org/695363 | 08:00 |
| *** dim79 has joined #openstack-containers | 08:19 | |
| *** flwang1 has joined #openstack-containers | 08:39 | |
| flwang1 | strigazi: around? | 08:39 |
| dim79 | brtknr https://storyboard.openstack.org/#!/story/2006846 - as a w/a - have you tried kubelet "--containerized" for k8s <= 1.16? | 08:50 |
| brtknr | dim79: no, is that working for you? | 08:51 |
| dim79 | k8s < 1.16, in 1.16 it is removed | 08:51 |
| dim79 | yes | 08:52 |
| brtknr | which version? | 08:52 |
| brtknr | which kube_tag did you use? | 08:52 |
| dim79 | 1.14.9, 1.15.6 | 08:52 |
| dim79 | I use my own images, rebuilt from magnum/dockerfiles from stein | 08:53 |
| brtknr | dim79: what about the upstream images? | 08:55 |
| brtknr | did you check if all pods are running in kube-system namespace? | 08:55 |
| dim79 | all except auto-scaler | 08:55 |
| dim79 | I haven't merged its fix yet | 08:55 |
| brtknr | dim79: hmm im curious | 08:56 |
| dim79 | yes, for 1.15 https://review.opendev.org/#/c/694032/3 is needed | 08:56 |
| dim79 | fedora atomic image I use - Fedora-AtomicHost-29-20191113.1 | 08:58 |
| dim79 | brtknr: > what about the upstream images? | 08:58 |
| brtknr | Yes, did you try the upstream kubernetes images from magnum repo? | 08:59 |
| dim79 | they are built from master, --containerized is absent in them. There from be matrix like implemented for ALLOW_PRIV | 08:59 |
| brtknr | I dont understand | 09:00 |
| brtknr | --containerized is absent for kubelet? | 09:02 |
| brtknr | or for all images? | 09:02 |
| *** namrata has joined #openstack-containers | 09:02 | |
| dim79 | may be I don't understand the issue correctly, but this option (--containerized) helped to solved 'absence of tokens in pods' issues | 09:02 |
| brtknr | Can you point me to the changelog where --containerized became a requirement? | 09:02 |
| dim79 | I can't | 09:03 |
| dim79 | I postponed investigation how to resolve the issue without it | 09:03 |
| brtknr | Thanks for digging into this dim79, this is very helpful | 09:04 |
| brtknr | Are you happy to post a patch? | 09:04 |
| brtknr | Looks like --containerized is deprecated | 09:05 |
| dim79 | Yes - in k8s 1.16 | 09:05 |
| dim79 | if you know what is instead - please share with me | 09:05 |
| openstackgerrit | Namrata proposed openstack/magnum stable/rocky: Fix proportional autoscaler image https://review.opendev.org/695371 | 09:08 |
| brtknr | dim79: looks like we may need to mount the volumes manually | 09:08 |
| dim79 | brtknr: have you found a k8s doc regarding this option removal? | 09:09 |
| brtknr | https://github.com/kubernetes/kubernetes/issues/74148 | 09:11 |
| brtknr | https://github.com/kubernetes/kubernetes/pull/74267 | 09:11 |
| brtknr | and the fact that its been removed in 1.16 | 09:11 |
| flwang1 | does that mean since 1.16. x, we can't run kubelet in a container? | 09:15 |
| flwang1 | but why v1.15.4+ is also affected? | 09:15 |
| brtknr | flwang1: it means that --containerized flag is deprecated as it was too unreliable | 09:16 |
| brtknr | but reading the notes online, all it does it a bunch of volume mounts | 09:16 |
| brtknr | we need to make sure we are mounting the same paths | 09:16 |
| flwang1 | https://github.com/kubernetes/kubernetes/pull/71351/files | 09:17 |
| flwang1 | FWIW, it's now much easier to find the solution | 09:19 |
| flwang1 | dim79: thank you so much | 09:19 |
| dim79 | no prob, help you for your help | 09:19 |
| dim79 | the most interesting link is https://github.com/kubernetes/kubernetes/issues/74148 | 09:20 |
| dim79 | from what I've read - I'll use '--containerized' till 1.16 :) | 09:20 |
| dim79 | and will dig deeper after all other issues will be fixed | 09:21 |
| flwang1 | is there any way we can see all the paths being mounted from a container inside? | 09:22 |
| *** ricolin has quit IRC | 09:22 | |
| flwang1 | if we can know what's the path we're missing, then it's easy to add it | 09:22 |
| dim79 | off-topic, fyi - switch to unprivileged flannel (https://review.opendev.org/#/c/694032/3) brokes its 'udp' mode due to: | 09:40 |
| dim79 | E1120 20:34:13.737832 1 main.go:289] Error registering network: failed to open TUN device: open /dev/net/tun: no such file or directory | 09:40 |
| dim79 | vxlan works ok | 09:40 |
| brtknr | dim79: i think vxlan is the default in train | 09:46 |
| dim79 | yes, but it is not in 'stein' | 09:47 |
| dim79 | does magnum support only default values? | 09:47 |
| brtknr | dim79: Hmm I have tried mounting other paths and still no luck | 10:50 |
| brtknr | Only way to get it working seems to be to add --containerized | 10:50 |
| brtknr | but only up to 1.15.x | 10:50 |
| dim79 | yep :( | 10:52 |
| dim79 | I believe another way is to run kubelet in host, but need to set it up and debug | 10:52 |
| brtknr | I dont understand how it works with podman though | 10:53 |
| brtknr | which also runs inside a container | 10:53 |
| dim79 | probably with podman kubelet runs in host's mount namespace (just an idea - I haven't looked at it yet) | 10:56 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Add prometheus-adapter https://review.opendev.org/691646 | 11:00 |
| *** udesale has quit IRC | 11:14 | |
| *** namrata has quit IRC | 11:22 | |
| *** ramishra has quit IRC | 11:38 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 12:22 |
| brtknr | dim79: ^ i think this fix will be fine for 1.{13,14,15} but after that, we will all have to move to podman | 12:23 |
| dim79 | agree | 12:23 |
| *** ricolin has joined #openstack-containers | 12:27 | |
| dim79 | regarding review - (sorry, I don't have account at opendev.org yet) - why do you fix only master's kubelet? | 12:27 |
| brtknr | good observation :) | 12:29 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 12:29 |
| dim79 | sorry, another observation - KUBELET_ARGS are rewritten later in both files. in -master.sh - at line 401, in -minion.sh - at line 243 | 12:37 |
| brtknr | dim79: yikes, | 12:47 |
| strigazi | brtknr: dim79 guys I left a comment in gerrit for the issue with --containerized. | 12:49 |
| brtknr | strigazi: why does it need to be where it was? | 12:49 |
| strigazi | brtknr: because it was designed like this. To fix the bug we should revert to the working state | 12:50 |
| brtknr | i just tested it by adding --containized argument to /etc/kubernetes/kubelet and restart and it works | 12:50 |
| dim79 | strigazi: than it will be applied to podman as well (because will be hardcoded in docker image) | 12:50 |
| brtknr | dim79: podman uses images from gcr.io, not the ones we build | 12:50 |
| dim79 | a | 12:51 |
| strigazi | brtknr: for all versions? eg, having two times --contairized works? | 12:51 |
| strigazi | as you want, I think undoing the errors is better that fixing in a different way. | 12:52 |
| strigazi | How about backports? and so on | 12:52 |
| brtknr | yup | 12:52 |
| brtknr | it just ignores the scond one | 12:52 |
| strigazi | we need to release rocky again with this fix, why not fix the image | 12:52 |
| brtknr | strigazi: okay fine.. :) | 12:52 |
| dim79 | if we are going to fix the image - need to add another build ARG for this - to do not put this option to 1.16 images | 12:54 |
| brtknr | 1.16 will not work with docker 1.13 | 12:54 |
| brtknr | "i dont think" 1.16 will not work with docker 1.13 | 12:54 |
| brtknr | it has weird mount propagation behaviour | 12:55 |
| brtknr | i remember hitting this last year when i was workign with something else | 12:55 |
| strigazi | I'm completely lost, what are we discussin now? | 12:55 |
| strigazi | 1.16 and 1.13 docker is new? | 12:55 |
| brtknr | Docker 1.13 has some issues with mount propagation | 12:57 |
| dim79 | I'm confused as well - how 1.16 docker images are built than? aren't thay built from magnum/dockerfiles/... ? | 12:57 |
| brtknr | the service account token which gets generated doesn't propagate to calico container | 12:58 |
| strigazi | brtknr: i tested that before and it worked | 12:58 |
| brtknr | without --containerized? | 12:58 |
| strigazi | dim79: the images for 1.16 and with use_podman are the hyperkube images. | 12:59 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 12:59 |
| dim79 | strigazi: are you going to build images for 1.16 without use_podman? | 13:00 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 13:01 |
| strigazi | dim79: we should stop building images for 1.16, they don't work with atomic. | 13:01 |
| dim79 | ok than | 13:03 |
| brtknr | strigazi: you happy with above | 13:04 |
| brtknr | ? | 13:04 |
| brtknr | i wont be able to merge it till we do somethng about python2 | 13:04 |
| brtknr | argh | 13:04 |
| strigazi | almost, we need to the builds again. Can you reference that this commit *should* remove the flag? I3efd4e55e885b95721f13279b44dc1246e2fd2e4 | 13:05 |
| strigazi | almost, we need to do the builds again. Can you reference that this commit *should* remove the flag? I3efd4e55e885b95721f13279b44dc1246e2fd2e4 | 13:05 |
| strigazi | almost, we need to do the builds again. Can you reference that this commit *should not* have removed the flag? I3efd4e55e885b95721f13279b44dc1246e2fd2e4 | 13:05 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 13:09 |
| brtknr | done^ | 13:09 |
| brtknr | strigazi: ^ | 13:09 |
| strigazi | ack | 13:09 |
| brtknr | strigazi: im excited about support for containerd | 13:13 |
| brtknr | does it play well with podman? | 13:13 |
| strigazi | it works, not extensive testing yet. | 13:14 |
| strigazi | we work on crfs and cvmfs+crfs too | 13:14 |
| strigazi | github.com/google/crfs | 13:14 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Drop python2 tests https://review.opendev.org/691721 | 13:14 |
| strigazi | brtknr: are we dropping python2 or non-voting? I don't mind dropping, does the TC want to drop it? | 13:15 |
| brtknr | flwang1 said he'd prefer non-voting | 13:16 |
| strigazi | but if openstack in general drops it, non-voting is not helping much. Do you want to clarify it? | 13:17 |
| brtknr | strigazi: i agree with you but he said he had customers who use python2 and would like to try fix the test if possible... i think it makes sense to set to non-voting if he's commited to try and fix it... | 13:19 |
| brtknr | strigazi: I'd prefer it gone tbh, wastes cpu cycles | 13:20 |
| brtknr | flwang | I understand the py2 pain and I'm trying to fix it | 13:22 |
| brtknr | flwang | i'm reluctant to totally drop the gate since i know some of the users are still using py2 | 13:22 |
| brtknr | flwang | i prefer to make it non vote instead of dropping it | 13:22 |
| brtknr | flwang | i ping you since i'd like to understand the current status of the v1.14.6/v1.15.4+ support | 13:22 |
| brtknr | flwang | is there any progress on this? | 13:22 |
| brtknr | strigazi: ^ | 13:22 |
| strigazi | so non-voting and he can have a look? | 13:28 |
| strigazi | brtknr: ^ | 13:28 |
| brtknr | sounds good to me | 13:28 |
| strigazi | dioguerra: can you update the patch? | 13:30 |
| *** yolanda has quit IRC | 13:33 | |
| *** yolanda has joined #openstack-containers | 13:36 | |
| *** udesale has joined #openstack-containers | 13:37 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Use configured heat-container-agent tag https://review.opendev.org/695196 | 13:44 |
| *** ramishra has joined #openstack-containers | 14:13 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add nginx_ingress_controller_chart_tag https://review.opendev.org/694286 | 14:15 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 14:20 |
| dioguerra | brtknr: already updated | 14:25 |
| dioguerra | strigazi ^ | 14:25 |
| brtknr | dioguerra: we are setting python2 to non voting | 14:25 |
| brtknr | the decision is to set python2 to nonvoting | 14:25 |
| brtknr | not drop it | 14:25 |
| dioguerra | The full removal of python2 needs openstack/requirents. | 14:26 |
| dioguerra | cool | 14:26 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Set python 2.7 non-voting https://review.opendev.org/695449 | 14:36 |
| *** goldyfruit has joined #openstack-containers | 14:46 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add nginx_ingress_controller_chart_tag https://review.opendev.org/694286 | 14:46 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 14:47 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Drop python2 tests https://review.opendev.org/691721 | 14:58 |
| *** pcaruana has quit IRC | 15:02 | |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Set python 2.7 non-voting https://review.opendev.org/695449 | 15:13 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 15:19 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add nginx_ingress_controller_chart_tag https://review.opendev.org/694286 | 15:20 |
| *** pcaruana has joined #openstack-containers | 15:20 | |
| *** jmlowe has joined #openstack-containers | 15:22 | |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Set python 2.7 non-voting https://review.opendev.org/695449 | 15:23 |
| dioguerra | OK, i think i fixed it brtknr | 15:24 |
| brtknr | Great! | 15:24 |
| dioguerra | [1] https://review.opendev.org/#/c/691721/ drops python but keeps requirements for python2. | 15:24 |
| dioguerra | although its running extra tests and i dont know why build-openstack-api-refsuccess test-release-openstack build-openstack-releasenotes magnum-container-build | 15:27 |
| *** elenalindq has quit IRC | 15:45 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 15:57 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add nginx_ingress_controller_chart_tag https://review.opendev.org/694286 | 15:57 |
| brtknr | dioguerra: flwang1 wants to keep python2 tests | 16:01 |
| brtknr | but as non-voting | 16:01 |
| dioguerra | i dont know how to keep the tests. I can disable the API check, but not the openstack-tox-py27 because they come from the templates (as far as i understand) | 16:03 |
| dioguerra | I tried to search and cannot set templates to non-voting. | 16:03 |
| dioguerra | I checked nova, they dropped everything | 16:03 |
| *** udesale has quit IRC | 16:04 | |
| *** sapd1 has joined #openstack-containers | 16:12 | |
| *** dim79 has quit IRC | 16:17 | |
| *** jmlowe has quit IRC | 16:17 | |
| dioguerra | flwang: https://review.opendev.org/#/c/691721/5 | 16:29 |
| *** ricolin has quit IRC | 16:39 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Make traefik compatible with 1.16.x https://review.opendev.org/695525 | 16:47 |
| brtknr | i think flwang1 is still in sleepyland | 16:47 |
| brtknr | dioguerra: | 16:48 |
| *** sapd1 has quit IRC | 16:56 | |
| *** goldyfruit_ has joined #openstack-containers | 17:14 | |
| *** goldyfruit has quit IRC | 17:17 | |
| *** pcaruana has quit IRC | 17:51 | |
| *** pcaruana has joined #openstack-containers | 18:14 | |
| *** gmann is now known as gmann_afk | 18:38 | |
| *** pcaruana has quit IRC | 18:42 | |
| *** pcaruana has joined #openstack-containers | 19:06 | |
| *** flwang1 has quit IRC | 19:20 | |
| *** openstackstatus has quit IRC | 19:50 | |
| *** openstackstatus has joined #openstack-containers | 19:50 | |
| *** ChanServ sets mode: +v openstackstatus | 19:50 | |
| flwang | brtknr: around? does https://review.opendev.org/#/c/695415/ work for v1.16.x? | 20:40 |
| brtknr | flwang: no it doesnt, only upto 1.15.x | 20:43 |
| brtknr | i dont think there is a way to use atomic with 1.16.x | 20:43 |
| *** gmann_afk is now known as gmann | 20:45 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Use --containerized flag to support 1.{13,14,15}.x in Atomic https://review.opendev.org/695415 | 20:46 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Only query Cinder API if volume size > 0 https://review.opendev.org/695239 | 20:47 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Make traefik compatible with 1.16.x https://review.opendev.org/695525 | 20:47 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Add nginx_ingress_controller_chart_tag https://review.opendev.org/694286 | 20:48 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 20:49 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Use configured heat-container-agent tag https://review.opendev.org/695196 | 20:50 |
| brtknr | flwang: can you please also take a look at the patches above | 20:51 |
| brtknr | i have rebased on top of the CI fix | 20:52 |
| *** openstack has joined #openstack-containers | 21:16 | |
| *** ChanServ sets mode: +o openstack | 21:16 | |
| *** dave-mccowan has joined #openstack-containers | 21:22 | |
| *** pcaruana has quit IRC | 21:25 | |
| brtknr | I’ll do it in a different patch | 21:49 |
| brtknr | we need to rebuild the current images diest | 21:50 |
| brtknr | 21:50 | |
| brtknr | first | 21:50 |
| flwang | seems it has rebuilt the existing v1.15.5 | 21:52 |
| flwang | could you propose a new patch to support v1.15.6 and v1.14.9? | 21:53 |
| flwang | i can +2 it quickly | 21:53 |
| *** ramishra has quit IRC | 21:57 | |
| *** rcernin has joined #openstack-containers | 22:09 | |
| *** sapd1 has joined #openstack-containers | 23:09 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!