| *** goldyfruit_ has quit IRC | 00:09 | |
| flwang | brtknr: around? | 00:46 |
|---|---|---|
| flwang | i tested the v1.15.5 new image, it doesn't work for me unfortunately | 00:46 |
| *** sapd1 has quit IRC | 00:47 | |
| brtknr | really? | 00:56 |
| brtknr | 1 sec lemme test | 00:56 |
| brtknr | it worked for me earlier | 00:57 |
| brtknr | does it fail in the same way? | 00:57 |
| brtknr | does the kubelet container have --containerized arg? | 00:59 |
| brtknr | flwang: ^ | 01:00 |
| *** openstackstatus has joined #openstack-containers | 01:03 | |
| *** ChanServ sets mode: +v openstackstatus | 01:03 | |
| flwang | it failed for the same way on v1.15.5 | 01:05 |
| flwang | i'm using the latest image | 01:06 |
| flwang | brtknr: ^ | 01:06 |
| brtknr | does the kubelet container have --containerized arg flwang ? | 01:06 |
| flwang | sure | 01:06 |
| flwang | http://paste.openstack.org/show/786517/ | 01:07 |
| flwang | what the version you tested? | 01:08 |
| flwang | brtknr: | 01:08 |
| brtknr | wait... the calico-node containers are running... | 01:09 |
| brtknr | this is different | 01:09 |
| brtknr | 1.15.5 too | 01:10 |
| flwang | calico node is not managed by k8s "directly" | 01:12 |
| flwang | brtknr: do you mean v1.15.5 works for you with flannel? | 01:13 |
| brtknr | its working for me with calico | 01:16 |
| brtknr | havent tested flannel | 01:16 |
| brtknr | it was* | 01:16 |
| brtknr | flwang: what about 1.15.4? | 01:19 |
| brtknr | is that working | 01:19 |
| brtknr | its late here, i need to go to sleep | 01:19 |
| flwang | brtknr: ok, go for your bed :) | 01:20 |
| flwang | we can discuss tomorrow | 01:20 |
| flwang | we don't have v1.15.4 image on openstackmagnum dockerhub | 01:21 |
| brtknr | btw we need to build newer images for etcd | 01:23 |
| brtknr | v3.2.7 is quite old | 01:23 |
| brtknr | flwang: http://paste.openstack.org/show/786518/ | 01:31 |
| flwang | brtknr: yep, we need to do that | 01:31 |
| brtknr | yep i can confirm its working on our pre-prod | 01:31 |
| flwang | can you please show me the image info | 01:32 |
| brtknr | flwang: http://paste.openstack.org/show/786519/ | 01:32 |
| flwang | i'd like to see 'atomic images list' | 01:33 |
| brtknr | https://seashells.io/p/ccExhNm3 | 01:33 |
| flwang | brtknr: is it stein? | 01:33 |
| brtknr | https://seashells.io/p/x4JcZW8R | 01:34 |
| brtknr | flwang: no master | 01:34 |
| brtknr | i mean, train | 01:34 |
| brtknr | wait, one of them is master, another is train 9.1.0 | 01:34 |
| brtknr | master is devstack | 01:35 |
| brtknr | 9.1.0 is our pre-rpod | 01:35 |
| brtknr | 9.1.0 is our pre-prod | 01:35 |
| brtknr | if you want to look at what vars I used, https://github.com/stackhpc/magnum-autoscale | 01:35 |
| brtknr | https://seashells.io/p/ZAcemWy3 | 01:36 |
| brtknr | and my terraform.tfvars: https://seashells.io/p/ZAcemWy3 | 01:36 |
| flwang | ok, cool | 01:36 |
| brtknr | g2g to sleep, goodnight | 01:36 |
| flwang | my testing is based on stein | 01:36 |
| flwang | have a good night | 01:36 |
| flwang | thank you for your hardwork | 01:37 |
| brtknr | np, we need this too :) | 01:37 |
| brtknr | as do others, thanks for the reviews | 01:37 |
| flwang | anytime, sir | 01:40 |
| flwang | brtknr: i found the root cause | 02:19 |
| *** ricolin has joined #openstack-containers | 02:50 | |
| *** udesale has joined #openstack-containers | 03:41 | |
| *** dave-mccowan has quit IRC | 04:58 | |
| *** namrata has joined #openstack-containers | 05:12 | |
| *** ramishra has joined #openstack-containers | 05:49 | |
| *** janki has joined #openstack-containers | 06:31 | |
| *** pritesh has joined #openstack-containers | 06:32 | |
| *** elenalindq has joined #openstack-containers | 06:44 | |
| *** namrata has quit IRC | 07:02 | |
| *** pritesh has quit IRC | 07:08 | |
| *** openstackgerrit has joined #openstack-containers | 07:23 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum stable/stein: k8s_fedora: Label master nodes with kubectl https://review.opendev.org/695632 | 07:23 |
| brtknr | flwang, ah excellent! | 07:37 |
| *** janki has quit IRC | 07:52 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/train: Fix cert_manager_api with x509keypair https://review.opendev.org/695638 | 08:11 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/train: Support TimeoutStartSec for etcd and heat agent systemd services https://review.opendev.org/695639 | 08:12 |
| *** pcaruana has joined #openstack-containers | 08:21 | |
| *** namrata has joined #openstack-containers | 08:59 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add prometheus-adapter https://review.opendev.org/691646 | 09:07 |
| *** dim79 has joined #openstack-containers | 09:25 | |
| *** ricolin has quit IRC | 09:43 | |
| openstackgerrit | Merged openstack/magnum stable/stein: k8s_fedora: Label master nodes with kubectl https://review.opendev.org/695632 | 09:50 |
| openstackgerrit | Merged openstack/magnum master: bug: Use configured heat-container-agent tag https://review.opendev.org/695196 | 09:53 |
| namrata | Hi Folks,When we spin up a magnum Kubernetes cluster.We see all the pods in pending state http://paste.openstack.org/show/786532/ as which means that the cluster has been unable to select a minion node for them After removing NoSchedule taint from the minion nodes manually we get it working The solution we got online for pods in pending | 09:53 |
| namrata | https://ask.openstack.org/en/question/120442/magnum-kubernetes-noschedule-taint/ is to add [trust]cluster_user_trust = true to magnum.conf but there is an OSA variable which sets magnum_cluster_user_trust to true. And this variable is set to true by default | 09:53 |
| namrata | https://opendev.org/openstack/openstack-ansible-os_magnum/commit/aceaa3e5f174af1e40060440c025f49cd1875e0b and we are not overriding it.Can I get help how to get it working without manually untainting nodes. | 09:53 |
| openstackgerrit | Merged openstack/magnum stable/train: Support TimeoutStartSec for etcd and heat agent systemd services https://review.opendev.org/695639 | 09:56 |
| namrata | Note: and we are running on Rocky | 09:57 |
| *** namrata has quit IRC | 10:12 | |
| brtknr | which version namrata? | 10:17 |
| brtknr | of kubernetes | 10:17 |
| *** namrata has joined #openstack-containers | 10:26 | |
| elenalindq | brtknr: since namrata was out, we use 1.14.8 1.15.5 , both patched for the "billion laughs" vulnerability | 10:29 |
| brtknr | elenalindq: did you try 1.13.12? | 10:31 |
| namrata | brtknr I have tried 1.13.12 also and I see the same issue | 10:31 |
| elenalindq | brtknr: same here | 10:31 |
| brtknr | 1.15.x will definitely not work on rocky | 10:31 |
| brtknr | but 1.14.x should | 10:32 |
| namrata | okay but I have already tried with 1.14.0 and 1.10.11 also I see same issue | 10:34 |
| openstackgerrit | Merged openstack/magnum master: Release k8s v1.14.9 and v1.15.6 https://review.opendev.org/695602 | 10:47 |
| openstackgerrit | Thomas Goirand proposed openstack/magnum master: Make it possible to use uwsgi easily https://review.opendev.org/659764 | 11:05 |
| *** yolanda__ has joined #openstack-containers | 11:08 | |
| *** yolanda has quit IRC | 11:09 | |
| *** udesale has quit IRC | 11:10 | |
| namrata | brtknr sorry i used kube_tag v1.10.11-1 and I see running pods http://paste.openstack.org/show/786545/ | 11:14 |
| namrata | brtknr for reference when I describe pods for kube_tag 1.10.11-1 http://paste.openstack.org/show/786547/ and for kube_tag 1.13.12 http://paste.openstack.org/show/786548/ | 11:19 |
| brtknr | what pod are you describing? | 11:20 |
| namrata | brtknr coredns | 11:21 |
| namrata | for v1.10.11-1 http://paste.openstack.org/show/786550/ for v1.13.12 http://paste.openstack.org/show/786549/ | 11:22 |
| *** namrata has quit IRC | 11:31 | |
| *** yolanda__ has quit IRC | 11:33 | |
| *** namrata has joined #openstack-containers | 11:45 | |
| *** yolanda has joined #openstack-containers | 11:46 | |
| *** yolanda has quit IRC | 11:52 | |
| *** rcernin has quit IRC | 12:01 | |
| *** yolanda has joined #openstack-containers | 12:05 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/694032 | 12:09 |
| openstackgerrit | Merged openstack/magnum master: Fix if condition to test for var==true https://review.opendev.org/671744 | 12:50 |
| brtknr | namrata: it is difficult to know what the issue is looking at those logs | 13:15 |
| *** goldyfruit_ has joined #openstack-containers | 13:20 | |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Scrape prometheus metrics from nginx https://review.opendev.org/695681 | 13:34 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Scrape prometheus metrics from nginx https://review.opendev.org/695681 | 13:37 |
| *** mgoddard has quit IRC | 13:44 | |
| *** mgoddard has joined #openstack-containers | 13:57 | |
| openstackgerrit | Merged openstack/magnum stable/stein: k8s_fedora_atomic: Add PodSecurityPolicy https://review.opendev.org/694032 | 14:03 |
| namrata | brtknr what more information is needed I can provide the same. Thanks for your help | 14:33 |
| brtknr | what is the output of kubectl get pods -A | 14:41 |
| brtknr | and the kube_tag | 14:41 |
| brtknr | of -A doesnt work, --all-namespaces | 14:41 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s: stop introspecting instance name https://review.opendev.org/695709 | 15:22 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s: stop introspecting instance name https://review.opendev.org/695709 | 15:25 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: Using vxlan as default value for flannel_backend https://review.opendev.org/695713 | 15:48 |
| *** bline has quit IRC | 16:11 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: k8s: stop introspecting instance name https://review.opendev.org/695709 | 16:26 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/stein: Using vxlan as default value for flannel_backend https://review.opendev.org/695713 | 16:26 |
| *** bline has joined #openstack-containers | 16:30 | |
| *** ricolin has joined #openstack-containers | 16:32 | |
| *** elenalindq has quit IRC | 16:35 | |
| *** jmlowe has joined #openstack-containers | 16:41 | |
| *** yolanda has quit IRC | 16:50 | |
| *** dim79 has quit IRC | 16:50 | |
| *** elenalindq has joined #openstack-containers | 16:56 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Only query Cinder API if volume size > 0 https://review.opendev.org/695239 | 16:59 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: bug: Only query Cinder API if volume size > 0 https://review.opendev.org/695239 | 17:01 |
| *** jmlowe has quit IRC | 17:10 | |
| *** ricolin has quit IRC | 17:21 | |
| *** mgariepy has quit IRC | 18:07 | |
| *** mgariepy has joined #openstack-containers | 18:10 | |
| *** bline has quit IRC | 18:15 | |
| *** irclogbot_2 has quit IRC | 18:27 | |
| *** spiette has quit IRC | 18:28 | |
| *** elenalindq has quit IRC | 18:29 | |
| *** irclogbot_3 has joined #openstack-containers | 18:30 | |
| *** spiette has joined #openstack-containers | 18:30 | |
| brtknr | flwang: not sure why the cinder patch is failing | 18:46 |
| brtknr | it passed before | 18:46 |
| *** elenalindq has joined #openstack-containers | 19:47 | |
| *** yolanda has joined #openstack-containers | 20:10 | |
| *** elenalindq has quit IRC | 20:45 | |
| *** rcernin has joined #openstack-containers | 20:50 | |
| *** elenalindq has joined #openstack-containers | 20:51 | |
| *** namrata has quit IRC | 21:04 | |
| *** elenalindq has quit IRC | 21:29 | |
| *** mrdoug has joined #openstack-containers | 22:21 | |
| *** rcernin has quit IRC | 22:24 | |
| *** KeithMnemonic has quit IRC | 22:44 | |
| *** jmlowe has joined #openstack-containers | 23:15 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!