Wednesday, 2019-11-27

*** ivve has quit IRC00:04
*** strobert1 has quit IRC00:32
*** strobert1 has joined #openstack-containers01:09
*** iokiwi has quit IRC01:39
*** spsurya has joined #openstack-containers01:40
*** iokiwi has joined #openstack-containers01:43
*** ricolin has joined #openstack-containers01:52
*** trungnv has joined #openstack-containers02:52
*** goldyfruit_ has joined #openstack-containers03:02
*** trungnv has quit IRC03:18
*** udesale has joined #openstack-containers03:44
*** goldyfruit_ has quit IRC04:01
*** namrata has joined #openstack-containers04:44
*** threestrands has quit IRC05:51
*** elenalindq has joined #openstack-containers06:16
*** pcaruana has joined #openstack-containers06:34
*** xinliang has joined #openstack-containers06:53
*** lpetrut has joined #openstack-containers07:13
*** flwang1 has joined #openstack-containers08:50
flwang1strigazi: brtknr: meeting ?08:50
flwang1strigazi: brtknr: if you guys are busy today, we can skip this one and go for the virtual PTG tomorrow08:55
*** ivve has joined #openstack-containers08:58
brtknrMorning flwang109:01
flwang1brtknr: morning09:01
flwang1seems strigazi is not around09:01
brtknrI thought we were skipping the meeting today but happy to discuss informally if any pressing mattera09:01
brtknri have added a few things to the virtual ptg agenda09:02
namrataflwang1 brtknr I have an issue to raise should i ask here or if you can tell me how to join the virtual PTG I can ask there09:02
flwang1brtknr: i saw that, thank you for your input09:02
flwang1namrata: the virtual PTG will be used to discuss the roadmap, features or critical bugs, etc09:03
flwang1namrata: that said, you're welcome to ask your question here09:03
namrataHi Folks,I would like to get your attention to an issue we are facing that kube-system pods are in pending status as the nodes are tainted and we also see missing openstack cloud controller pod. The issue is explained here briefly http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011166.html.09:04
brtknrnamrata: nais this releated to the mai?l you sent yesterday09:04
flwang1brtknr: need your review for https://review.opendev.org/#/c/696204/09:04
namratabrtknr yes and our internal discussion had a conclusion that running different services on different regions with different releases is not a good idea09:05
flwang1namrata: what's your node status?09:05
brtknrflwang1: see the discussion in the link namrata shared09:06
namrataflwang1 namrata@namrata-HP-Pavilion-13-Notebook-PC:~$ kubectl get nodesNAME                                 STATUS   ROLES    AGE     VERSIONcluster-test-tzwgj5g6klcc-master-0   Ready    <none>   3h12m   v1.13.12cluster-test-tzwgj5g6klcc-minion-0   Ready    <none>   3h12m   v1.13.1209:06
flwang1brtknr: can you show me 'kubectl get deployments -n kube-system'09:07
brtknrflwang1:  I am not sure if we are plannig to do backports to stabilise rocky release09:07
brtknrflwang1: you mean namrata  :P09:08
flwang1flwang: yep, sorry09:08
elenalindqbrtknr flwang1 We are in a very difficult situation with part of our production down for two weeks now. :( To update Magnum is not an option for us.09:08
flwang1namrata: can you pls run 'kubectl get node -o wide'09:08
namrataand brtknr and flwang1 if i use kube_tag 1.13.12 I see deamonset cloud controller manager http://paste.openstack.org/show/786776/09:08
brtknrnamrata: so it is working?09:08
namratahttp://paste.openstack.org/show/786777/09:09
namratano then too its not working09:09
flwang1namrata: ok, can you run  kubectl patch node ${INSTANCE_NAME} --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}'     on your master nodes?09:10
flwang1replace instance_name with your master node name09:10
*** xinliang has quit IRC09:10
brtknrkubectl patch node `hostname -s` --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}'09:11
brtknrif executing on master node itself ^09:11
namratarunning cloud controller manager pod http://paste.openstack.org/09:11
flwang1now you should be able to get IP for your nodes09:12
brtknrelenalindq: thats a shame, were you given any reason why they dont want to run newer release?09:12
flwang1elenalindq: are you in same company with namrata ?09:13
namratayes we work together09:13
elenalindqbrtknr It means we need to upgrade other openstack components, at least heat, from what I understand. There are other customers hosted in this one region out of seven.09:13
elenalindqflwang1:  yes, me and namrata are colleagues09:14
flwang1ok, good to know09:14
brtknrelenalindq: that is an invalid assumption, magnum train release requires minimum queens heat09:14
flwang1namrata: can you let me know if you can see the ccm is being created now?09:14
brtknrmagnum train release only requires train heat to run fedora coreos09:14
namrataflwang1 checking09:15
namrataflwang1 worked for me on kube_tag 1.13.1209:16
namratabut is there other way without manually patching it09:16
namrataif you suggest any backports needed and help me I can do the same09:16
flwang1namrata: cherry pick this patch https://review.opendev.org/#/c/695632/09:16
brtknrnamrata: you need this backport https://review.opendev.org/#/c/695632/09:16
brtknrflwang1: oops you beat me to it09:17
namratabrtknr flwang1 thanks a lot for your help and time. Highly appreciated09:17
flwang1brtknr: i'm quick typer :)09:17
flwang1namrata: you're welcome, i found it just several days ago09:18
namrataand brtknrĀ  special thanksĀ ! For always answering and replying to the issues09:18
flwang1namrata: both you and elenalindq work for city networks, right?09:19
openstackgerritNamrata proposed openstack/magnum stable/rocky: k8s_fedora: Label master nodes with kubectl  https://review.opendev.org/69625309:19
elenalindqflwang1:  yes, correct09:19
namratayes correct09:19
flwang1elenalindq: cool09:19
flwang1it would be nice if you guys can join magnum community09:20
flwang1so that you can always get the up to date status of this project09:20
namrataI am interested in contributing to the project09:20
flwang1and we would be very happy to see the feedback from your side too09:20
brtknrflwang1: rocky tests are failimg09:21
flwang1namrata: cool09:21
brtknri wonder if we need to disable  for tests09:21
flwang1brtknr: what's the problem?09:21
brtknre.g. https://review.opendev.org/#/c/695371/09:21
flwang1py2?09:21
brtknrfunctional-api09:21
flwang1hmm... seems the devstack failed09:22
brtknrelenalindq: did you see my message that stein magnum is compatible with queens heat09:22
brtknrflwang1: thats it... this test has been rechecked lots of times... i am not sure how we can backport anything to rocky09:23
brtknrsuppose we could make the functional api non-voting09:24
namrataflwang1 is this fix going to work with kube_tag 1.14.8 also where  daemonset.apps/openstack-cloud-controller-manager is mising09:24
flwang1hmm... i will double check it tomorrow. i have no idea at this moment09:24
flwang1namrata: i'm pretty sure it will fix the issue for v.14.8 and v1.15.609:25
namrataokay v.1.5.6 also. Okay great i will test and will update you09:25
brtknrnamrata: from what you shared with me lat time, 1.14.8 did not have a daemonset at all09:25
brtknr1.15.6 will 100% not work on rocky09:25
brtknrflwang1: ^ we only just managed to get 1.15 working on stable/stein09:26
elenalindqbrtknr I did , I really appreciate your feedback and all your help. I cannot comment on that, other than I think upgrade of one openstack component is not a good practice.09:26
flwang1namrata: sorry, you're using rocky?09:27
flwang1then i'm not sure09:27
flwang1v1.15.6 can work on stein09:27
namratabrtknr yes 1.14.8 had no deamonset09:28
namratayes rocky09:28
flwang1namrata: what do you mean "had no daemonset"?09:28
brtknrelenalindq: then i suggest upgrading your openstack :) i understand upgrading single openstack component is not generally a good idea but in the case of magnum, this is not strictly true09:28
openstackgerritFeilong Wang proposed openstack/magnum master: WIP: [k8s] Support docker storage driver for fedora coreos driver  https://review.opendev.org/69625609:29
namrataflwang1 http://paste.openstack.org/show/786729/ this is for v1.14.809:29
brtknrbecause magnum more tries to keep up with kubernetes releases more than openstack releases... hence magnum train supports all the way back to queens heat09:29
flwang1then you should debug why there is no ds for ccm09:29
brtknrwhich is the main magnum dependency09:29
*** openstackstatus has quit IRC09:30
namratads only gets created with one kube_tag version i.e v1.13.1209:30
flwang1namrata: check if this line is called https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/kube-apiserver-to-kubelet-role.sh#L32209:31
flwang1when using kube_tag v1.14.809:31
flwang1i can't stay long, it's 10:30PM here, sorry09:32
flwang1brtknr maybe able to help09:32
namrataokay thanks flwang1 I will test and let you know09:32
namrataokay09:32
brtknrnamrata: try doing kubectl apply -f /srv/magnum/kubernetes/openstack-cloud-controller-manager.yaml on master node and see what it complains about09:32
flwang1brtknr: pls help review https://review.opendev.org/#/c/696204/ when you have time09:33
brtknrflwang1: will do09:33
flwang1we also need to cherrypick it to train09:33
brtknrflwang1: one last tthing09:33
namratabrtknr i will create one cluster of v1.14.8 and test and ping you09:33
brtknrany suggestions about how to get around breaking tests on rocky?09:33
elenalindqbrtknr I will pass on your comments :). Wish I could snap my fingers and upgrade Openstack, if only it was that simple. :)09:33
flwang1brtknr: i will check if the devstack works for other services firstly09:34
flwang1if it works for other services, then we may need to debug the devstack plugin of magnum rocky09:34
flwang1otherwise, we may need to ask help from openstack infra team to fix the devstack for rocky09:34
brtknrflwang1: hmm okay09:35
*** udesale has quit IRC09:35
brtknrflwang1:  i see wht the problem is09:36
brtknrits trying to download fedora atomic 27 image which is now EOL09:36
brtknrthat URL no longer exists09:36
*** udesale has joined #openstack-containers09:37
flwang1brtknr: ok, then it should be easy to fix09:37
flwang1i have to offline now09:38
*** udesale has quit IRC09:39
*** udesale has joined #openstack-containers09:39
*** udesale has quit IRC09:41
*** udesale has joined #openstack-containers09:41
*** udesale has quit IRC09:42
*** udesale has joined #openstack-containers09:42
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: Using Fedora Atomic 29 as default image  https://review.opendev.org/69626209:43
brtknrflwang1: goodnight :)09:44
namratabrtknr after applying  kubectl apply -f /srv/magnum/kubernetes/openstack-cloud-controller-manager.yaml tp the master node it gives a deamonset09:48
namratahttp://paste.openstack.org/show/786783/09:48
brtknrnamrata: now try labelling the master node again?09:53
openstackgerritDiogo Guerra proposed openstack/magnum master: Add prometheus-adapter  https://review.opendev.org/69164609:54
namratabrtknr means kubectl patch node ${INSTANCE_NAME} --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}'     on your master nodes09:54
openstackgerritAndrei Nistor proposed openstack/magnum-ui stable/train: Add missing hidden option to cluster template  https://review.opendev.org/69627009:54
andreinGood morning!09:57
andreinbrtknr: I see the hidden option fix was merged, I've proposed a cherry pick into stable/stein ^^09:58
*** udesale has quit IRC10:03
*** udesale has joined #openstack-containers10:04
*** rcernin has quit IRC10:31
brtknrandrein: yep we move fast :D10:32
namratabrtknr by labeeling do you meant ubectl patch node ${INSTANCE_NAME} --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}'     on your master nodes10:35
brtknrnamrata: thats right10:35
brtknrreplace ${INSTANCE_NAME} with `hostname -s` for ease10:36
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: Using Fedora Atomic 29 as default image  https://review.opendev.org/69626210:38
namrata  brtknr http://paste.openstack.org/show/786786/10:38
namratanodes are tainted thats why other pods are in pending state10:39
brtknrcan you share me the output of journalctl -u heat-container-agent | nc seashells.io 133710:40
brtknrfrom master node10:40
namrataoh I spoke too early10:40
namratahttp://paste.openstack.org/show/786787/10:40
namratabrtknr pod are running10:40
brtknrnamrata: okay good10:40
brtknryou just need that 1 backport then10:40
brtknrare you using fa29?10:41
namratano f2710:41
brtknrok10:41
brtknrhave you tested with fa29?10:41
brtknrby any chance?10:42
namratayes I have tested with fa29 and I get same result10:42
brtknrok10:42
namrataso I need only https://review.opendev.org/#/c/695632/ this backport nothing for v1.14.8 ds not coming up10:42
brtknrcan you share with me the output of heat container agent10:43
brtknrds is not applying for a different reason10:43
brtknrthats what I'm tryign to work out10:43
brtknralso the output of /var/log/cloud-init-output.log10:43
brtknrbest thing to do is `cat /var/log/cloud-init-output.log | nc seashells.io 1337`10:44
brtknrthat i know of :)10:44
andreinbrtknr: is there any list somewhere with the supported OS image/kubernetes versions?10:44
brtknrandrein: such a list quickly goes out of date...10:44
brtknrhence we havent maintained it10:45
brtknrandrein: i tried to make one for kube_tag a little while ago: https://review.opendev.org/#/c/685675/10:45
andreinthat's one of the major issues I've had when starting up with magnum... spent a couple of days trying stuff10:45
brtknrandrein: thats usually the best way really... there are so many configurations and everyone has a different requirement10:46
openstackgerritMerged openstack/magnum-ui stable/train: Add missing hidden option to cluster template  https://review.opendev.org/69627010:46
brtknrandrein: i am not sure if magnum-ui works without cutting a release...10:46
*** rcernin has joined #openstack-containers10:47
andreinbrtknr: How can I pull https://review.opendev.org/#/c/696179/2 into my local checkout?10:48
andreinI've reproduced the bug, I'd like to test the fix10:49
brtknrandrein: git review -d 69617910:49
brtknris that what you mean?10:49
namratabrtknr journalctl -u heat-container-agent | nc seashells.io 1337serving at https://seashells.io/v/TBenhQ6mcat /var/log/cloud-init-output.log | nc seashells.io 1337serving at https://seashells.io/v/pgmBNe2T10:50
andreinbrtknr: yep, that's what I was looking for10:51
brtknrnamrata: i found your problem: error: SchemaError(io.k8s.api.apps.v1beta1.StatefulSetSpec): invalid object doesn't have additional properties10:51
namrataso is this issue with the particular image10:52
brtknrnamrata: can you try with label `heat_container_agent_tag=train-stable`10:52
brtknrwhen you spawn the cluster10:52
namrataokay will try10:53
brtknrand see if the damonset appears10:53
namrataokay10:53
*** sapd1 has joined #openstack-containers10:59
*** udesale has quit IRC11:19
andreinbrtknr: you mentioned something about a release of magnum-ui earlier. who can do that?11:26
brtknrandrein: anyone in there11:29
brtknrandrein: anyone in theory11:29
brtknropenstack/releases repo11:29
brtknrandrein: do you want to cut a new release? 5.2.1?11:31
*** rcernin has quit IRC11:31
andreinooooo, sounds fancy :) I'm in11:31
andreinshall we wait for https://review.opendev.org/#/c/696179/2 first?11:31
brtknrandrein: https://review.opendev.org/#/c/691311/11:32
brtknrandrein: example ^11:33
brtknrandrein: I've just +2ed it11:33
andreinthanks!11:33
openstackgerritAndrei Nistor proposed openstack/magnum-ui stable/train: Set empty default value for docker_storage_driver  https://review.opendev.org/69628911:34
andrein^^ cherry pick11:34
brtknrIt needs -x11:34
brtknrthe (cherry picked from) bit11:34
brtknrandrein:11:34
andreinhmmm11:37
andreinwhat do you mean by -x ?11:37
brtknrgit cherry-pick -x11:39
brtknrfor provenance11:39
*** sapd1 has quit IRC11:39
brtknrandrein:11:39
andreinI just used the cherry-pick button in gerrit.11:40
andreinhow do I fix this?11:40
brtknryou could manually edit the commit message11:40
brtknr(cherry picked from commit 2d2ea7422a6c761b5d8a99e684c5267912afdbe3)11:42
openstackgerritMerged openstack/magnum-ui master: Set empty default value for docker_storage_driver  https://review.opendev.org/69617911:44
brtknrandrein: ^11:51
openstackgerritNamrata proposed openstack/magnum stable/rocky: k8s_fedora: Label master nodes with kubectl  https://review.opendev.org/69625311:51
brtknrnamrata: it would be good if you could cherry-pick with -x flag11:52
namrata so should I do it again11:53
andreinbrtknr: https://review.opendev.org/#/c/696289/ is it OK now?11:54
brtknrnamrata: just edit the commit message, same as adrein11:56
brtknrandrein: you might need to "publish" changes11:59
openstackgerritAndrei Nistor proposed openstack/magnum-ui stable/train: Set empty default value for docker_storage_driver  https://review.opendev.org/69628911:59
brtknrif you are doing via gerrit12:00
andreinah, didn't see the button :)12:00
andreindone now12:00
openstackgerritNamrata proposed openstack/magnum stable/rocky: k8s_fedora: Label master nodes with kubectl  https://review.opendev.org/69625312:09
brtknrandrein: cool, merged, feel free to propose a release12:09
brtknrnamrata: where does commit id 6b0fa1805f954c13d64c2905e7e9f3e4ba33c5ae come from?12:15
brtknrhmm looks like other cherry-picks have it too12:16
brtknrstrange12:16
namratayes12:17
namratai also checked12:17
namrataother commits also have12:17
namratathat12:17
openstackgerritMerged openstack/magnum-ui stable/train: Set empty default value for docker_storage_driver  https://review.opendev.org/69628912:22
andreinnamrata: I think gerrit adds the "cherry picked from" only if the cherry picked commit has already been merged.13:31
andreinbrtknr: I'll start working on the release13:32
andreinbrtknr: is new-release some kind of helper script? https://review.opendev.org/#/c/691311/2/deliverables/train/magnum.yaml@2413:37
*** udesale has joined #openstack-containers13:38
brtknrandrein: yes, in the releases checkout, you need to run something like: ./tools/new_release.sh train magnum-ui bugfix14:00
brtknrnamrata: that commit id must be 9dc92654d269e4d4b78c83d01854306f852e281214:04
brtknralso the CI has been fixed so could you please rebase on top of latest stable/rocky14:04
brtknractually, you could just wait for this change to be merged: https://review.opendev.org/#/c/696262/14:05
brtknreither way, rebase on top of https://review.opendev.org/#/c/696262/14:06
brtknrotherwise your CI job will keep failing14:06
namrata@brtknr after  adding label heat_container_agent_tag=train-stable with v1.14.8 I see DS http://paste.openstack.org/show/786801/14:07
brtknrnamrata: okay thats good, its because an older kubernetes client is running in rocky-stable heat_container_agent which is not compatible with 1.14.814:13
brtknrso we just need that one backport and you're sorted14:13
namrataokay whcih backport14:13
namrata*which14:13
brtknrnamrata: the one you made14:14
namrataokay that one only14:14
namratathanks brtknr14:14
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: Fix proportional autoscaler image  https://review.opendev.org/69537114:15
namrataso brtknr if  we need to use v1.14.8 we need to use label heat_container_agent_tag=train-stable14:18
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora: Label master nodes with kubectl  https://review.opendev.org/69625314:18
brtknrnamrata: thats right14:19
namrataokay thanks14:19
brtknryes, train-stable is best because its also easier to debug14:20
namrataand I see you have committed https://review.opendev.org/#/c/696253/ so no need to rebase now14:22
andreinbrtknr: https://review.opendev.org/#/c/696313/14:34
andreinmagnum-ui 5.2.1 release proposal14:35
brtknrandrein: cool, i can only +1 in that repo :)14:36
brtknrthanks! lgtm14:36
andreinCool, I'll wait for the other reviewers :)14:47
andreinbrtknr: what's the difference between kube_tag and kube_version labels?14:54
*** goldyfruit_ has joined #openstack-containers14:55
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora: Label master nodes with kubectl  https://review.opendev.org/69625315:07
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: k8s: stop introspecting instance name  https://review.opendev.org/69632015:07
brtknrandrein: use kube_tag15:08
brtknri personally do not know what kube_version is for myself15:08
brtknrandrein: it might be an ancient artifact?15:09
andreinI haven't been here that long :) still haven't figured out how the magic comes together yet15:11
*** dioguerra has quit IRC15:17
*** ricolin_ has joined #openstack-containers15:26
*** ricolin has quit IRC15:28
*** ricolin_ is now known as ricolin15:29
*** ricolin has quit IRC15:34
*** ricolin has joined #openstack-containers15:43
namratabrtknr https://review.opendev.org/#/c/696253/ you recent patch set removes instance definition15:53
brtknrnamrata: thats right, I cherry-picked the patch that defines INSTANCE_NAME correctly15:53
brtknrnamrata: this one https://review.opendev.org/#/c/696320/115:54
brtknrnamrata: make sense?15:54
namratayeah got that. Thanks15:55
*** lpetrut has quit IRC16:03
*** ivve has quit IRC16:08
*** namrata has quit IRC16:09
*** udesale has quit IRC16:53
*** udesale has joined #openstack-containers16:55
*** udesale has quit IRC17:06
*** ricolin has quit IRC18:16
*** gouthamr has quit IRC18:20
*** gouthamr has joined #openstack-containers18:21
openstackgerritDiogo Guerra proposed openstack/magnum master: [k8s] Update metrics-server  https://review.opendev.org/63793918:54
*** spsurya has quit IRC19:30
*** flwang1 has quit IRC19:36
*** elenalindq has quit IRC21:45
*** rcernin has joined #openstack-containers21:56
*** pcaruana has quit IRC22:06
*** goldyfruit_ has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!