Wednesday, 2019-12-18

*** dave-mccowan has joined #openstack-containers00:19
*** xinliang has joined #openstack-containers00:31
*** nwonknu has quit IRC00:39
*** nwonknu has joined #openstack-containers00:40
*** dave-mccowan has quit IRC01:16
*** dave-mccowan has joined #openstack-containers01:51
openstackgerritSimon Merrick proposed openstack/magnum-ui master: Improve cluster launch workflow  https://review.opendev.org/69861602:43
*** gouthamr has quit IRC03:14
*** gouthamr has joined #openstack-containers03:24
*** xinliang has quit IRC03:29
*** ykarel|away has joined #openstack-containers03:46
*** goldyfruit has quit IRC04:16
*** ykarel|away has quit IRC04:18
*** ykarel|away has joined #openstack-containers04:35
*** dave-mccowan has quit IRC04:46
*** ykarel|away is now known as ykarel05:00
*** zainub_wahid has joined #openstack-containers05:05
*** ramishra has joined #openstack-containers05:23
*** pcaruana has joined #openstack-containers05:24
*** ricolin has joined #openstack-containers05:29
openstackgerritFeilong Wang proposed openstack/magnum master: Fix proxy issue for k8s fedora drivers  https://review.opendev.org/69835305:59
*** elenalindq has joined #openstack-containers06:19
*** xinliang has joined #openstack-containers06:32
osmanlicilegimorning06:34
*** janki has joined #openstack-containers06:48
*** rcernin has quit IRC06:56
zainub_wahidkolla-ansible (Train) supports Fedora CoreOS or Fedora 30?07:07
*** lpetrut has joined #openstack-containers07:26
*** elenalindq has quit IRC07:32
*** ykarel is now known as ykarel|lunch07:47
*** elenalindq has joined #openstack-containers08:05
*** vesper11 has quit IRC08:09
*** ricolin has quit IRC08:11
*** flwang1 has joined #openstack-containers08:13
*** vesper11 has joined #openstack-containers08:14
*** ricolin has joined #openstack-containers08:18
flwang1brtknr: strigazi: meeting in 10 mins?08:51
strigazio/08:56
flwang1strigazi: hey, how are you08:58
strigaziis brtknr here?08:59
strigaziflwang1: all good08:59
flwang1not sure if brtknr is around08:59
strigaziflwang1: you asked this question? Support multi AZ k8s cluster09:00
brtknro.09:01
brtknro/09:01
flwang1strigazi: yep, does cern like it?09:01
flwang1#startmeeting magnum09:01
openstackMeeting started Wed Dec 18 09:01:26 2019 UTC and is due to finish in 60 minutes.  The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot.09:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.09:01
*** openstack changes topic to " (Meeting topic: magnum)"09:01
openstackThe meeting name has been set to 'magnum'09:01
strigaziflwang1: we have implemented 12 months ago.......09:01
flwang1#topic roll call09:01
*** openstack changes topic to "roll call (Meeting topic: magnum)"09:01
strigazior more09:01
strigazio/09:01
flwang1o/09:02
brtknro/09:02
brtknrdtomasgu:09:02
flwang1#topic multi az support09:02
*** openstack changes topic to "multi az support (Meeting topic: magnum)"09:02
flwang1let's continue the az topic ten09:02
flwang1then09:03
flwang1strigazi: could you please introduce your work? is it upstreamable?09:03
strigaziI'm sorry, we did it upstream two years ago https://review.opendev.org/#/c/527663/09:03
strigaziwe did nodegroups for having an easy way to use this labels.09:04
strigaziwe did nodegroups for having an easy way to use this label.09:04
flwang1i think we are talking different multi az features09:05
strigaziin nova there is onlt --availability-zone09:05
flwang1strigazi: do you mean we can deploy different node to different AZ for one cluster?09:05
strigaziyes.09:05
flwang1how?09:06
strigazithis was the main reason we implemneted nodegroups09:06
strigaziopenstack coe nodegroup create --labels availability_zone=the-other-az09:07
flwang1you mean when create new NG, user need to specify the AZ so as to create the NEW ng to a different AZ?09:07
strigaziyes09:07
flwang1it makes me confused, since i don't remember we have a label named availability_zone09:09
strigazihttps://docs.openstack.org/magnum/latest/user/#availability-zone09:09
brtknr:D magnum labels is a box full of hidden treasures09:10
strigazithe docs are not up to date.09:10
strigaziwhy did you think we implemneted NGs?09:10
brtknrstrigazi: GPU flavor nodes? :P09:10
strigaziI think we discussed in the channel moving that labels as a field.09:10
flwang1ok, that's cool09:11
strigazibrtknr: this is the 3% of the use case09:11
flwang1i will test this later09:11
strigazi97% is AZ09:11
flwang1strigazi: did you try this one?09:11
strigaziwe are not in train yet09:11
flwang1so you can't test it atm?09:12
brtknrhow do you simulate multiple AZ on devstack?09:12
flwang1catalyst cloud are not using multi az but some of our private cloud customers are using multi az09:12
flwang1brtknr: i think you have to have at least two 2 nodes09:13
flwang1then create 2 host aggr09:13
flwang1and then 2 az09:13
flwang1i haven't tested it before with devstack09:13
flwang1strigazi: this is a fantastic one, i'm very happy to see it's cover by NG09:14
flwang1you saved my time :)09:14
brtknrhmm okay, i think this one is resolved for now :) lets move to another topic, we have 45 mins left09:14
flwang1#topic fc driver on master09:14
*** openstack changes topic to "fc driver on master (Meeting topic: magnum)"09:14
flwang1based on my testing, i think the current fcos driver is broken09:15
brtknrflwang1: is it working for you on 9.1.0?09:15
flwang1i don't have time to try, i had 5 meetings today :(09:15
brtknrflwang1: also its broken for a  different reason on fcos 3109:15
flwang16, includes this one09:15
flwang1fcos 31 is a broken image, don't try it09:16
brtknroh! how do you mean?09:16
brtknrits been released as "stable"09:16
flwang1https://github.com/coreos/fedora-coreos-tracker/issues/32709:16
flwang1?09:16
flwang1are we talking the same one?09:17
brtknrno that is the development image09:17
flwang1brtknr: where did you get the stable one?09:17
brtknrhttps://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20191210.3.0/x86_64/fedora-coreos-31.20191210.3.0-ostree.x86_64.tar09:17
brtknrthis is the one they say is stable: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/31.20191210.3.0/x86_64/fedora-coreos-31.20191210.3.0-openstack.x86_64.qcow2.xz09:18
brtknrthe second one, not the first09:18
flwang1but from the issue above, i won't try it09:18
flwang1better try next one09:19
flwang1anyway, it's not the main problem we're facing now09:19
flwang1strigazi: i'd like to see your comments09:19
flwang1did you test the fcos driver recently?09:19
strigaziI am completely lost with what you are saying09:19
flwang1strigazi: hah09:19
strigaziI am testing as we speak09:19
brtknrwe are talking about 2 different issues09:19
flwang1in short, master branch doesn't work for me09:20
brtknr1 is that fcos 31 is broken09:20
brtknr2. on master branch, fcos 30 is broken09:20
strigaziI have some ideas about 2.09:21
strigazibut i need to verify09:21
flwang1strigazi: what i have seen is09:21
brtknrstrigazi: so for me, it works until the minions spawn09:21
brtknrbut the network plugins complain about subnet manager09:21
flwang1the os will be restarted when the heat agent run the scripts, before or after the configure_kubernetes_master.sh09:21
brtknrflwang1: did you try with the latest master branch flwang1? that might have been because of installing empty helm chart09:22
brtknrthat got fixed in the metrics_server patch09:23
flwang1brtknr: i did09:23
flwang1and i also tried back to 1 months ago or even older09:23
brtknralso only ussuri-dev hca works, because train-stable doesnt have python binary09:23
flwang1if strigazi have no idea, then we can discuss this offline09:24
flwang1since this may be hard to figure out in this meeting09:24
brtknrflwang1: i also see the instance restarting issue09:24
strigaziI will fix it today. no worries.09:24
flwang1strigazi: hey man09:24
brtknrstrigazi: can you explain what you think the problem is?09:24
flwang1strigazi: what's the problem?09:24
brtknrit will save me banging my head against the wall09:25
strigazidon't know yet, I'll find soon :)09:25
brtknrlol09:25
flwang1brtknr: if strigazi can't fix it today, should we ask him to buy a beer for us when we meet at Vancouver?09:26
brtknrwell, i think we need to hold off 9.2.0 release until we merge the fix09:26
strigazisure thing09:26
brtknrflwang1: strigazi: btw i have to tell you guys, i will be off for 1 month in january as i am expecting to be a daddy09:26
strigazixD excellent news09:26
flwang1strigazi: cool, i have taken the log as a snapshot with my phone09:27
flwang1brtknr: that's great09:27
flwang1brtknr: first baby?09:27
strigazibrtknr when are you off?09:27
flwang1i will be in holiday for next 2 weeks09:27
brtknrs/baby/babies09:27
flwang1so this should be our last meeting of 201909:27
brtknrthey're going to be twins09:27
flwang1brtknr: that's fabulous09:28
brtknroff from ~27 Dec until 1 Feb ish09:28
brtknrI'll be working hard over xmas :P09:28
flwang1brtknr:  i know twins is very hard09:29
flwang1you're lucky man09:29
brtknr:D thanks, looking forward to the sleep deprivation.09:30
strigazicongrats again Bharat, this is really cool.09:30
strigazishall we end early? I think we can focus in 9.2.009:30
strigaziand latest fedora coreos09:31
flwang1strigazi: did you test v1.17?09:31
flwang1brtknr: ^09:31
brtknrthanks guys09:31
brtknrhttps://review.opendev.org/#/c/69950409:31
strigaziI'm doing them in one shot. fedora core 31, master, v1.17.009:32
flwang1strigazi: ok, good, good luck09:32
brtknrstrigazi: are you using the "stable" image09:32
flwang1strigazi: brtknr: so there is no meeting in the next 2 week09:32
flwang1weeks09:32
flwang1are you ok with that?09:32
flwang1strigazi: what's your xmas plan?09:33
flwang1i will be back on 6 Jan09:33
strigaziyes, i think it will be more.09:33
strigaziI'm back on the 8th not sure ontime for the meeting.09:33
flwang1then let's skip that one as well09:33
flwang1let's have a quick chat on 15th Jan then, ok?09:34
brtknryes :( so the next one is likely to be 15th Jan09:34
flwang1strigazi: ^09:34
brtknrI will try to pop by and say hello09:34
flwang1brtknr: you don't have to, stay with your babies09:34
flwang1and the mom09:34
strigazisure thing09:34
strigazi15th09:34
flwang1cool09:34
flwang1i will send out an email to community09:35
strigazihm, wait I have a presentation that day09:35
flwang1strigazi: oh, yes, i do as well09:35
brtknrI'll  have already spent 2 weeks with the family by that time :) so no problem09:35
strigaziLet's sort it out via email this week09:35
flwang1i will be in Australia for Linux Conf09:35
brtknrSo 22 Jan09:36
strigaziI present some not so exciting news on calico and flannel09:36
flwang1so far 22 Jan works09:36
flwang1let's say 22nd Jan now09:36
strigaziok09:37
flwang1anything else?09:38
strigazithat's it for me09:38
flwang1ok, i'm happy i can sleep earlier today :) and i'm looking forward to seeing strigazi's patch for master :D09:39
brtknrThats all I'd say, please tick this off: https://review.opendev.org/699504 and look forward to your mystery fix for fcos 31 + k8s 1.17.0 strigazi09:39
flwang1hahahahhahahhaha09:39
brtknrthen we can release 9.2.009:39
flwang1strigazi:  i will buy strigazi a beer if he can fix it tomorrow09:40
flwang1his today09:40
strigaziwell one issue is devstack as always...09:40
strigazido we have a story for this?09:40
flwang1(22:24:28) strigazi: I will fix it today. no worries.09:41
flwang1strigazi: no we haven't09:41
strigaziI'll ping you there09:41
brtknrstrigazi: btw what happened to the cleanup script?09:42
strigazi?09:42
strigazireviews?09:42
strigazihaven't finished it, I'm close09:43
strigazilet'09:43
strigazilet's end the meeting? I will ping you09:43
brtknrok  thanks09:43
*** ykarel|lunch is now known as ykarel09:44
flwang1#endmeeting09:44
*** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting"09:44
openstackMeeting ended Wed Dec 18 09:44:19 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)09:44
openstackMinutes:        http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-12-18-09.01.html09:44
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-12-18-09.01.txt09:44
openstackLog:            http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-12-18-09.01.log.html09:44
flwang1thank you guys09:44
flwang1i'm leaving, i will watch the logs tomorrow09:44
strigaziflwang1: I will ping in one of email/gerrit/storyboard09:45
flwang1strigazi: i would love to see that09:45
strigaziflwang1: first clue https://storyboard.openstack.org/#!/story/200703109:48
*** xinliang has quit IRC09:58
brtknrstrigazi: is that related to the subnetmanager error?09:59
*** janki has quit IRC10:35
*** sapd1_x has joined #openstack-containers10:38
*** tbarron_ has joined #openstack-containers11:00
*** ykarel is now known as ykarel|afk11:20
*** ykarel|afk has quit IRC11:30
*** ykarel|afk has joined #openstack-containers12:21
*** mgoddard has quit IRC12:31
zainub_wahidI am using kolla-Ansible (Train). trying to deploy 2masters and 2 nodes. it failed. Then i Tried 2 masters and 1 node. It failed too. I master and 2 nodes is working fine.. but multi master is not. any idea? and i am using Fedora 2712:32
*** KeithMnemonic1 has joined #openstack-containers12:44
*** KeithMnemonic has quit IRC12:45
*** mgoddard has joined #openstack-containers12:49
*** mgoddard has quit IRC12:54
*** zainub_wahid has quit IRC12:58
openstackgerritSpyros Trigazis proposed openstack/magnum master: Fix entrypoint for k8s components in podman  https://review.opendev.org/69967413:05
*** udesale has joined #openstack-containers13:11
*** mgoddard has joined #openstack-containers13:19
*** mkuf has joined #openstack-containers13:36
*** mkuf has quit IRC13:36
*** ykarel|afk is now known as ykarel13:36
*** dave-mccowan has joined #openstack-containers13:57
*** goldyfruit has joined #openstack-containers14:01
*** udesale has quit IRC14:02
*** ramishra has quit IRC14:13
openstackgerritBharat Kunwar proposed openstack/magnum master: k8s: enable services before starting them  https://review.opendev.org/69968714:14
openstackgerritBharat Kunwar proposed openstack/magnum master: k8s: enable services before starting them  https://review.opendev.org/69968714:17
openstackgerritSpyros Trigazis proposed openstack/magnum master: Fix entrypoint for k8s components in podman  https://review.opendev.org/69967414:18
*** ykarel is now known as ykarel|away15:11
*** ramishra has joined #openstack-containers15:12
*** udesale has joined #openstack-containers15:17
*** lpetrut has quit IRC15:23
*** udesale has quit IRC15:25
*** udesale has joined #openstack-containers15:25
*** udesale has quit IRC15:56
*** elenalindq has quit IRC16:22
*** alti_17 has joined #openstack-containers16:45
*** alti_17 has quit IRC16:58
openstackgerritBharat Kunwar proposed openstack/magnum master: k8s: enable services before starting them  https://review.opendev.org/69968717:16
openstackgerritBharat Kunwar proposed openstack/magnum master: k8s: Fix RBAC for OCCM v1.17.0  https://review.opendev.org/69973617:16
*** sapd1_x has quit IRC17:23
*** ykarel|away has quit IRC17:26
openstackgerritBharat Kunwar proposed openstack/magnum master: [k8s] Remove indentation in /etc/sysconfig/heat-params  https://review.opendev.org/69904917:26
openstackgerritBharat Kunwar proposed openstack/magnum master: [k8s] Enable services before starting them  https://review.opendev.org/69968717:28
openstackgerritBharat Kunwar proposed openstack/magnum master: [k8s] Fix RBAC for OCCM v1.17.0  https://review.opendev.org/69973617:28
*** yolanda__ is now known as yolanda18:18
*** openstackgerrit has quit IRC18:43
*** openstackgerrit has joined #openstack-containers19:11
openstackgerritMerged openstack/magnum master: [k8s] Remove indentation in /etc/sysconfig/heat-params  https://review.opendev.org/69904919:11
*** mgoddard has quit IRC19:33
*** mgoddard has joined #openstack-containers19:34
*** flwang1 has quit IRC19:52
*** ramishra has quit IRC20:31
openstackgerritBharat Kunwar proposed openstack/magnum stable/train: [k8s] Remove indentation in /etc/sysconfig/heat-params  https://review.opendev.org/69986520:31
*** elenalindq has joined #openstack-containers20:56
flwangbrtknr: ping21:16
brtknrflwang: pong21:19
brtknrflwang: strigazi fixed 1.1721:20
brtknrand turns out there was never any problem with fcos3121:20
flwangbrtknr: did you test it and any weird reboot?21:20
brtknrflwang: Not recently with fcos3121:21
flwangwhere did you get the stable URL of fcos31?21:21
brtknrflwang: The one on the stable build stream21:21
flwangi always go to the website21:22
brtknrhttps://builds.coreos.fedoraproject.org/browser21:22
brtknri dunno tbqh but I wanted to find historical images21:23
brtknrthe website makes it hard to find them21:23
brtknrhiwever that link has everything21:23
flwangok, i will test spyros' fix today and approve it if it works21:26
flwangi never seen weird reboot like this on any other os i used21:26
brtknrflwang: Me neither tbh21:37
flwangi will keep an close eye on this21:41
flwangi don't want to run into this issue after ask our customer move to fcos21:41
flwangbrtknr: are you using multi az in your DC?21:43
flwangwe still need some work to support fully multi AZ though we now have NG21:44
*** elenalindq has quit IRC21:44
flwangwe need to support distribute master nodes to different AZ to get high Availibity21:44
brtknrflwang: no we dont have any multi AZ users or anticipate any demand for it in the near future, although its a nice idea21:45
brtknrdistributed master seems sensible21:46
flwangyep, e.g. you have 3 az, then it would be nice to distribute your 3 masters for each az21:46
brtknrwe should check this out: https://github.com/kubernetes-sigs/cluster-api-provider-openstack21:52
*** pcaruana has quit IRC21:54
brtknrflwang: it would make deployment to different AZ a piece of cake21:58
brtknrflwang: but last commit was 2 months ago so looks like it is losing steam21:59
brtknrflwang: how is your test coming along?22:25
flwangbrtknr: i'm busy on our customer stuff, i will do that this afternoon22:26
*** KeithMnemonic1 has quit IRC22:38
flwangbrtknr: what's teh status of the CSI support?22:52
brtknrflwang: Cinder? I havent had the chance to look at it yet tbg22:53
brtknrtbh22:53
brtknrI wanted to release 9.2.0 first with all the fixes as we need that urgently22:54
flwangbrtknr: ok22:54
flwangstackHPC would like to upgrade to strain?22:54
flwangtrain22:54
brtknrflwang: we already have at 2 sites :)22:54
flwangah, i see22:54
brtknronly magnum though22:54
brtknrto support podman22:54
flwangyep, podman is a pain22:55
flwangi would like to reword it a bit22:55
flwangpod man is not a pain, but the fedora atomic to fedora coreos22:55
brtknrwell, we just started using it... it will take time to iron out the bugs22:55
brtknri personally quite like podman22:57
brtknrespecially the feature to run containers in user spac22:57
brtknrand api parity with docker is nice22:57
brtknrflwang:22:57
brtknrbut i agree with you, transition from fatomic -> fcos is a pain, also exciting though22:58
flwangthe main problem for us is, the atomic is stable, but the fcos is quite new, we need to be verrrrrrrrrry careful to avoid annoying our customer when doing the migration22:59
brtknrflwang: we just need to make sure we test it lots then23:06
brtknri havent seen any reboot issues since using the new image btw23:06
brtknrfcos3123:06
flwangbrtknr: do i have to use https://review.opendev.org/#/c/699687/4 with strigazi's patch ?23:07
flwangi didn't see the restart, but the installation stopped when it's trying to start those systemd services23:08
*** rcernin has joined #openstack-containers23:10
flwangbrtknr: i just got another restart :(23:12
flwangbut with fc3023:12
flwangi will try with fc3123:12
brtknrflwang: umm i havent tried without that patch23:17
brtknrbut i noticed that the restart was happening at that point23:18
brtknrso i made sure all the services are enabled before they were started just in case23:18
flwangbut it doesn't make any sense to restart the os23:18
flwangit shouldn't23:18
flwangbut i think it's still related to the code we have or hyperkube23:23
flwangsince I didn't see that before23:23
*** goldyfruit_ has joined #openstack-containers23:23
*** goldyfruit has quit IRC23:24
flwangdownloaded a new fc31 and will test again23:30
flwangfc31 works for me23:35
brtknrflwang: No restores?23:52
brtknrrestarts?23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!