09:01:09 <flwang1> #startmeeting magnum 09:01:10 <openstack> Meeting started Wed Oct 23 09:01:09 2019 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:01:13 <openstack> The meeting name has been set to 'magnum' 09:01:21 <flwang1> #topic roll call 09:01:59 <flwang1> o/ 09:02:13 <strigazi> o/ 09:02:43 <brtknr> o/ 09:03:56 <flwang1> let's stop typing on etherpad for now 09:04:02 <flwang1> and focus here? 09:04:16 <flwang1> #topic updates 09:04:28 <flwang1> as i mentioned in the ehterpad 09:04:43 <flwang1> the ignition patch has been merged and cherrypicked to train 09:05:11 <flwang1> we may need a tiny patch to drop the copy local-data to cfn-init-data 09:06:01 <flwang1> strigazi: brtknr: ^ 09:06:26 <strigazi> I don't know if we need it 09:06:42 <flwang1> strigazi: brtknr: i think we should focus on irc 09:06:46 <flwang1> strigazi: why? 09:06:55 <strigazi> flwang1: why we need it 09:06:56 <strigazi> ? 09:07:08 <brtknr> if its copying, it will behave the same as before? 09:07:26 <brtknr> does the heat patch prioritise local-data? 09:07:49 <flwang1> strigazi: see https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/user_data.json#L80 09:08:09 <flwang1> you added ExecStartPre=-mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data 09:08:23 <flwang1> not copy, it's a mv 09:08:36 <brtknr> doh! its mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data, not cp! 09:08:50 <brtknr> either we should change it to cp.... 09:08:59 <flwang1> we should drop it 09:09:00 <brtknr> because heat will have already cut a release 09:09:17 <brtknr> when magnum cuts a release, it will break compatibility 09:09:22 <flwang1> ah, i see your point 09:09:30 <flwang1> that's shit 09:09:48 <brtknr> yeah :( 09:10:21 <flwang1> anyway, we can keep it maybe 09:10:34 <brtknr> thats assuming heat has cut a release 09:10:35 <strigazi> ExecStartPre=- means fail safely 09:10:39 <brtknr> we should check with them 09:10:51 <flwang1> strigazi: that's good 09:10:56 <strigazi> why we need it? 09:11:12 <flwang1> strigazi: you're confusing me again, why we need what? 09:11:18 <strigazi> I did it this way because I was expecting this 09:11:32 <strigazi> We don't need to drop the mv 09:11:51 <flwang1> because it's like a patching 09:11:58 <flwang1> not a solution 09:12:34 <strigazi> ;et 09:12:51 <flwang1> but as brtknr said, heat have already released a version, so we can keep it 09:13:28 <flwang1> don't you think it's ugly to have a mv like this? 09:13:28 <strigazi> we can drop it, the agent will work unless someone has a broken heat. we can keep it, the agent will work with a fixed or broken heat 09:13:48 <flwang1> yep, that's why i think now we can live with that 09:14:09 <flwang1> after I realized that heat have already released 09:14:19 <flwang1> clear now? 09:15:02 <strigazi> yes. That's why I said, "why we need it?" 09:15:27 <strigazi> move on, we keep it. 09:15:27 <flwang1> ok, fail enough 09:15:30 <brtknr> lol... 09:16:29 <strigazi> flwang1: let's move on? 09:16:44 <flwang1> as for fa29 support, personally i would say support it until we think fa30 is stable 09:17:15 <brtknr> #topic FA29 support 09:17:26 <flwang1> #topic fa29 support 09:17:32 <brtknr> :( I dont have the wizardry 09:17:53 <flwang1> brtknr: i can sell it to you if you can buy me a beer 09:18:11 <brtknr> seems like a good trade 09:18:27 <flwang1> strigazi: how do you think？ 09:18:35 <brtknr> flwang1: I am happy with that proposal 09:19:03 <brtknr> or when FA31 enters development? 09:20:06 <flwang1> brtknr: either works i think, and i think it won't take too long, because the coreos driver is sharing most of the atomic driver 09:20:44 <brtknr> works for me 09:21:52 <flwang1> next one? 09:21:54 <brtknr> I would prefer is use_podman=true 09:22:18 <strigazi> brtknr: no, default behaviour is better to not change. 09:22:19 <brtknr> is that doable? or would it cause issues for upgrade? 09:22:30 <brtknr> okay fine 09:22:55 <strigazi> user creates a cluster in stein with a CT. In train they should get the same. makes sense? 09:22:56 <flwang1> strigazi: i would like to understand why didn't you introduce this label at the first time? 09:23:12 <strigazi> flwang1: because I got complains later :) 09:23:26 <flwang1> personally, like this 09:23:58 <flwang1> as i mentioned in the comments, it gives us more time to get the confidence to migrate to fedora coreos 09:24:42 <brtknr> sure, I'm happy with it 09:24:47 <flwang1> the only thing i'd like to propose is, just skip this label for fedora coreos 09:24:49 <brtknr> but needed some convincing 09:25:20 <brtknr> maybe we should drop podman support for atomic altogether? 09:25:30 <brtknr> so its a clean separation 09:25:43 <flwang1> it's a bit late at this moment, i would say 09:26:07 <brtknr> its a lot of extra code for a feature with a short shelf life 09:26:20 <flwang1> we can have a separate, fully isolated code, as long as we deprecate the fedora atomic driver 09:26:57 <flwang1> are you suggesting only for master? 09:26:58 <brtknr> we just need to drop the label... default to using podman for coreos and atomic for fa29 09:27:09 <brtknr> with the same driver 09:27:21 <strigazi> give me a moment 09:28:17 <strigazi> I didn't mention this in the commit message unfortunately, but i mentioned it in a meeting. 09:28:22 <brtknr> so USE_PODMAN is an internal label rather than something that the user specifies 09:28:29 <strigazi> brtknr: please waot 09:28:31 <strigazi> brtknr: please wait 09:28:58 <strigazi> kubelet in an atomic container, doesn't work 09:29:10 <strigazi> that is why we have podman in the atomic driver. 09:29:29 <flwang1> iirc, that's for v1.16.x, right? 09:29:54 <strigazi> If you don't want to have fedora atomic and and >= 1.16 we can drop it 09:30:32 <strigazi> brtknr: flwang1 what do you prefer? 09:30:49 <flwang1> strigazi: i'm ok for have it for fedora atomic driver 09:30:54 <strigazi> brtknr: ? 09:31:07 <strigazi> you have some concerns 09:31:19 <flwang1> but i prefer to use true for fedora coreos 09:31:28 <flwang1> instead of letting user to set it for fedora coreos 09:31:42 <strigazi> flwang1: https://review.opendev.org/#/c/690053/2/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml@775 09:32:21 <strigazi> flwang1: https://review.opendev.org/#/c/690053/2/doc/source/user/index.rst@1385 09:32:43 <brtknr> hmm let me thing 09:34:03 <brtknr> i think by the time we start deploying train release to customer sites, which is at least 3 months away, fcos30 should be stable 09:34:28 <flwang1> strigazi: what happened if i set use_podman=false for driver=fedora_coreos? 09:34:33 <brtknr> we dont need the use_podman label, if flwang1 needs it, lets feep it 09:34:40 <strigazi> we also have users, so don't forget us too :) 09:34:54 <strigazi> flwang1: brtknr creation will fail 09:34:56 <strigazi> with: 09:35:24 <brtknr> strigazi: yes you too! if your users need it! 09:35:35 <flwang1> yep, but i don't want to see that kind of "surprise" 09:35:55 <flwang1> wait 09:36:05 <strigazi> flwang1: | status_reason | ERROR: Parameter 'use_podman' is invalid: "false" is not an allowed value [true] | 09:36:27 <strigazi> it is pretty clear and prertty fast 09:36:49 <strigazi> http://paste.openstack.org/show/785502/ 09:36:51 <flwang1> unless i set use_podman=false for a fedora_coreos template 09:36:58 <flwang1> ok, i'm good then 09:37:55 <strigazi> brtknr: thoughts? 09:38:21 <brtknr> is it better to call it podman_enabled to keep with the convention 09:38:48 <flwang1> brtknr: good point? 09:38:53 <flwang1> good point 09:38:54 <strigazi> brtknr: anything else than naming? 09:39:01 <brtknr> nope 09:39:08 <strigazi> are you sure? 09:39:13 <flwang1> hah 09:39:23 <flwang1> strigazi: don't be pushy for our lovely brtknr 09:39:27 <brtknr> i am outnumberd 09:40:02 <strigazi> I mean, I might miss something 09:40:17 <brtknr> we need to document it 09:40:31 <strigazi> https://review.opendev.org/#/c/690053/2/doc/source/user/index.rst 09:40:51 <brtknr> explain in the doc that we might need use_podman in atomic to support 1.16.x+ 09:40:59 <brtknr> explain in the doc that we need use_podman in atomic to support 1.16.x+ 09:41:24 <brtknr> i will add this to review 09:42:04 <brtknr> i need to look at the code again 09:42:15 <brtknr> i was off yesterday so havent had a chance to review 09:43:13 <flwang1> let's move? 09:43:19 <flwang1> we have 17 mins left 09:43:42 <flwang1> heat-agent: Check if scripts exists https://review.opendev.org/#/c/689704/ 09:43:47 <flwang1> i have +2ed 09:44:48 <brtknr> I will review today 09:45:08 <brtknr> But happy with it in principle 09:45:48 <brtknr> Should address the weird behaviour during reboot in heat-contaner-agent log 09:46:01 <flwang1> brtknr: yep 09:46:07 <flwang1> i notice that recently as well 09:46:29 <strigazi> weird behaviour during reboot in heat-contaner-agent log 09:46:35 <strigazi> please elaborate 09:47:28 <flwang1> my case is not reboot, i saw heat-container-agent stuck several times 09:47:30 <brtknr> strigazi: once the heat-container-agent finishes its thing, it starts writing: Oct 22 20:48:26 k8s-dlgmc7wmwxlo-master-0.novalocal podman[1492]: /var/lib/os-collect-config/local-data not found. Skipping to journald 09:47:33 <flwang1> not useful log 09:47:35 <flwang1> no 09:47:47 <brtknr> but after reboot, it would complain about a missing file 09:47:50 <strigazi> brtknr: I mean about reboot 09:47:57 <strigazi> ok 09:48:14 <brtknr> just rebooting the node in the dev environment now 09:48:15 <strigazi> which file? I think my patch fixes it 09:48:42 <brtknr> strigazi: one sec, just waiting for the reboot to complete 09:50:01 <strigazi> https://github.com/openstack/magnum/commit/3674b3617a770bd71d09e23137ff96f90eb1241a My mistake, it wasn't verbose enough. Using the atomic cli to install kubelet breaks mount 09:50:06 <strigazi> propagation of secrets, configmaps and so on. Using podman 09:50:10 <strigazi> in a systemd unit works. 09:50:15 <brtknr> strigazi: why do you strongly dislike PODMAN_ENABLED btw? 09:50:31 <brtknr> i am not that fussed 09:50:59 <strigazi> it doesn't enable anything. But maybe from the user/ops pov it is better 09:52:29 <strigazi> logs? 09:52:30 <flwang1> strigazi: that's a valid point 09:52:42 <flwang1> generally, we use XXX_enabled only for addons 09:52:45 <brtknr> look at the end: https://seashells.io/p/WseYPMUE 09:52:47 <flwang1> not for this kind of case 09:53:23 <brtknr> lets leave it as USE_PODMAN then 09:53:44 <strigazi> brtknr: after the stop? 09:54:07 <strigazi> brtknr: which line? 09:54:16 <brtknr> Oct 23 09:50:29 09:54:41 <strigazi> brtknr: my patch fixes this 09:54:44 <flwang1> strigazi: brtknr: can we discuss details offline? 09:54:56 <brtknr> strigazi: yes, i already ack this 09:55:07 <strigazi> flwang1: 09:55:09 <strigazi> flwang1: ok 09:55:17 <flwang1> sorry, guys 09:55:20 <strigazi> anything last comments? 09:55:28 <flwang1> it's late here 09:55:45 <brtknr> okay, there's the issue with octavia and master lb fip 09:55:47 <strigazi> oh, have a look at the three patches from Theodoros I added in the etherpad 09:55:50 <brtknr> its working for flwang 09:55:54 <brtknr> not for me 09:56:10 <brtknr> could you please test this strigazi 09:56:15 <flwang1> strigazi: i will leave that for brtknr 09:57:41 <strigazi> one final thing from me 09:58:30 <strigazi> We need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros: 09:58:35 <strigazi> https://review.opendev.org/#/c/687879/ 09:58:35 <strigazi> https://review.opendev.org/#/c/688400/ 09:58:35 <strigazi> https://review.opendev.org/#/c/688346/ 09:58:47 <strigazi> in the train branch 09:58:53 <flwang1> CERN is trying to upgrade to train ? 09:58:55 <strigazi> and release 9.1.0 09:59:00 <strigazi> yes 09:59:06 <flwang1> ok, i see then 09:59:28 <flwang1> to use the v1.16.x and later, right? 09:59:36 <strigazi> yes 09:59:44 <strigazi> 17 in two months 09:59:48 <flwang1> same here 10:00:04 <flwang1> catalyst is on stable/stein 10:00:11 <flwang1> with several private patches 10:00:11 <strigazi> we as well 10:00:18 <flwang1> cherrypicked from master 10:00:21 <strigazi> flwang1: how many? 10:00:29 <flwang1> more than 10 10:00:37 <strigazi> 48 here 10:00:39 <flwang1> from master(train) 10:00:43 <strigazi> who wins? 10:00:48 <flwang1> you win 10:00:54 <strigazi> xD 10:00:56 <flwang1> my friend, so you're keen than me 10:01:13 <strigazi> CherryPickaaS 10:01:18 <flwang1> ok, let's wrap up this meeting 10:01:21 <brtknr> :D 10:01:41 <flwang1> anything else? 10:01:42 <brtknr> we have only started upgrading to stein 10:02:02 <strigazi> I'm good, before closing 10:02:14 <strigazi> +1 or -1 to: We need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros: 10:02:21 <strigazi> brtknr: flwang1 ^^ 10:02:28 <strigazi> and then 9.1.0 10:02:30 <strigazi> brtknr: flwang1 ^^ 10:02:31 <flwang1> strigazi: buy me a beer 10:02:47 <strigazi> sure thing? IPA? 10:02:51 <strigazi> sure thing, IPA? 10:03:16 <flwang1> hopefully i can join the next openstack summit 10:03:22 <flwang1> do you know where is it? 10:03:24 <flwang1> after shanghai 10:03:37 <strigazi> this is a good local one https://www.magictomato.ch/fr/geneve/item_details/531-biere-pale-ale-tangente-brasserie-du-virage 10:03:56 <strigazi> flwang1: I think Vancouver, but not sure 10:04:07 <flwang1> the logo looks cool 10:04:18 <flwang1> if it's Vancouver, i want to go 10:04:27 <strigazi> brtknr: flwang1 agree with the plan for 9.1.0? 10:04:28 <flwang1> if it's EU, no go 10:04:34 <brtknr> strigazi: sounds good 10:04:36 <flwang1> strigazi: works for me 10:04:40 <flwang1> i have to go 10:04:41 <strigazi> excellent 10:04:55 <flwang1> have fun, guys 10:04:59 <strigazi> good night, get some rest 10:05:04 <flwang1> o/ 10:05:18 <brtknr> good nite! 10:05:53 <strigazi> #endmeeting