09:01:09 <flwang1> #startmeeting magnum
09:01:10 <openstack> Meeting started Wed Oct 23 09:01:09 2019 UTC and is due to finish in 60 minutes.  The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:01:13 <openstack> The meeting name has been set to 'magnum'
09:01:21 <flwang1> #topic roll call
09:01:59 <flwang1> o/
09:02:13 <strigazi> o/
09:02:43 <brtknr> o/
09:03:56 <flwang1> let's stop typing on etherpad for now
09:04:02 <flwang1> and focus here?
09:04:16 <flwang1> #topic updates
09:04:28 <flwang1> as i mentioned in the ehterpad
09:04:43 <flwang1> the ignition patch has been merged and cherrypicked to train
09:05:11 <flwang1> we may need a tiny patch to drop the copy  local-data to cfn-init-data
09:06:01 <flwang1> strigazi: brtknr: ^
09:06:26 <strigazi> I don't know if we need it
09:06:42 <flwang1> strigazi: brtknr: i think we should focus on irc
09:06:46 <flwang1> strigazi: why?
09:06:55 <strigazi> flwang1: why we need it
09:06:56 <strigazi> ?
09:07:08 <brtknr> if its copying, it will behave the same as before?
09:07:26 <brtknr> does the heat patch prioritise local-data?
09:07:49 <flwang1> strigazi: see https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/user_data.json#L80
09:08:09 <flwang1> you added   ExecStartPre=-mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data
09:08:23 <flwang1> not copy, it's a mv
09:08:36 <brtknr> doh! its mv /var/lib/os-collect-config/local-data /var/lib/cloud/data/cfn-init-data, not cp!
09:08:50 <brtknr> either we should change it to cp....
09:08:59 <flwang1> we should drop it
09:09:00 <brtknr> because heat will have already cut a release
09:09:17 <brtknr> when magnum cuts a release, it will break compatibility
09:09:22 <flwang1> ah, i see your point
09:09:30 <flwang1> that's shit
09:09:48 <brtknr> yeah :(
09:10:21 <flwang1> anyway, we can keep it maybe
09:10:34 <brtknr> thats assuming heat has cut a release
09:10:35 <strigazi> ExecStartPre=- means fail safely
09:10:39 <brtknr> we should check with them
09:10:51 <flwang1> strigazi: that's good
09:10:56 <strigazi> why we need it?
09:11:12 <flwang1> strigazi: you're confusing me again, why we need what?
09:11:18 <strigazi> I did it this way because I was expecting this
09:11:32 <strigazi> We don't need to drop the mv
09:11:51 <flwang1> because it's like a patching
09:11:58 <flwang1> not a solution
09:12:34 <strigazi> ;et
09:12:51 <flwang1> but as brtknr said, heat have already released a version, so we can keep it
09:13:28 <flwang1> don't you think it's ugly to have a mv like this?
09:13:28 <strigazi> we can drop it, the agent will work unless someone has a broken heat. we can keep it, the agent will work with a fixed or broken heat
09:13:48 <flwang1> yep, that's why i think now we can live with that
09:14:09 <flwang1> after I realized that heat have already released
09:14:19 <flwang1> clear now?
09:15:02 <strigazi> yes. That's why I said, "why we need it?"
09:15:27 <strigazi> move on, we keep it.
09:15:27 <flwang1> ok, fail enough
09:15:30 <brtknr> lol...
09:16:29 <strigazi> flwang1: let's move on?
09:16:44 <flwang1> as for fa29 support, personally i would say support it until we think fa30 is stable
09:17:15 <brtknr> #topic FA29 support
09:17:26 <flwang1> #topic fa29 support
09:17:32 <brtknr> :( I dont have the wizardry
09:17:53 <flwang1> brtknr: i can sell it to you if you can buy me a beer
09:18:11 <brtknr> seems like a good trade
09:18:27 <flwang1> strigazi: how do you think？
09:18:35 <brtknr> flwang1: I am happy with that proposal
09:19:03 <brtknr> or when FA31 enters development?
09:20:06 <flwang1> brtknr: either works i think, and i think it won't take too long, because the coreos driver is sharing most of the atomic driver
09:20:44 <brtknr> works for me
09:21:52 <flwang1> next one?
09:21:54 <brtknr> I would prefer is use_podman=true
09:22:18 <strigazi> brtknr: no, default behaviour is better to not change.
09:22:19 <brtknr> is that doable? or would it cause issues for upgrade?
09:22:30 <brtknr> okay fine
09:22:55 <strigazi> user creates a cluster in stein with a CT. In train they should get the same. makes sense?
09:22:56 <flwang1> strigazi: i would like to understand why didn't you introduce this label at the first time?
09:23:12 <strigazi> flwang1: because I got complains later :)
09:23:26 <flwang1> personally, like this
09:23:58 <flwang1> as i mentioned in the comments, it gives us more time to get the confidence to migrate to fedora coreos
09:24:42 <brtknr> sure, I'm happy with it
09:24:47 <flwang1> the only thing i'd like to propose is, just skip this label for fedora coreos
09:24:49 <brtknr> but needed some convincing
09:25:20 <brtknr> maybe we should drop podman support for atomic altogether?
09:25:30 <brtknr> so its a clean separation
09:25:43 <flwang1> it's a bit late at this moment, i would say
09:26:07 <brtknr> its a lot of extra code for a feature with a short shelf life
09:26:20 <flwang1> we can have a separate, fully isolated code, as long as we deprecate the fedora atomic driver
09:26:57 <flwang1> are you suggesting only for master?
09:26:58 <brtknr> we just need to drop the label... default to using podman for coreos and atomic for fa29
09:27:09 <brtknr> with the same driver
09:27:21 <strigazi> give me a moment
09:28:17 <strigazi> I didn't mention this in the commit message unfortunately, but i mentioned it in a meeting.
09:28:22 <brtknr> so USE_PODMAN is an internal label rather than something that the user specifies
09:28:29 <strigazi> brtknr:  please waot
09:28:31 <strigazi> brtknr:  please wait
09:28:58 <strigazi> kubelet in an atomic container, doesn't work
09:29:10 <strigazi> that is why we have podman in the atomic driver.
09:29:29 <flwang1> iirc, that's for v1.16.x, right?
09:29:54 <strigazi> If you don't want to have fedora atomic and and >= 1.16 we can drop it
09:30:32 <strigazi> brtknr: flwang1 what do you prefer?
09:30:49 <flwang1> strigazi: i'm ok for have it for fedora atomic driver
09:30:54 <strigazi> brtknr: ?
09:31:07 <strigazi> you have some concerns
09:31:19 <flwang1> but i prefer to use true for fedora coreos
09:31:28 <flwang1> instead of letting user to set it for fedora coreos
09:31:42 <strigazi> flwang1: https://review.opendev.org/#/c/690053/2/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml@775
09:32:21 <strigazi> flwang1: https://review.opendev.org/#/c/690053/2/doc/source/user/index.rst@1385
09:32:43 <brtknr> hmm let me thing
09:34:03 <brtknr> i think by the time we start deploying train release to customer sites, which is at least 3 months away, fcos30 should be stable
09:34:28 <flwang1> strigazi: what happened if i set use_podman=false for driver=fedora_coreos?
09:34:33 <brtknr> we dont need the use_podman label, if flwang1 needs it, lets feep it
09:34:40 <strigazi> we also have users, so don't forget us too :)
09:34:54 <strigazi> flwang1: brtknr creation will fail
09:34:56 <strigazi> with:
09:35:24 <brtknr> strigazi: yes you too! if your users need it!
09:35:35 <flwang1> yep, but i don't want to see that kind of "surprise"
09:35:55 <flwang1> wait
09:36:05 <strigazi> flwang1: | status_reason        | ERROR: Parameter 'use_podman' is invalid: "false" is not an allowed value [true] |
09:36:27 <strigazi> it is pretty clear and prertty fast
09:36:49 <strigazi> http://paste.openstack.org/show/785502/
09:36:51 <flwang1> unless i set use_podman=false for a fedora_coreos template
09:36:58 <flwang1> ok, i'm good then
09:37:55 <strigazi> brtknr: thoughts?
09:38:21 <brtknr> is it better to call it podman_enabled to keep with the convention
09:38:48 <flwang1> brtknr: good point?
09:38:53 <flwang1> good point
09:38:54 <strigazi> brtknr: anything else than naming?
09:39:01 <brtknr> nope
09:39:08 <strigazi> are you sure?
09:39:13 <flwang1> hah
09:39:23 <flwang1> strigazi: don't be pushy for our lovely brtknr
09:39:27 <brtknr> i am outnumberd
09:40:02 <strigazi> I mean, I might miss something
09:40:17 <brtknr> we need to document it
09:40:31 <strigazi> https://review.opendev.org/#/c/690053/2/doc/source/user/index.rst
09:40:51 <brtknr> explain in the doc that we might need use_podman in atomic to support 1.16.x+
09:40:59 <brtknr> explain in the doc that we need use_podman in atomic to support 1.16.x+
09:41:24 <brtknr> i will add this to review
09:42:04 <brtknr> i need to look at the code again
09:42:15 <brtknr> i was off yesterday so havent had a chance to review
09:43:13 <flwang1> let's move?
09:43:19 <flwang1> we have 17 mins left
09:43:42 <flwang1> heat-agent: Check if scripts exists https://review.opendev.org/#/c/689704/
09:43:47 <flwang1> i have +2ed
09:44:48 <brtknr> I will review today
09:45:08 <brtknr> But happy with it in principle
09:45:48 <brtknr> Should address the weird behaviour during reboot in heat-contaner-agent log
09:46:01 <flwang1> brtknr: yep
09:46:07 <flwang1> i notice that recently as well
09:46:29 <strigazi> weird behaviour during reboot in heat-contaner-agent log
09:46:35 <strigazi> please elaborate
09:47:28 <flwang1> my case is not reboot, i saw heat-container-agent stuck several times
09:47:30 <brtknr> strigazi: once the heat-container-agent finishes its thing, it starts writing: Oct 22 20:48:26 k8s-dlgmc7wmwxlo-master-0.novalocal podman[1492]: /var/lib/os-collect-config/local-data not found. Skipping to journald
09:47:33 <flwang1> not useful log
09:47:35 <flwang1> no
09:47:47 <brtknr> but after reboot, it would complain about a missing file
09:47:50 <strigazi> brtknr: I mean about reboot
09:47:57 <strigazi> ok
09:48:14 <brtknr> just rebooting the node in the dev environment now
09:48:15 <strigazi> which file? I think my patch fixes it
09:48:42 <brtknr> strigazi: one sec, just waiting for the reboot to complete
09:50:01 <strigazi> https://github.com/openstack/magnum/commit/3674b3617a770bd71d09e23137ff96f90eb1241a My mistake, it wasn't verbose enough. Using the atomic cli to install kubelet breaks mount
09:50:06 <strigazi> propagation of secrets, configmaps and so on. Using podman
09:50:10 <strigazi> in a systemd unit works.
09:50:15 <brtknr> strigazi: why do you strongly dislike PODMAN_ENABLED btw?
09:50:31 <brtknr> i am not that fussed
09:50:59 <strigazi> it doesn't enable anything. But maybe from the user/ops pov it is better
09:52:29 <strigazi> logs?
09:52:30 <flwang1> strigazi: that's a valid point
09:52:42 <flwang1> generally, we use XXX_enabled only for addons
09:52:45 <brtknr> look at the end: https://seashells.io/p/WseYPMUE
09:52:47 <flwang1> not for this kind of case
09:53:23 <brtknr> lets leave it as USE_PODMAN then
09:53:44 <strigazi> brtknr: after the stop?
09:54:07 <strigazi> brtknr: which line?
09:54:16 <brtknr> Oct 23 09:50:29
09:54:41 <strigazi> brtknr: my patch fixes this
09:54:44 <flwang1> strigazi: brtknr: can we discuss details offline?
09:54:56 <brtknr> strigazi: yes, i already ack this
09:55:07 <strigazi> flwang1:
09:55:09 <strigazi> flwang1: ok
09:55:17 <flwang1> sorry, guys
09:55:20 <strigazi> anything last comments?
09:55:28 <flwang1> it's late here
09:55:45 <brtknr> okay, there's the issue with octavia and master lb fip
09:55:47 <strigazi> oh, have a look at the three patches from Theodoros I added in the etherpad
09:55:50 <brtknr> its working for flwang
09:55:54 <brtknr> not for me
09:56:10 <brtknr> could you please test this strigazi
09:56:15 <flwang1> strigazi: i will leave that for brtknr
09:57:41 <strigazi> one final thing from me
09:58:30 <strigazi> We need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros:
09:58:35 <strigazi> https://review.opendev.org/#/c/687879/
09:58:35 <strigazi> https://review.opendev.org/#/c/688400/
09:58:35 <strigazi> https://review.opendev.org/#/c/688346/
09:58:47 <strigazi> in the train branch
09:58:53 <flwang1> CERN is trying to upgrade to train ?
09:58:55 <strigazi> and release 9.1.0
09:59:00 <strigazi> yes
09:59:06 <flwang1> ok, i see then
09:59:28 <flwang1> to use the v1.16.x and later, right?
09:59:36 <strigazi> yes
09:59:44 <strigazi> 17 in two months
09:59:48 <flwang1> same here
10:00:04 <flwang1> catalyst is on stable/stein
10:00:11 <flwang1> with several private patches
10:00:11 <strigazi> we as well
10:00:18 <flwang1> cherrypicked from master
10:00:21 <strigazi> flwang1: how many?
10:00:29 <flwang1> more than 10
10:00:37 <strigazi> 48 here
10:00:39 <flwang1> from master(train)
10:00:43 <strigazi> who wins?
10:00:48 <flwang1> you win
10:00:54 <strigazi> xD
10:00:56 <flwang1> my friend, so you're keen than me
10:01:13 <strigazi> CherryPickaaS
10:01:18 <flwang1> ok, let's wrap up this meeting
10:01:21 <brtknr> :D
10:01:41 <flwang1> anything else?
10:01:42 <brtknr> we have only started upgrading to stein
10:02:02 <strigazi> I'm good, before closing
10:02:14 <strigazi> +1 or -1 to: We need the patches I proposed, for podman and heat-agent, plus the three fixes from Theodoros:
10:02:21 <strigazi> brtknr: flwang1 ^^
10:02:28 <strigazi> and then 9.1.0
10:02:30 <strigazi> brtknr: flwang1 ^^
10:02:31 <flwang1> strigazi: buy me a beer
10:02:47 <strigazi> sure thing? IPA?
10:02:51 <strigazi> sure thing, IPA?
10:03:16 <flwang1> hopefully i can join the next openstack summit
10:03:22 <flwang1> do you know where is it?
10:03:24 <flwang1> after shanghai
10:03:37 <strigazi> this is a good local one https://www.magictomato.ch/fr/geneve/item_details/531-biere-pale-ale-tangente-brasserie-du-virage
10:03:56 <strigazi> flwang1: I think Vancouver, but not sure
10:04:07 <flwang1> the logo looks cool
10:04:18 <flwang1> if it's Vancouver, i want to go
10:04:27 <strigazi> brtknr: flwang1 agree with the plan for 9.1.0?
10:04:28 <flwang1> if it's EU, no go
10:04:34 <brtknr> strigazi: sounds good
10:04:36 <flwang1> strigazi: works for me
10:04:40 <flwang1> i have to go
10:04:41 <strigazi> excellent
10:04:55 <flwang1> have fun, guys
10:04:59 <strigazi> good night, get some rest
10:05:04 <flwang1> o/
10:05:18 <brtknr> good nite!
10:05:53 <strigazi> #endmeeting