08:59:25 <flwang1> #startmeeting magnum
08:59:27 <openstack> Meeting started Wed Apr 29 08:59:25 2020 UTC and is due to finish in 60 minutes.  The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:59:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:59:30 <openstack> The meeting name has been set to 'magnum'
08:59:35 <flwang1> #topic roll call
09:00:32 <strigazi> o/
09:01:46 <flwang1> o/
09:01:52 <flwang1> brtknr:
09:02:48 <flwang1> strigazi: should we wait brtknr?
09:03:12 <brtknr> o/
09:03:17 <brtknr> oops sorry
09:03:28 <brtknr> thanks for waiting
09:03:42 <flwang1> #topic FCOS docker storage driver https://review.opendev.org/#/c/718296/
09:03:53 <flwang1> brtknr: strigazi: let's discuss this one and get it done
09:04:31 <flwang1> brtknr: strigazi: are we on the same page re https://review.opendev.org/#/c/718296/ ?
09:04:32 <strigazi> flwang1: what is the problem there?
09:04:44 <flwang1> brtknr still has concern
09:05:22 <brtknr> my concern is, user can specify any input that is not devicemapper and they always get overlay2
09:05:41 <strigazi> "and they get overlay2 silently." this is an extremely wrong assumption.
09:06:21 <strigazi> The field is openended to support custom storage-drivers and I linked the patch for this
09:06:22 <brtknr> is it? because the fcos driver doesnt work in the same way as fedora atomic driver
09:06:32 <strigazi> It says generic
09:07:11 <brtknr> i saw the patch but at present fedora coreos only supports overlay2 correct?
09:07:19 <strigazi> very wrong
09:07:48 <strigazi> docker is plugable and support any number of drivers installed as a docker plugin
09:07:59 <strigazi> docker plugin install ...
09:08:08 <strigazi> let me fetch the code
09:08:11 <brtknr> can you explain how it supports any other driver?
09:08:14 <strigazi> or man or smth
09:08:36 <strigazi> https://docs.docker.com/engine/reference/commandline/plugin_install/
09:09:17 <strigazi> The field was enum and changed to string to be able to add more plugin
09:09:20 <strigazi> The field was enum and changed to string to be able to add more plugins
09:09:49 <brtknr> I understand you can install this manually but is this a feature that is supported through magnum?
09:10:19 <openstackgerrit> Merged openstack/magnum-tempest-plugin master: [ci] Support fedora-coreos in magnum-tempest-plugin  https://review.opendev.org/721675
09:10:57 <strigazi> you have a magnum deployemnt and you want to add a plugin, what do to you do? db migration to change from enum to string?
09:11:30 <strigazi> It is not supported automagically
09:12:16 <strigazi> It is a bit more open on purpose.
09:12:52 <brtknr> Do you have custom code at CERN to support other storage drivers? I understand you use CVMFS?
09:12:59 <strigazi> What is the problem here? Protect users that put random strings everywhere?
09:13:03 <openstackgerrit> Merged openstack/python-magnumclient stable/ussuri: Update .gitreview for stable/ussuri  https://review.opendev.org/719212
09:13:34 <strigazi> Protect users that put random strings everywhere? Is what we are trying to fix?
09:13:43 <flwang1> brtknr: my understanding is, that's the current behaviour. if we do have concern about this, we should have a separate patch to resolve it properly
09:13:49 <strigazi> No
09:13:56 <strigazi> we should resolve it now
09:14:12 <brtknr> strigazi: My main concern is that users "think" they can get overlay or one of the other supported docker storage plugins but they *always* get overlay2 with Fedora CoreOS driver
09:14:32 <openstackgerrit> Merged openstack/python-magnumclient stable/ussuri: Update TOX/UPPER_CONSTRAINTS_FILE for stable/ussuri  https://review.opendev.org/719214
09:14:35 <strigazi> as I said this is wrong
09:14:43 <flwang1> brtknr: we do have warnings about this, isn't it?
09:14:44 <strigazi> if you set overlay you overlay
09:14:53 <strigazi> if you set overlay you get overlay
09:15:02 <brtknr> I understand the situation is different with Atomic as we actually set the configuration in /etc/docker/docker-storage file
09:15:18 <brtknr> but in FCOS, this config is completely ignored
09:15:24 <brtknr> and the system default is used
09:15:35 <brtknr> and it has no effect
09:15:47 <brtknr> thats the point im trying to make
09:15:52 <strigazi> ok, so the goal again is to protect the user that set overlay, correct?
09:16:02 <brtknr> yes, as an example
09:16:35 <strigazi> So magnum should validate all addons we deploy for correct configuration?
09:16:57 <brtknr> ideally, i would like something similar to what we do with Atomic
09:17:03 <strigazi> do you guys thing we can afford this?
09:17:14 <strigazi> do you guys think we can afford this?
09:17:34 <brtknr> e.g. https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver
09:17:47 <strigazi> so brtknr proposes to allow only overlay2, correct?
09:18:40 <strigazi> change to daemon.json then instead of allowing only overlay2
09:18:48 <flwang1> brtknr: can I remove your concern by add the file /etc/docker/daemon.json ?
09:19:08 <brtknr> flwang1: yes
09:19:16 <flwang1> strigazi: are you happy with that?
09:19:18 <brtknr> and do something with the docker-storage-driver variable
09:19:53 <flwang1> what do you mean do something? my understanding is inject whatever we got into the daemon.sjon
09:20:45 <strigazi> "storage-driver": "$DOCKER_STORAGE_DRIVER"
09:21:13 <flwang1> strigazi: yes, that's what i'm thinking
09:21:19 <brtknr> flwang1: we already do something similar in https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/fragments/configure_docker_storage_driver_atomic.sh#L31
09:21:57 <flwang1> brtknr: that's  same as what i'm talking about, just inject the driver into the json file
09:22:07 <flwang1> i will propose a patchset
09:22:13 <brtknr> flwang1:  thanks
09:22:20 <flwang1> are we all on the same page now?
09:22:20 <strigazi> brtknr: are your users using overlay?
09:22:33 <flwang1> strigazi: are you OK?
09:23:04 <brtknr> strigazi: not that I know of
09:23:24 <strigazi> flwang1: sure
09:23:28 <strigazi> brtknr: who does?
09:23:48 <brtknr> I just dont want docker_storage_driver variable to be a zombie variable that has no effect
09:24:00 <flwang1> move on?
09:24:38 <strigazi> let's move on
09:25:19 <flwang1> #topic CA certs rotate   https://review.opendev.org/#/c/724203/
09:25:32 <flwang1> the ca certs rotate
09:25:45 <flwang1> i saw we have the API for quite a while, but we don't have the implementation
09:25:55 <flwang1> strigazi: do you know more background?
09:26:02 <strigazi> To add in the previous topic, CMVFS is not a storage driver.
09:26:35 <brtknr> strigazi: hehe, apologies for my limited understanding
09:27:02 <flwang1> it's shame for me that we have the api but with an empty function, so i'd like to get it done
09:27:07 <flwang1> strigazi: comments?
09:27:13 <flwang1> brtknr:
09:27:23 <strigazi> flwang1: contributors proposed it and left
09:28:13 <flwang1> is CERN interested in this ?
09:28:27 <flwang1> brtknr: SKA?
09:28:41 <brtknr> flwang1: i'm sure it'd be useful for long running clusters
09:28:51 <flwang1> brtknr: yes, it is
09:29:02 <strigazi> we are working on using keycloack
09:29:26 <strigazi> We have a central instance and we want to use that on our clusters.
09:29:50 <flwang1> strigazi: ok, fair enough, but are you happy we get it done in Magnum anyway?
09:30:01 <brtknr> have
09:30:31 <brtknr> flwang1: what is your use case for this?
09:30:35 <strigazi> we can do it
09:31:02 <strigazi> users leaves org, has the certs , can access cluster
09:31:11 <flwang1> brtknr: user want to rotate the ca certs to let all user to replace their certs
09:31:20 <flwang1> strigazi: exactly
09:31:53 <strigazi> flwang1: keystone webhook is better, for your own piece of mind
09:32:10 <strigazi> cert rotation is welcome
09:32:23 <strigazi> more importantly, trust rotation
09:32:31 <flwang1> strigazi: yep, keystone webhook is better, but we can't enforce that
09:32:51 <flwang1> trust rotation patch is being reviewed in Heat
09:32:54 <strigazi> flwang1: you mean technically?
09:32:55 <flwang1> i need to test it
09:33:08 <flwang1> strigazi: :)
09:33:34 <flwang1> i mean we still want to keep the cert login as an option so far
09:34:08 <strigazi> ah ok, when we  are done with keycloack we will try to add an option to resrict cert access
09:34:19 <strigazi> opt-in of course
09:34:32 <flwang1> strigazi: cool, i'm interested in it
09:34:57 <flwang1> move on?
09:35:02 <strigazi> yes
09:35:12 <flwang1> #topic Magnum Labels Override https://review.opendev.org/#/c/716571/
09:35:54 <strigazi> brtknr: are you happy with the spec?
09:36:26 <brtknr> Yes, just some nits for ttsiouts to address
09:36:42 <ttsiouts> o/
09:36:48 <ttsiouts> sorry for being late
09:37:00 <ttsiouts> I'll try to address them asap
09:37:11 <strigazi> how serious nits?
09:37:17 <ttsiouts> do you guys agree with the 3 fields?
09:37:24 <strigazi> flwang1: are you up to date?
09:37:29 <strigazi> +1
09:37:35 <strigazi> ttsiouts: ^^
09:37:51 <flwang1> strigazi: i haven't review the latest version
09:37:54 <flwang1> is it ready?
09:38:03 <brtknr> strigazi: minor nits
09:38:09 <brtknr> typos
09:38:10 <ttsiouts> strigazi: I need to add more about the 3 fields in the get output
09:38:42 <brtknr> do you guys prefer labels_skipped, labels_overridden, labels_added or added_labels, overridden_labels, skipped_labels?
09:39:13 <flwang1> strigazi: i just done a very  quick glance and i think it matched the POC patch
09:39:18 <strigazi> labels_ first
09:39:20 <flwang1> so I'm happy with it overall
09:39:27 <ttsiouts> great!
09:39:37 <brtknr> i am also labels_* camp
09:39:42 <flwang1> ttsiouts: pls start to polish the code patch
09:39:48 <ttsiouts> flwang1: sure!
09:39:49 <brtknr> flwang1: what about you?
09:39:58 <brtknr> *_labels or labels_*
09:40:22 <ttsiouts> labels_* feels better
09:40:25 <flwang1> labels_*
09:40:45 <brtknr> great we have a unanimous vote
09:41:00 <ttsiouts> awesome!
09:41:09 <flwang1> minors changes can be updated into the spec even after it merged i think, if we want
09:41:34 <flwang1> we don't have much time for Ussuri, so we need to be quick
09:41:59 <strigazi> (ctrl + shift + c or ctrl + shift + z removes colors in etherpad....)
09:43:10 <flwang1> strigazi: i like your review  query
09:43:12 <flwang1> thanks for sharing
09:44:52 <brtknr> strigazi: yes I bookmarked it
09:45:52 <brtknr> ok next topic?
09:46:01 <brtknr> flwang1:
09:46:05 <flwang1> we finished all topics :)
09:46:06 <strigazi> I wanted to added in our docs. I want to do many things but it seems I can not expand time yet, I hope someone at CERN works on it
09:46:24 <strigazi> I added one more, Create API to query nodes in a cluster https://storyboard.openstack.org/#!/story/2007590
09:46:25 <flwang1> strigazi: brtknr: open discussion?
09:46:50 <strigazi> This was opened in kubernetes/autoscaler by brtknr
09:47:29 <flwang1> strigazi: btw, could you please ask Thomas to finish the API ref for node group?
09:47:41 <strigazi> he actively works on it
09:47:44 <brtknr> yes, this was when I hadn't understood that cluster autoscaler only works on default worker nodegroup
09:47:59 <flwang1> strigazi: great, thanks
09:48:14 <strigazi> Can someone implement https://storyboard.openstack.org/#!/story/2007590 ?
09:48:24 <flwang1> brtknr: query the node or query the node group of the node?
09:48:27 <openstackgerrit> Merged openstack/magnum-tempest-plugin master: [ci] Define/enable magnum-tempest-plugin-tests-api  https://review.opendev.org/721077
09:48:46 <flwang1> i had a implementation in auto healer
09:48:51 <flwang1> let me find the link
09:49:26 <flwang1> https://github.com/kubernetes/cloud-provider-openstack/pull/1032/files
09:50:17 <strigazi> flwang1:  node -> nodegroup
09:50:28 <flwang1> strigazi: based on my understanding, that's only for the resize api, right?
09:50:44 <flwang1> then how about we make the node group parameter optional
09:51:06 <flwang1> and automatically get the nodegroup for each node passed in
09:51:15 <flwang1> strigazi: brtknr: thoughts?
09:51:28 <strigazi> sorry, https://github.com/kubernetes/autoscaler/issues/2802#issuecomment-592481526 we need to implement the Nodes() method
09:52:24 <flwang1> an API to list all nodes and their node group?
09:52:48 <strigazi> I think all nodes in a NG
09:53:12 <brtknr> flwang1: openstack coe node list <cluster> --nodegroup <nodegroup>
09:53:24 <strigazi> https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L141
09:53:29 <flwang1> ok, i will think about this
09:53:55 <flwang1> i don't have much band width atm
09:54:45 <flwang1> i'm happy to pickup this after i finish the ca rotate work
09:55:06 <flwang1> strigazi: is it so urgent?
09:55:19 <brtknr> I can attempt this but as you probably saw, I havent had a chance to pickup the resize worker to 0 patch yet
09:55:36 <brtknr> So this will need to join the queue
09:55:37 <strigazi> brtknr: is it urgent?
09:56:01 <flwang1> brtknr: right, i'm interested in the 0 worker cluster
09:56:39 <brtknr> strigazi: Not urgent but will will need this by the end of this year
09:56:55 <flwang1> brtknr: by the end of 2020?
09:57:03 <brtknr> The project we need this for doesnt start until June
09:57:09 <brtknr> hence I am holding off
09:57:22 <flwang1> brtknr: i can get it done before that for sure :D
09:57:39 <flwang1> no worries
09:58:19 <brtknr> flwang1: I am keen to do it too because I am not 100% comfortable working with the API logic yet, and wouldnt mind some experience
09:58:20 <strigazi> flwang1: then you can take https://storyboard.openstack.org/#!/story/2007590 :)
09:58:56 <flwang1> strigazi: no problem, i can help brtknr to get it done
09:59:14 <brtknr> flwang1: thanks
09:59:20 <strigazi> end meeting ?
09:59:21 <flwang1> anything else?
09:59:29 <flwang1> #endmeeting