08:59:25 <flwang1> #startmeeting magnum 08:59:27 <openstack> Meeting started Wed Apr 29 08:59:25 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:59:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:59:30 <openstack> The meeting name has been set to 'magnum' 08:59:35 <flwang1> #topic roll call 09:00:32 <strigazi> o/ 09:01:46 <flwang1> o/ 09:01:52 <flwang1> brtknr: 09:02:48 <flwang1> strigazi: should we wait brtknr? 09:03:12 <brtknr> o/ 09:03:17 <brtknr> oops sorry 09:03:28 <brtknr> thanks for waiting 09:03:42 <flwang1> #topic FCOS docker storage driver https://review.opendev.org/#/c/718296/ 09:03:53 <flwang1> brtknr: strigazi: let's discuss this one and get it done 09:04:31 <flwang1> brtknr: strigazi: are we on the same page re https://review.opendev.org/#/c/718296/ ? 09:04:32 <strigazi> flwang1: what is the problem there? 09:04:44 <flwang1> brtknr still has concern 09:05:22 <brtknr> my concern is, user can specify any input that is not devicemapper and they always get overlay2 09:05:41 <strigazi> "and they get overlay2 silently." this is an extremely wrong assumption. 09:06:21 <strigazi> The field is openended to support custom storage-drivers and I linked the patch for this 09:06:22 <brtknr> is it? because the fcos driver doesnt work in the same way as fedora atomic driver 09:06:32 <strigazi> It says generic 09:07:11 <brtknr> i saw the patch but at present fedora coreos only supports overlay2 correct? 09:07:19 <strigazi> very wrong 09:07:48 <strigazi> docker is plugable and support any number of drivers installed as a docker plugin 09:07:59 <strigazi> docker plugin install ... 09:08:08 <strigazi> let me fetch the code 09:08:11 <brtknr> can you explain how it supports any other driver? 09:08:14 <strigazi> or man or smth 09:08:36 <strigazi> https://docs.docker.com/engine/reference/commandline/plugin_install/ 09:09:17 <strigazi> The field was enum and changed to string to be able to add more plugin 09:09:20 <strigazi> The field was enum and changed to string to be able to add more plugins 09:09:49 <brtknr> I understand you can install this manually but is this a feature that is supported through magnum? 09:10:19 <openstackgerrit> Merged openstack/magnum-tempest-plugin master: [ci] Support fedora-coreos in magnum-tempest-plugin https://review.opendev.org/721675 09:10:57 <strigazi> you have a magnum deployemnt and you want to add a plugin, what do to you do? db migration to change from enum to string? 09:11:30 <strigazi> It is not supported automagically 09:12:16 <strigazi> It is a bit more open on purpose. 09:12:52 <brtknr> Do you have custom code at CERN to support other storage drivers? I understand you use CVMFS? 09:12:59 <strigazi> What is the problem here? Protect users that put random strings everywhere? 09:13:03 <openstackgerrit> Merged openstack/python-magnumclient stable/ussuri: Update .gitreview for stable/ussuri https://review.opendev.org/719212 09:13:34 <strigazi> Protect users that put random strings everywhere? Is what we are trying to fix? 09:13:43 <flwang1> brtknr: my understanding is, that's the current behaviour. if we do have concern about this, we should have a separate patch to resolve it properly 09:13:49 <strigazi> No 09:13:56 <strigazi> we should resolve it now 09:14:12 <brtknr> strigazi: My main concern is that users "think" they can get overlay or one of the other supported docker storage plugins but they *always* get overlay2 with Fedora CoreOS driver 09:14:32 <openstackgerrit> Merged openstack/python-magnumclient stable/ussuri: Update TOX/UPPER_CONSTRAINTS_FILE for stable/ussuri https://review.opendev.org/719214 09:14:35 <strigazi> as I said this is wrong 09:14:43 <flwang1> brtknr: we do have warnings about this, isn't it? 09:14:44 <strigazi> if you set overlay you overlay 09:14:53 <strigazi> if you set overlay you get overlay 09:15:02 <brtknr> I understand the situation is different with Atomic as we actually set the configuration in /etc/docker/docker-storage file 09:15:18 <brtknr> but in FCOS, this config is completely ignored 09:15:24 <brtknr> and the system default is used 09:15:35 <brtknr> and it has no effect 09:15:47 <brtknr> thats the point im trying to make 09:15:52 <strigazi> ok, so the goal again is to protect the user that set overlay, correct? 09:16:02 <brtknr> yes, as an example 09:16:35 <strigazi> So magnum should validate all addons we deploy for correct configuration? 09:16:57 <brtknr> ideally, i would like something similar to what we do with Atomic 09:17:03 <strigazi> do you guys thing we can afford this? 09:17:14 <strigazi> do you guys think we can afford this? 09:17:34 <brtknr> e.g. https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver 09:17:47 <strigazi> so brtknr proposes to allow only overlay2, correct? 09:18:40 <strigazi> change to daemon.json then instead of allowing only overlay2 09:18:48 <flwang1> brtknr: can I remove your concern by add the file /etc/docker/daemon.json ? 09:19:08 <brtknr> flwang1: yes 09:19:16 <flwang1> strigazi: are you happy with that? 09:19:18 <brtknr> and do something with the docker-storage-driver variable 09:19:53 <flwang1> what do you mean do something? my understanding is inject whatever we got into the daemon.sjon 09:20:45 <strigazi> "storage-driver": "$DOCKER_STORAGE_DRIVER" 09:21:13 <flwang1> strigazi: yes, that's what i'm thinking 09:21:19 <brtknr> flwang1: we already do something similar in https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/fragments/configure_docker_storage_driver_atomic.sh#L31 09:21:57 <flwang1> brtknr: that's same as what i'm talking about, just inject the driver into the json file 09:22:07 <flwang1> i will propose a patchset 09:22:13 <brtknr> flwang1: thanks 09:22:20 <flwang1> are we all on the same page now? 09:22:20 <strigazi> brtknr: are your users using overlay? 09:22:33 <flwang1> strigazi: are you OK? 09:23:04 <brtknr> strigazi: not that I know of 09:23:24 <strigazi> flwang1: sure 09:23:28 <strigazi> brtknr: who does? 09:23:48 <brtknr> I just dont want docker_storage_driver variable to be a zombie variable that has no effect 09:24:00 <flwang1> move on? 09:24:38 <strigazi> let's move on 09:25:19 <flwang1> #topic CA certs rotate https://review.opendev.org/#/c/724203/ 09:25:32 <flwang1> the ca certs rotate 09:25:45 <flwang1> i saw we have the API for quite a while, but we don't have the implementation 09:25:55 <flwang1> strigazi: do you know more background? 09:26:02 <strigazi> To add in the previous topic, CMVFS is not a storage driver. 09:26:35 <brtknr> strigazi: hehe, apologies for my limited understanding 09:27:02 <flwang1> it's shame for me that we have the api but with an empty function, so i'd like to get it done 09:27:07 <flwang1> strigazi: comments? 09:27:13 <flwang1> brtknr: 09:27:23 <strigazi> flwang1: contributors proposed it and left 09:28:13 <flwang1> is CERN interested in this ? 09:28:27 <flwang1> brtknr: SKA? 09:28:41 <brtknr> flwang1: i'm sure it'd be useful for long running clusters 09:28:51 <flwang1> brtknr: yes, it is 09:29:02 <strigazi> we are working on using keycloack 09:29:26 <strigazi> We have a central instance and we want to use that on our clusters. 09:29:50 <flwang1> strigazi: ok, fair enough, but are you happy we get it done in Magnum anyway? 09:30:01 <brtknr> have 09:30:31 <brtknr> flwang1: what is your use case for this? 09:30:35 <strigazi> we can do it 09:31:02 <strigazi> users leaves org, has the certs , can access cluster 09:31:11 <flwang1> brtknr: user want to rotate the ca certs to let all user to replace their certs 09:31:20 <flwang1> strigazi: exactly 09:31:53 <strigazi> flwang1: keystone webhook is better, for your own piece of mind 09:32:10 <strigazi> cert rotation is welcome 09:32:23 <strigazi> more importantly, trust rotation 09:32:31 <flwang1> strigazi: yep, keystone webhook is better, but we can't enforce that 09:32:51 <flwang1> trust rotation patch is being reviewed in Heat 09:32:54 <strigazi> flwang1: you mean technically? 09:32:55 <flwang1> i need to test it 09:33:08 <flwang1> strigazi: :) 09:33:34 <flwang1> i mean we still want to keep the cert login as an option so far 09:34:08 <strigazi> ah ok, when we are done with keycloack we will try to add an option to resrict cert access 09:34:19 <strigazi> opt-in of course 09:34:32 <flwang1> strigazi: cool, i'm interested in it 09:34:57 <flwang1> move on? 09:35:02 <strigazi> yes 09:35:12 <flwang1> #topic Magnum Labels Override https://review.opendev.org/#/c/716571/ 09:35:54 <strigazi> brtknr: are you happy with the spec? 09:36:26 <brtknr> Yes, just some nits for ttsiouts to address 09:36:42 <ttsiouts> o/ 09:36:48 <ttsiouts> sorry for being late 09:37:00 <ttsiouts> I'll try to address them asap 09:37:11 <strigazi> how serious nits? 09:37:17 <ttsiouts> do you guys agree with the 3 fields? 09:37:24 <strigazi> flwang1: are you up to date? 09:37:29 <strigazi> +1 09:37:35 <strigazi> ttsiouts: ^^ 09:37:51 <flwang1> strigazi: i haven't review the latest version 09:37:54 <flwang1> is it ready? 09:38:03 <brtknr> strigazi: minor nits 09:38:09 <brtknr> typos 09:38:10 <ttsiouts> strigazi: I need to add more about the 3 fields in the get output 09:38:42 <brtknr> do you guys prefer labels_skipped, labels_overridden, labels_added or added_labels, overridden_labels, skipped_labels? 09:39:13 <flwang1> strigazi: i just done a very quick glance and i think it matched the POC patch 09:39:18 <strigazi> labels_ first 09:39:20 <flwang1> so I'm happy with it overall 09:39:27 <ttsiouts> great! 09:39:37 <brtknr> i am also labels_* camp 09:39:42 <flwang1> ttsiouts: pls start to polish the code patch 09:39:48 <ttsiouts> flwang1: sure! 09:39:49 <brtknr> flwang1: what about you? 09:39:58 <brtknr> *_labels or labels_* 09:40:22 <ttsiouts> labels_* feels better 09:40:25 <flwang1> labels_* 09:40:45 <brtknr> great we have a unanimous vote 09:41:00 <ttsiouts> awesome! 09:41:09 <flwang1> minors changes can be updated into the spec even after it merged i think, if we want 09:41:34 <flwang1> we don't have much time for Ussuri, so we need to be quick 09:41:59 <strigazi> (ctrl + shift + c or ctrl + shift + z removes colors in etherpad....) 09:43:10 <flwang1> strigazi: i like your review query 09:43:12 <flwang1> thanks for sharing 09:44:52 <brtknr> strigazi: yes I bookmarked it 09:45:52 <brtknr> ok next topic? 09:46:01 <brtknr> flwang1: 09:46:05 <flwang1> we finished all topics :) 09:46:06 <strigazi> I wanted to added in our docs. I want to do many things but it seems I can not expand time yet, I hope someone at CERN works on it 09:46:24 <strigazi> I added one more, Create API to query nodes in a cluster https://storyboard.openstack.org/#!/story/2007590 09:46:25 <flwang1> strigazi: brtknr: open discussion? 09:46:50 <strigazi> This was opened in kubernetes/autoscaler by brtknr 09:47:29 <flwang1> strigazi: btw, could you please ask Thomas to finish the API ref for node group? 09:47:41 <strigazi> he actively works on it 09:47:44 <brtknr> yes, this was when I hadn't understood that cluster autoscaler only works on default worker nodegroup 09:47:59 <flwang1> strigazi: great, thanks 09:48:14 <strigazi> Can someone implement https://storyboard.openstack.org/#!/story/2007590 ? 09:48:24 <flwang1> brtknr: query the node or query the node group of the node? 09:48:27 <openstackgerrit> Merged openstack/magnum-tempest-plugin master: [ci] Define/enable magnum-tempest-plugin-tests-api https://review.opendev.org/721077 09:48:46 <flwang1> i had a implementation in auto healer 09:48:51 <flwang1> let me find the link 09:49:26 <flwang1> https://github.com/kubernetes/cloud-provider-openstack/pull/1032/files 09:50:17 <strigazi> flwang1: node -> nodegroup 09:50:28 <flwang1> strigazi: based on my understanding, that's only for the resize api, right? 09:50:44 <flwang1> then how about we make the node group parameter optional 09:51:06 <flwang1> and automatically get the nodegroup for each node passed in 09:51:15 <flwang1> strigazi: brtknr: thoughts? 09:51:28 <strigazi> sorry, https://github.com/kubernetes/autoscaler/issues/2802#issuecomment-592481526 we need to implement the Nodes() method 09:52:24 <flwang1> an API to list all nodes and their node group? 09:52:48 <strigazi> I think all nodes in a NG 09:53:12 <brtknr> flwang1: openstack coe node list <cluster> --nodegroup <nodegroup> 09:53:24 <strigazi> https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L141 09:53:29 <flwang1> ok, i will think about this 09:53:55 <flwang1> i don't have much band width atm 09:54:45 <flwang1> i'm happy to pickup this after i finish the ca rotate work 09:55:06 <flwang1> strigazi: is it so urgent? 09:55:19 <brtknr> I can attempt this but as you probably saw, I havent had a chance to pickup the resize worker to 0 patch yet 09:55:36 <brtknr> So this will need to join the queue 09:55:37 <strigazi> brtknr: is it urgent? 09:56:01 <flwang1> brtknr: right, i'm interested in the 0 worker cluster 09:56:39 <brtknr> strigazi: Not urgent but will will need this by the end of this year 09:56:55 <flwang1> brtknr: by the end of 2020? 09:57:03 <brtknr> The project we need this for doesnt start until June 09:57:09 <brtknr> hence I am holding off 09:57:22 <flwang1> brtknr: i can get it done before that for sure :D 09:57:39 <flwang1> no worries 09:58:19 <brtknr> flwang1: I am keen to do it too because I am not 100% comfortable working with the API logic yet, and wouldnt mind some experience 09:58:20 <strigazi> flwang1: then you can take https://storyboard.openstack.org/#!/story/2007590 :) 09:58:56 <flwang1> strigazi: no problem, i can help brtknr to get it done 09:59:14 <brtknr> flwang1: thanks 09:59:20 <strigazi> end meeting ? 09:59:21 <flwang1> anything else? 09:59:29 <flwang1> #endmeeting