21:00:11 <strigazi> #startmeeting containers 21:00:12 <openstack> Meeting started Tue Oct 23 21:00:11 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:16 <openstack> The meeting name has been set to 'containers' 21:00:19 <strigazi> #topic Roll Call 21:00:29 <strigazi> o/ 21:00:32 <ttsiouts> o/ 21:01:17 <flwang> o/ 21:02:47 <imdigitaljim> o/ 21:03:14 <strigazi> brtknr: you here? 21:03:17 <eandersson> o/ 21:03:53 <strigazi> Thanks for joining the meeting ttsioutsflwang imdigitaljim eandersson 21:03:59 <strigazi> Agenda: 21:04:05 <strigazi> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2018-10-23_2100_UTC 21:04:16 <strigazi> it has some items 21:04:27 <strigazi> #topic Stories/Tasks 21:04:51 <strigazi> 1. node groups https://review.openstack.org/#/c/607363/ 21:05:13 <strigazi> I think we are pretty close to the final state of the spec 21:05:21 <strigazi> please take a look 21:05:44 <ttsiouts> strigazi: tomorrow I will push again 21:06:02 <ttsiouts> adapting ricardo's comments 21:06:41 <strigazi> oh I thought you pushed today, ok so take a looks guys take a look tmr as well :) 21:07:22 <ttsiouts> :) tmr better 21:07:24 <strigazi> @all do you want to discuss anything about nodegroups today? 21:07:46 <strigazi> questions about nodegroups? 21:08:23 <strigazi> ok, next then 21:08:26 <schaney> o/ sorry for lateness, but yes 21:09:06 <strigazi> schaney: hello. you have smth about nodegroups? 21:09:08 <schaney> any mechanism to interact with NGs individually? 21:09:29 <schaney> opposed to the top level cluster or stack 21:09:57 <strigazi> the api will be like: 21:10:01 <ttsiouts> schaney: do you mean updating a specific nodegroup? 21:10:20 <schaney> yes for scaling or the like 21:10:33 <strigazi> cluster/<cluster-identity>/nodegroup/<nodegroup-identity> 21:10:48 <strigazi> so PATCH cluster/<cluster-identity>/nodegroup/<nodegroup-identity> 21:10:48 <ttsiouts> https://review.openstack.org/#/c/607363/2/specs/stein/magnum-nodegroups.rst@117 21:11:37 <colin-> sorry i'm late! 21:11:49 <strigazi> colin-: welcome 21:12:44 <schaney> oh gotcha, ill have to dig into the work a bit, under the hood magnum just targets the name of the node group represented by the heat parameter though? 21:13:05 <flwang> ttsiouts: the node groups is basically the same thing like node pool in GKE, right? 21:13:32 <ttsiouts> schaney: that's the plan 21:13:48 <ttsiouts> flwang: exactly 21:14:05 <flwang> ttsiouts: cool 21:14:37 <schaney> i see, did there happen to be any work with the "random node scale down" when magnum shrinks the cluster? 21:14:40 <flwang> ttsiouts: i will review the spec first 21:14:56 <schaney> from the API route it would seem so? 21:14:59 <flwang> i think we probably better read the spec first and put comments in the code 21:15:07 <flwang> instead of discussing design details here 21:15:12 <schaney> good call 21:16:05 <ttsiouts> schaney: we want to add a CLI for removing specific nodes from the cluster 21:16:18 <ttsiouts> but this will come further down the way 21:16:21 <strigazi> schaney: this won't be covered by this spec, but we should track it somewhere else 21:16:39 <schaney> gotcha, thanks for the info 21:16:48 <ttsiouts> flwang: thanks! 21:17:05 <ttsiouts> flwang: tmr it will be more complete 21:17:17 <imdigitaljim> were looking at modifying our driver to perhaps consume senlin clusters for each group 21:17:39 <imdigitaljim> masters, minions-group-1, ... minions-group-n 21:17:49 <flwang> imdigitaljim: is it a hard depedency? 21:17:51 <imdigitaljim> would all have a senlin profile/cluster in heat 21:17:53 <flwang> i mean for senlin 21:18:01 <strigazi> imdigitaljim: that could be done, that is why we have drivers 21:18:14 <strigazi> flwang: should be optional 21:18:34 <strigazi> like alternative 21:18:41 <imdigitaljim> it would probably be the driver would take on a senlin dependency (not magnum as a whole) 21:18:44 <imdigitaljim> just like octavia or not 21:19:08 <strigazi> when the cluster drivers where proposed senlin and ansble were the reasoning behind it 21:19:20 <imdigitaljim> we're more focused on autoscaling/autohealing rather than cli manual scaling 21:20:02 <imdigitaljim> the senlin PTL is here and is actively talking to the heat PTL on managing the senlin resources in heat so we might be able to inhouse create a better opportunity for senlin + heat + magnum 21:20:47 <strigazi> imdigitaljim: here, like in this meeting? 21:20:51 <imdigitaljim> no 21:21:00 <imdigitaljim> sorry i just mean he works at blizzard 21:22:19 <strigazi> This plan is compatible with nodegroups and nodegroups make it actually easier 21:22:55 <imdigitaljim> we think so 21:23:04 <strigazi> I'm not aware of it in detail, but it sounds doable 21:23:17 <schaney> Senlin would work well with the NG layout, one thing to note is Senlin's dedicated API 21:23:26 <imdigitaljim> yeah we arent either but we'll be working out over the next couple weeks 21:23:50 <imdigitaljim> and see if its within reason on feasibility 21:23:50 <eandersson> The Senlin PTL will be in Berlin btw 21:23:53 <imdigitaljim> ^ 21:24:06 <strigazi> cool 21:24:16 <cbrumm__> Its honestly too early to be talking about, there's a lot of heat/senlin ground work to do first 21:24:32 <cbrumm__> But hey, it's a thing we're thinking about. 21:24:43 <strigazi> fair enough 21:25:38 <strigazi> shall we move on? 21:26:01 <cbrumm__> yes 21:26:30 <strigazi> I pushed two patches we were talking about, one is: 21:26:41 <strigazi> Add heat_container_agent_tag label https://review.openstack.org/#/c/612727 21:27:20 <strigazi> we discussed with flwang and eandersson already, others have a look too 21:27:52 <strigazi> the tag of the heat-agent was hardcoded this makes it a label. 21:28:21 <strigazi> the 2nd one needs some discussion, it is 21:28:30 <strigazi> deploy tiller by default https://review.openstack.org/#/c/612336/ 21:29:30 <strigazi> Shall we have it in by default or optional? 21:30:39 <flwang> strigazi: any potential issue if we enable it by default? 21:30:41 <strigazi> and then next steps are, with tls or without? a tiller per ns or with the cluster-role? 21:31:16 <strigazi> flwang: the user might want a diffrent tiller config 21:31:25 <flwang> strigazi: that's the problem i think 21:31:34 <strigazi> flwang: other than that, tiller will be there silent 21:32:16 <flwang> we have seen similar 'issue' with other enabling, like the keystone auth integration 21:32:17 <strigazi> flwang: so you are in-favor of optional 21:32:38 <flwang> a new enabling feature may introduce a bunch of config 21:32:51 <flwang> but now in mangum, we don't have a good way to maintain those config 21:33:08 <flwang> labels are too flexible and lose 21:33:14 <flwang> i prefer optional 21:33:38 <flwang> based on the feedback we got so far, most of customer just want a vanilla k8s cluster 21:34:05 <flwang> if they want something, they can DIY 21:34:26 <strigazi> what vanilla means? api, sch, cm, kubelet, proxy, dns, cni 21:34:38 <flwang> and i agree it's because we(catalyst cloud) are public cloud and our customers' requirements are vary 21:34:57 <flwang> vanilla means a pure cluster, not too much plugins/addons 21:35:11 <cbrumm__> flwang: We've been getting some similar feedback from power users too 21:35:17 <flwang> for private cloud, the thing maybe different 21:35:40 <cbrumm__> but I think that's expected from power users that are used to DYI 21:35:48 <flwang> most of the customers of k8s know how to play with it 21:36:13 <flwang> what they want is just a stable k8s cluster with good HA and integration with the underhood cloud provider 21:36:48 <flwang> cbrumm__: what do you mean 'power users'? 21:37:00 <strigazi> so optional 21:37:21 <cbrumm__> people who've used k8s before that was not a managed service 21:37:34 <flwang> cbrumm__: ok, i see, thx 21:38:24 <strigazi> flwang: any argument against having this optional? 21:39:07 <flwang> strigazi: TBH, I would suggest we start to define a better addon architecture 21:39:10 <imdigitaljim> ^ 21:39:29 <flwang> like refactor the labels 21:39:37 <imdigitaljim> thats one of the goals our driver plans to be solving 21:39:53 <flwang> imdigitaljim: show me the code ;) 21:40:21 <strigazi> so we don't add anything until we refactor? 21:40:25 <flwang> i have heard about the v2 driver million times, i want to see the code :D 21:40:33 <flwang> strigazi: i'm not saying that 21:40:38 <imdigitaljim> :D when i get some free cycles and feel like its in a great uploading spot 21:40:42 <flwang> i just say we should be more careful 21:42:49 <strigazi> the current model is not unreasonable. we need to define careful and not make the service a framework 21:43:47 <strigazi> I think for v1 which can be refactored only replaced/deprecated the model of addons is there 21:44:00 <strigazi> s/can/can't/ 21:44:27 <strigazi> labels for on/off and tags 21:44:57 <imdigitaljim> imo we should look into a config file 21:45:13 <imdigitaljim> kubernetes followed the same pattern when they realized there were too many flags 21:45:38 <strigazi> config file to create the labels/fields or a config file to pass to the cluster 21:45:42 <strigazi> ? 21:46:12 <openstackgerrit> Merged openstack/magnum stable/queens: Provide a region to the K8S Fedora Atomic config https://review.openstack.org/612199 21:46:19 <imdigitaljim> openstack coe cluster template create --config myvalues.yaml 21:46:48 <strigazi> these values are like labels? 21:46:53 <imdigitaljim> could be everything 21:46:57 <imdigitaljim> could be just labels 21:47:02 <strigazi> code too? 21:47:13 <strigazi> like to code? 21:47:19 <imdigitaljim> ? 21:47:20 <strigazi> or link to code? 21:47:53 <imdigitaljim> instead of like --labels octavia_enabled=true, etc etc 21:47:56 <imdigitaljim> it could be 21:48:03 <imdigitaljim> [Loadbalancer} 21:48:09 <imdigitaljim> [LoadBalancer] 21:48:12 <strigazi> got it 21:48:15 <imdigitaljim> octavia_enabled=true 21:48:27 <imdigitaljim> but you could also do 21:48:29 <imdigitaljim> [Network] 21:48:55 <imdigitaljim> floating_ ... fixed_network= .. fixed_subnet= 21:48:58 <flwang> yaml or the ini format 21:49:02 <imdigitaljim> either or 21:49:02 <imdigitaljim> any 21:49:04 <imdigitaljim> json 21:49:09 <flwang> yep 21:49:09 <imdigitaljim> doesnt matter however we want to do it 21:49:18 <flwang> agree 21:49:21 <imdigitaljim> imo im a fan of that model 21:49:43 <flwang> and with that case, we can publish sample config files 21:49:48 <strigazi> flwang: what would cover your concern about the loose label design? 21:49:52 <flwang> and user can decide how to combine the config 21:50:11 <flwang> strigazi: yep, that's basically the arch in my mind 21:51:05 <strigazi> #action strigazi to draft a spec and story for creating cluster with a config file. 21:51:13 <flwang> strigazi: can we discuss this one https://github.com/kubernetes/cloud-provider-openstack/issues/280 next? 21:51:30 <strigazi> I'll try to bring smth in the next meeting for this 21:51:39 <flwang> strigazi: thanks 21:52:24 <strigazi> before going into the bug of CPO 21:53:02 <strigazi> @all take a look to the rest of the list of review in the agenda. they are ready to go in 21:53:07 <strigazi> https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2018-10-23_2100_UTC 21:53:18 <colin-> will do 21:54:00 <strigazi> flwang: what might help a little is using the config drive before the metadata service 21:54:05 <strigazi> colin-: thx 21:55:03 <flwang> strigazi: does that need change in magnum? 21:55:14 <flwang> imdigitaljim: did you ever see this issue https://github.com/kubernetes/cloud-provider-openstack/issues/280 ? 21:55:16 <strigazi> imdigitaljim: eandersson colin- what is your experience with CPO 21:55:31 <flwang> worker nodes are missing ips 21:55:35 <imdigitaljim> i have not experienced this issue 21:55:36 <strigazi> flwang: in the config of the CPO 21:55:48 <flwang> imdigitaljim: probably because you're using v1.12? 21:55:57 <imdigitaljim> we have an internal downstream with a few patches on CPO 21:56:06 <imdigitaljim> were waiting on upstream commit permission for kubernetes org 21:56:12 <imdigitaljim> (blizzard admin stuff) 21:56:20 <imdigitaljim> we're on 1.12.1 correct 21:56:24 <strigazi> patches regarding this bug? 21:56:30 <imdigitaljim> no 21:56:51 <imdigitaljim> UDP support, LBaaS naming, and a LBaaS edge case 21:56:59 <flwang> https://github.com/kubernetes/kubernetes/pull/65226#issuecomment-431933545 21:57:04 <colin-> to jim, outloud just now i said "it's been better than starting from scratch"? :) 21:57:19 <colin-> found it useful and it has definitely saved us some time, but as he said we've also found some gaps we want to address 21:57:34 <flwang> it seems like a very common, high-chance problem 21:58:10 <strigazi> colin-: you talk about k/cpo? 21:58:15 <colin-> yes 21:58:34 <flwang> strigazi: when you say 'config of CPO', does that mean we at least have to use cm+CPO mode? 21:59:27 <strigazi> flwang: https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/write-kube-os-config.sh#L12 22:00:08 <strigazi> it is adding [Metadata]search-order=configDriver,metadataService 22:01:11 <flwang> strigazi: got it. but based on https://github.com/kubernetes/cloud-provider-openstack/issues/280#issuecomment-427416908 22:01:57 <flwang> does that mean we only need add this one [Metadata]search-order=configDriver,metadataService ? 22:02:25 <strigazi> flwang: the code is... even if you disable set [Metadata]search-order=configDrive it still calls the metadataservice 22:03:11 <strigazi> I tried with configdrive only and it was still doing calls to the APIs. 22:03:30 <flwang> i'm confused 22:03:56 <flwang> does that need any change in nova? 22:04:04 <flwang> i mean nova config 22:04:23 <strigazi> I'll end the meeting to be ~1 hour and we continue 22:04:33 <flwang> cool, thanks 22:04:40 <openstackgerrit> Merged openstack/magnum master: Minor fixes to re-align with Ironic https://review.openstack.org/612748 22:04:53 <strigazi> @all thanks for joining the meeting 22:05:02 <strigazi> see you next week 22:05:22 <strigazi> #endmeeting