Tuesday, 2018-06-12

*** yamahata has joined #openstack-meeting-500:14
*** ricolin__ has joined #openstack-meeting-502:06
*** yamahata has quit IRC02:17
*** yamahata has joined #openstack-meeting-502:46
*** piotrrr has quit IRC03:21
*** piotrrr has joined #openstack-meeting-503:27
*** ianychoi has joined #openstack-meeting-503:51
*** yamamoto has joined #openstack-meeting-504:08
*** ricolin__ has quit IRC04:21
*** ricolin has joined #openstack-meeting-504:22
*** radeks has joined #openstack-meeting-504:24
*** radeks has quit IRC05:05
*** MarkBaker has quit IRC05:07
*** radeks has joined #openstack-meeting-505:37
*** radeks has quit IRC05:50
*** slaweq has joined #openstack-meeting-507:11
*** radeks has joined #openstack-meeting-507:16
*** slaweq has quit IRC07:43
*** derekh has joined #openstack-meeting-508:17
*** slaweq has joined #openstack-meeting-509:32
*** radeks has quit IRC09:46
*** slaweq has quit IRC09:47
*** radeks has joined #openstack-meeting-509:55
*** ricolin has quit IRC09:57
*** slaweq has joined #openstack-meeting-510:20
*** slaweq has quit IRC10:34
*** slaweq has joined #openstack-meeting-511:02
*** slaweq has quit IRC11:06
*** slaweq has joined #openstack-meeting-511:14
*** slaweq has quit IRC11:16
*** cristicalin has joined #openstack-meeting-512:09
*** roman_g has joined #openstack-meeting-512:11
*** mjturek has joined #openstack-meeting-513:03
*** mjturek has quit IRC13:03
*** mjturek has joined #openstack-meeting-513:06
*** slaweq has joined #openstack-meeting-513:10
*** slaweq_ has joined #openstack-meeting-513:13
*** slaweq has quit IRC13:16
*** kman has joined #openstack-meeting-513:16
*** cristicalin has quit IRC13:19
*** felipemonteiro_ has joined #openstack-meeting-513:23
*** felipemonteiro_ has quit IRC13:29
*** yamamoto has quit IRC13:36
*** yamamoto has joined #openstack-meeting-513:37
*** slaweq_ has quit IRC13:57
*** sgrasley has joined #openstack-meeting-514:25
*** slaweq has joined #openstack-meeting-514:29
*** slaweq has quit IRC14:36
*** hongbin has joined #openstack-meeting-514:43
*** slaweq has joined #openstack-meeting-514:45
*** kman has quit IRC14:45
*** gagehugo_ has joined #openstack-meeting-514:49
*** gmmaha has joined #openstack-meeting-514:57
*** slaweq has quit IRC14:59
mattmceuen#startmeeting openstack-helm15:00
openstackMeeting started Tue Jun 12 15:00:13 2018 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack-helm)"15:00
openstackThe meeting name has been set to 'openstack_helm'15:00
mattmceuen#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: openstack-helm)"15:00
rwellumo/15:00
mattmceuenAgenda for today:  https://etherpad.openstack.org/p/openstack-helm-meeting-2018-06-1215:00
gmmahao/15:00
piotrrro/15:01
mattmceuenplease go ahead and add anything you'd like to discuss.  We have a pretty full agenda (including some things that got pushed from last time)15:01
mattmceuenGM all15:01
*** radeks has quit IRC15:01
portdirectO/15:01
mattmceuenAlright -- first15:01
mattmceuen#topic Storyboard15:02
*** openstack changes topic to "Storyboard (Meeting topic: openstack-helm)"15:02
mattmceuenAs you're all probably aware, we switched from Launchpad to Storyboard on Friday!15:02
mattmceuenHere is our project group: https://storyboard.openstack.org/#!/project_group/6415:02
mattmceuenFrom there you can click on different projects for osh, osh-infra, osh-addons15:02
rwellumYay!!15:03
mattmceuenWe also have a Worklist for OSH 1.0, which is something we really needed: https://storyboard.openstack.org/#!/worklist/34115:03
portdirectW00t15:03
mattmceuenThis list captures all* the outstanding items from the OSH 1.0 spec that have yet to be completed15:04
gmmaha\o/15:04
mattmceuen* I haven't added the documentation updates needed -- that still needs to be added15:04
mattmceuenSo y'all are aware of the structure15:04
mattmceuenThere are a bunch of stories15:04
mattmceuenstories have some number of tasks15:04
mattmceuenthe worklist pulls together the tasks15:04
mattmceuenAnd if you start tagging your PS commit msg with "Task: XXXX" it will update task status automatically15:05
portdirectSeems tasks are a one merge thing?15:05
mattmceuenAnd if you tag your PS commit msg with "Story: XXXX" it will link back to the story from the PS15:05
mattmceuenYes - I think you are right portdirect15:05
mattmceuenOne task per PS15:05
portdirectSo we may need to create more tasks as we go for bigger items15:05
mattmceuenPlus tasks for things other than PS when those come up15:05
mattmceuen+++15:05
mattmceuenThere were a few tasks I put out there that I prefixed with "Spike: "15:06
mattmceuenWhich are explicit investigation tasks for creating "work" tasks, because I am lazy^H^H^H a great PTL15:06
rwellum+!15:06
*** wxy| has joined #openstack-meeting-515:07
mattmceuenI'll be honest - putting it all down in storyboard gave me a new appreciation for the volume of work standing between us an 1.015:07
mattmceuenThere is no rocket science there15:07
mattmceuenJust a lot of tractable things15:07
mattmceuenSo if anyone has been looking for something to sink their teeth into in the code (or soon, docs) -- WE NEED YOU!!!15:07
mattmceuen:)15:07
mattmceuenPlease go grab a task here (assign it to yourself)  and then we will help you succeed  at it https://storyboard.openstack.org/#!/worklist/34115:08
rwellumCool - great job Matt15:08
mattmceuenThanks rwellum and thank you for your help getting us here too15:08
mattmceuenThat's all I got, anything else storyboard related guys?15:08
srwilkerso/15:09
alanmeadowso/15:09
mattmceuen#topic Discuss multi-node deployer feedback15:09
*** openstack changes topic to "Discuss multi-node deployer feedback (Meeting topic: openstack-helm)"15:09
mattmceuenGM folks!15:09
mattmceuenrwellum -  you've been going through some deployment activities using the multi-node guide15:10
rwellumRegarding this, I spoke to a couple of people at the Summit - mainly from kolla-k8s, and we've all tried the multinode and all had failed - but in a good way - as in we learned but had more questions.15:10
rwellumMain question was environment - the guide is heavily linked to the Gate scripts correct?15:11
portdirectyup15:11
portdirectsimply as no one had had the time to write better docs15:11
portdirectand so we knew we had somthing that worked15:11
rwellumYeah understood.15:11
portdirectabit in a highly opinionated way15:11
mattmceuenwhat kinds of issues did you run into rwellum?15:11
rwellumI'll reach out to them and I'll gather more concrete evidence and then ask for advice. I need to redeploy as well and I know Matt is doing it now too.15:12
rwellumMainly assumptions - missing packages for example.15:12
portdirectthis heat template, though horrible may help: https://github.com/portdirect/openstack-helm-dev/blob/master/osh-cluster.yaml15:12
portdirecti use it to deploy a multinode osh cluster 2-3 times a week15:13
rwellumWill take a look for sure15:13
mattmceuenI started going through the multi-node guide this weekend for the first time in a bit, and ran into a couple User Error issues, a couple minor doc updates I'll make, and an ingress thing that I will figure out next.  For me though there was a fair amount of user error stemming from reusing a previously-used environment and not reading all the instructions 100% carefully :D15:13
mattmceuenThe "assumptions" part is the most likely and insidious thing15:14
rwellumI think also there's a bit of ambiguousness around which is the master node, and which is a minion. At least that was me - as I tried to make the scripts work with my setup.15:15
mattmceuenThat is probably something that can be called out more explicitly - agree15:15
mattmceuenI'll add that into my "minor updates" PS15:16
rwellumSounds good.15:16
mattmceuenWould it help to have an etherpad to throw these things on as they come up?15:16
rwellumYeah for sure.15:16
mattmceuenhttps://etherpad.openstack.org/p/openstack-helm-multinode-doc15:18
mattmceuenLet's do it - I want to get the doc all trued up by next week15:18
roman_go/15:19
roman_gGood morning.15:19
mattmceuenhey roman_g15:19
mattmceuenAnything else on the multinode guide right today rwellum, or are we good to has through the update process?15:19
rwellumNo that's fine thanks15:20
mattmceuenCool beans - you have the next one too15:20
mattmceuen#topic Vitrage chart15:21
*** openstack changes topic to "Vitrage chart (Meeting topic: openstack-helm)"15:21
mattmceuenHow is Vitrage looking!15:21
rwellumI have commitment from my team to start working on this June 20th. Ultimately I want to get team members on-board with osh through this effort.15:21
sgrasleystack15:21
sgrasleysry wrong window15:22
rwellumAlso I have commitment to run a POC for OSH - replacing TripleO - so quite a bit of OSH involvement hopefully.15:22
mattmceuenAwesome rwellum - agree, and if there's anything we can do to help your team members get off the ground let us know15:22
*** cfriesen has joined #openstack-meeting-515:23
mattmceuennice!15:23
rwellumWhen we kick off I'll invite them to this meeting and do a round of intro's if that's ok?15:23
mattmceuendefinitely.15:23
rwellumcool then.15:23
mattmceuenHey sgrasley o/ no worries :)15:23
rwellumThat was it for me. Except I need to get multinode up and running to show them how easy it is to set up and use etc.15:24
mattmceuenThe proof is in the pudding15:24
mattmceuenLet's get all issues into that etherpad stat so we can get things solved and/or clarified - I will do this as well15:25
rwellumOr 'hardtack'15:25
mattmceuenhttps://etherpad.openstack.org/p/openstack-helm-multinode-doc15:25
portdirect+++15:25
portdirectwithout logs/info its very hard if not impossible to help15:25
rwellumAh - that's why we started to put the debug guide together right?15:26
rwellumhttps://etherpad.openstack.org/p/openstack-helm-troubleshooting15:26
mattmceuenYeap they're kind of sister things15:26
mattmceuenIn the sense of -- some issues should result in doc updates15:26
mattmceuenAnd some issues shouldn't, but people will run into them anyway15:27
rwellumSo our job is to feed back good info for you guys - so we don't waste your time with basic questions etc.15:27
mattmceuenAnd we should get those into the troubleshooting guide15:27
rwellumAgreed15:27
portdirectjust info would be good - theres a lot of discussion of difficulty - but no direction on what we should be looking at yet :)15:28
mattmceuenAs a rule of thumb, if someone runs into an issue it's either15:28
mattmceuen1) They didn't read (guilty)15:28
mattmceuen2) The doc wasn't clear enough15:28
mattmceuen3) It's just something that happens, maybe with #1 or #2, that should be documented in the troubleshooting guide15:28
mattmceuenSo capturing all of them is a good start15:28
portdirect4) they installed on a dirtly env15:28
portdirectso brought in a load of pre-existing issues15:28
mattmceuenAlrighty moving on15:29
mattmceuenalanmeadows15:29
mattmceuen#topic Production vs Bare Bones values.yaml15:29
*** openstack changes topic to "Production vs Bare Bones values.yaml (Meeting topic: openstack-helm)"15:29
mattmceuenWe didn't get a chance to discuss last time15:29
mattmceuenPaint us the picture here15:30
alanmeadowssure15:31
alanmeadowsthis was discussed a bit in the following PS: https://review.openstack.org/#/c/543012/15:31
alanmeadowsHistorically, the ask has always been: let's try and have OSH run OOTB as it would in production, otherwise we sweep operator issues under the rug15:32
alanmeadowschallenging things such as, making ceph a first class citizen, enabling HA by default and so on15:32
alanmeadowsas the developer audience grows, and we have solved for many of these already, it makes sense to prune back this mandate a bit15:33
alanmeadowsvalues.yaml can instead be the bare bones necessary to stand an openstack/openstack-helm environment up, while we continue to test more multi-node/high availability options elsewhere15:34
alanmeadowswhether thats a "production.yaml" armada gate or whatever we decide15:34
portdirect++15:34
rwellumWhy? As time goes by and you solve many of the issues - surely it becomes easier to deploy as full production?15:35
portdirectrwellum: good example if things like those raised in the above ps15:35
alanmeadows@rwellum: the resources necessary for the deployment grow and grow if we continue to exercise full high availability as the `default`15:35
alanmeadowslimiting our potential developer pool15:35
portdirectwhere 'production like' setting demand specific (and large) hardware configs15:36
alanmeadowsthe benefit here is two fold15:36
alanmeadowswe slim down whats needed to deploy it OOTB, default values becomes easy to explain, and there is a path for deploying this in production that both has optimal settings and uses an optimal mechanism (armada)15:36
alanmeadows(e.g. you shouldn't be `$ helm install` (x30) in production)15:37
rwellumSo by 'full production' you mean as portdirect is hinting - large scale?15:37
mattmceuenIs it fair to say that we want to15:37
mattmceuen1) enable the defaults to run on a wide variety of deployment types (e.g. laptop)15:37
mattmceuen2) we want to enable as much prod-like / HA functionality as will run across that range by default15:37
mattmceuen3) the remining prod-specific overrides would be captured in an override yaml15:37
portdirectrwellum: your touching on the core issue15:38
portdirectProduction = opionion15:38
alanmeadowsthe term is a little loose.. but how any sane person would generally want to deploy this thing into either a lab or production15:38
cfriesenalanmeadows: what about small-scale "production"?  like dual-node?15:38
alanmeadowsand whenever you say that there is... opinion involved :)15:38
alanmeadowsI think what we want to avoid is the helm community spending lots of time keeping dozens of references `up to date` or having some of them languish15:39
alanmeadowsI think ideally we would maintain developer values and a `large scale` and `resilient` production manifest15:39
alanmeadowsanything in the middle would be an exercise to the user based on the production manifest15:39
*** yamamoto has quit IRC15:39
rwellumI think you don't want to debase your product by making it so simple to deploy that it's not recognizably a realistic OpenStack deployment. I think Armada is amazing, but if you're telling people OSH is so complicated in production you need this tool - then I think it's a not a great message.15:40
*** yamamoto has joined #openstack-meeting-515:40
portdirectWe need to work on messaging it seems15:41
rwellumit's like I showed my team OSH running on a VM, and they loved it but they immediately asked me about multi-node, real hardware, CEPH, SRIOV etc.15:41
portdirectBut what we are trying to say is the opposite15:41
alanmeadowsrwellum: its a fair point, but we've already received enough messaging at workshops to demonstrate there is need to educate people on at least how we envision this running and being *maintained* in production15:41
alanmeadowsquestions such as "would we be running all of these bash commands in production" like confusion15:41
mattmceuenYep15:42
rwellumI see..15:42
piotrrrwhy are we assuming that small scale, but HA-capable, setup would be too big for developers? I mean, one thing we could aim at is have one of the default deployments be HA-capable, small-scale (dev friendly), but at the same also easily overridable (customizabale) for large scales.15:42
rwellumYes well said piotrrr15:42
mattmceuenI don't think we should strip out prod-grade configuration out of the defaults until it becomes constrictive.15:42
mattmceuenI.e. we want for the defaults to run on a laptop, basically.  A lot of HA functionality can run on a laptop, and that's exactly the dev environment I want15:43
piotrrryup, +115:43
alanmeadowswell keep in mind folks, to be HA/resilient/so on or not is a simple matter of overrides15:43
alanmeadowsnothing changes about the core product15:43
portdirectand to be honest - we are pretty much at the base level that we are refering to15:44
alanmeadowsanywhere these are removed to stream line resource usage has to be gated somewhere though (again as the PS says)15:44
portdirectthis is more about adding functionaility via over-rides and tooling15:44
alanmeadowswe cant remove them until something else is caring for them, so this conversation is really just starting a dialog about what that thing is15:44
alanmeadowsthe last thing we want is to remove something like l3 ha being a default and then find out it doesnt work when someone turns it on15:45
*** yamamoto has quit IRC15:45
portdirect+++15:46
mattmceuenAgree15:46
portdirectand what is super ironic here - we turned that off in the gate anyway15:46
portdirectso it was *never* tested  - lol15:46
mattmceuenInstead of thinking about it as "resilient overrides" vs "dumbed down defaults"  I'd think of it as "generically resilient defaults" vs "any additional prod-only overrides"15:46
mattmceuensigh15:47
alanmeadowslets take a another perfect example that requires extra oomph to validate15:47
alanmeadowsmultiple mariadb replicas...15:47
alanmeadowssomething needs to validate this functions15:47
alanmeadowsbut I certainly dont want to demand developers have mariadb(x3)15:47
alanmeadowswhich means it shouldn't be OOTB15:48
alanmeadowsbut it should be part of a production manifest we gate on15:48
portdirectand even then - you may find some production deployments that dont want galera15:48
mattmceuen+115:48
rwellumBut mariadb 3x is nothing to do with 'production' is it?15:48
portdirectexactly rwellum15:48
rwellumIt's more like a scale test15:48
piotrrralan, fair point, but what is bad about having 3x mariadb in a dev environment? is it only about memory usage, or is it something else?15:49
portdirectno  - its opinion - we need to validate our chart works with 3x replication15:49
portdirectbut wheter its required for production (i think it is) is up to the de15:49
alanmeadowsrwellum: I suppose different ways of looking at it, and perhaps this is where the opinion comes in -- mariadb (x3) is production for us and I think somewhat typical15:49
alanmeadows(for galera users)15:49
rwellumI see. Trying not to get to hung up on the word production here.15:50
mattmceuenpiotrrr that is an excellent point - and for lighterweight services than mariadb, I'd argue it may make sense to have x3 replicas on a single node IMO15:50
alanmeadowspiotrrr: I think the point is you take that example and multiply it by 30 other things like that, and you get a heavy deployment15:50
alanmeadowsor maybe your laptop is really big :)15:50
rwellumAlso I am unsure about mattmceuen point about most users want to run on a laptop - I think that's so limiting, and you're not trying to replace stackdev right?15:51
mattmceuenI suppose we could have a constrained gate that makes sure that the defaults work well with only e.g. 16GB RAM :)15:51
portdirectmattmceuen: 8gb ram in gate15:51
rwellumIn other words the base package shouldn't be delimited by running on a laptop. imo,.15:51
mattmceuenMy thought was more that it should be the settings that run "anywhere", and running on a laptop helps bring that into relief15:51
mattmceuenas it doesn't get much more constrained than that15:52
mattmceuenand it still is a very common use case, at least for folks doing OSH dev15:52
rwellumYeah but not sure I agree - don't pick the lowest common denominator to define your product.15:52
piotrrrthere's not that many services in openstack which need to be configured extremely different way between non-HA vs HA deployments. e.g. nova-api doesn't care. you could have a OOTB lightweigh HA-deployment  with 3xmariadb and any number of nova-api-like-services (even 1).15:52
piotrrrso, services which require different config to run in HA- deploy them in HA by default. Other services, those that easily scale in active-active manner, allow people to choose how many they want. Dev will choose "1". production will choose "3" or more.15:53
alanmeadowsI think whats great about OSH is values.yaml in each chart != product15:54
rwellumTBF piotrrr alanmeadows is discussing a 'bare-bones' solution.15:54
portdirectwhat we want is osh to work out of the box for everyone15:55
portdirectand have a robust set of documentation, gating and examples for various deployments with it15:55
alanmeadowsno one in any kind of environment, whether dual-host HA or a lab, or production is going to run it without overriding it15:55
portdirect+++15:55
alanmeadowsso we need to show them the way on how thats manageable15:55
alanmeadowsand for all others, who are OOTB, they must be developers15:56
alanmeadowsI think if we align on that, we may start seeing eye-to-eye15:56
portdirectand a big part of 'production ready' is day215:56
alanmeadowsdevelopers or just kicking the tires, I should say15:56
portdirecthow do we manage this thing15:56
portdirectosh makes openstack simpler15:56
portdirectbut its still openstack15:56
alanmeadowsno less than a million knobs to turn ;-)15:57
portdirectand requires careful consideration of what you are deplying15:57
rwellumSmall, medium and large working out the box.15:57
rwellumExamples on how to overide anything15:57
rwellumvoila?15:58
portdirectkinda rwellum15:58
portdirectthough we need to work out what small, med and large are15:58
portdirect:)15:58
alanmeadowswhat we were kicking around is a small works OOTB (remember there is only one box)15:58
portdirectmedium = you need to configure neutron..15:58
alanmeadowslarge is cared for with an example and gated armada manifest, turning on all typical large knobs to show they work, while simultaneously serving as an example of how to stand it up with a single command, manage it day 2, and override anything15:59
mattmceuenSpeaking of boxes15:59
mattmceuentimeboxes15:59
mattmceuenout of time - thanks everyone :)15:59
cfriesenspeaking of that...I've asked on the main channel a few times about how to configure the nova-compute chart to handle heterogeneous compute nodes, multiple sizes of hugepages, PCI devices, dedicated CPUs, etc.  Can anyone point me at discussions/documentation on how this is supposed to be done?16:00
mattmceuenGreat discussion, we can keep it going in the OSH chat room but I think we're coaliescing16:00
mattmceuen#endmeeting16:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"16:00
openstackMeeting ended Tue Jun 12 16:00:10 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-12-15.00.html16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-12-15.00.txt16:00
*** gmmaha has left #openstack-meeting-516:00
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-12-15.00.log.html16:00
mattmceuensorry to cut you off cfriesen16:00
cfriesenno worries16:00
*** cfriesen has left #openstack-meeting-516:01
*** gagehugo_ has left #openstack-meeting-516:01
*** yamahata has quit IRC16:03
*** slaweq has joined #openstack-meeting-516:05
*** notmyname has quit IRC16:11
*** notmyname has joined #openstack-meeting-516:15
*** slaweq has quit IRC16:26
*** yamahata has joined #openstack-meeting-516:29
*** yamamoto has joined #openstack-meeting-516:42
*** yamamoto has quit IRC16:49
*** wxy| has quit IRC16:49
*** radeks has joined #openstack-meeting-516:54
*** derekh has quit IRC17:00
*** sshank has joined #openstack-meeting-517:02
*** radeks has quit IRC18:01
*** slaweq has joined #openstack-meeting-518:21
*** spzala has joined #openstack-meeting-518:22
*** yamamoto has joined #openstack-meeting-518:47
*** slaweq has quit IRC19:22
*** sshank has quit IRC19:27
*** slaweq has joined #openstack-meeting-520:22
*** slaweq has quit IRC20:22
*** yamamoto has quit IRC20:23
*** mjturek has quit IRC20:48
*** mjturek has joined #openstack-meeting-520:49
*** mjturek has quit IRC20:49
*** yamamoto has joined #openstack-meeting-521:24
*** yamamoto has quit IRC21:39
*** slaweq has joined #openstack-meeting-521:43
*** slaweq has quit IRC21:48
*** hongbin has quit IRC23:15

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!