Tuesday, 2018-06-19

*** yamamoto has joined #openstack-meeting-500:47
*** yamamoto has quit IRC00:51
*** yamamoto has joined #openstack-meeting-501:47
*** yamahata has quit IRC01:50
*** yamamoto has quit IRC01:53
*** roman_g has quit IRC02:22
*** yamamoto has joined #openstack-meeting-502:49
*** yamamoto has quit IRC02:55
*** ricolin has joined #openstack-meeting-503:06
*** yamamoto has joined #openstack-meeting-503:51
*** yamamoto has quit IRC03:57
*** yamamoto has joined #openstack-meeting-504:01
*** hoangcx has joined #openstack-meeting-504:15
*** hoangcx has left #openstack-meeting-504:16
*** radeks has joined #openstack-meeting-504:32
*** markvoelker has quit IRC05:01
*** sgrasley has quit IRC05:29
*** yamahata has joined #openstack-meeting-506:17
*** markvoelker has joined #openstack-meeting-507:01
*** slaweq has joined #openstack-meeting-507:02
*** shananigans has quit IRC07:33
*** markvoelker has quit IRC07:36
*** shananigans has joined #openstack-meeting-507:38
*** roman_g has joined #openstack-meeting-508:05
*** markvoelker has joined #openstack-meeting-508:33
*** derekh has joined #openstack-meeting-508:44
*** ricolin has quit IRC08:47
*** radek__ has joined #openstack-meeting-508:53
*** markvoelker has quit IRC09:06
*** markvoelker has joined #openstack-meeting-510:03
*** radek__ has quit IRC10:18
*** markvoelker has quit IRC10:37
*** yamamoto has quit IRC10:50
*** roman_g has quit IRC10:56
*** roman_g has joined #openstack-meeting-510:56
*** yamamoto has joined #openstack-meeting-510:58
*** yamamoto has quit IRC10:58
*** radeks has quit IRC11:24
*** markvoelker has joined #openstack-meeting-511:34
*** yamamoto has joined #openstack-meeting-511:48
*** markvoelker has quit IRC12:01
*** markvoelker has joined #openstack-meeting-512:02
*** yamamoto has quit IRC12:15
*** sgrasley has joined #openstack-meeting-512:42
*** yamamoto has joined #openstack-meeting-512:46
*** TuanVu has joined #openstack-meeting-513:21
*** radeks has joined #openstack-meeting-513:27
*** radek__ has joined #openstack-meeting-513:27
*** yamamoto has quit IRC13:34
*** mjturek has joined #openstack-meeting-513:42
*** d|k has quit IRC13:45
*** d|k has joined #openstack-meeting-513:45
*** hongbin has joined #openstack-meeting-513:53
*** yamamoto has joined #openstack-meeting-513:55
*** yamamoto has quit IRC13:55
*** yamamoto has joined #openstack-meeting-513:57
*** TuanVu has quit IRC14:02
*** wxy| has joined #openstack-meeting-514:06
*** felipemonteiro_ has joined #openstack-meeting-514:15
*** yamamoto has quit IRC14:16
*** felipemonteiro__ has joined #openstack-meeting-514:17
*** felipemonteiro_ has quit IRC14:21
*** yamamoto has joined #openstack-meeting-514:30
*** yamamoto has quit IRC14:35
*** TuanVu has joined #openstack-meeting-514:36
*** felipemonteiro__ has quit IRC14:43
*** TuanVu_ has joined #openstack-meeting-514:44
*** yamamoto has joined #openstack-meeting-514:46
*** TuanVu has quit IRC14:48
*** yamamoto has quit IRC14:50
*** slaweq has quit IRC14:53
*** slaweq has joined #openstack-meeting-514:54
*** gagehugo has joined #openstack-meeting-514:59
mattmceuen#startmeeting openstack-helm15:00
openstackMeeting started Tue Jun 19 15:00:41 2018 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack-helm)"15:00
mattmceuen#topic rollcall15:00
openstackThe meeting name has been set to 'openstack_helm'15:00
*** openstack changes topic to "rollcall (Meeting topic: openstack-helm)"15:00
*** yamamoto has joined #openstack-meeting-515:01
mattmceuenGM all15:01
mattmceuenHere's our agenda for today's OpenStack-Helm meeting: https://etherpad.openstack.org/p/openstack-helm-meeting-2018-06-1915:01
rwellumo/15:01
mattmceuenPlease add anything to it you'd like to discuss, PS needing review, etc!15:01
radekso/15:01
mattmceueno/15:01
srwilkerso/15:02
portdirecto/15:02
rwellummattmceuen: radeks is from my team, we're going to be slowly attending more and doing more with osh as I've spoken about before.15:03
mattmceuenWelcome radeks!15:03
mattmceuenGlad to have you with us :)15:03
roman_go/15:04
mattmceuenAlrighty - first off do we have Robert Choi in the house?15:04
*** jgu has joined #openstack-meeting-515:04
mattmceuen(o/ roman_g!)15:04
mattmceuenHe's got the first agenda item, but we can come back to him if he's not on yet.15:04
mattmceuen#topic Doc Update Touchpoint15:05
*** openstack changes topic to "Doc Update Touchpoint (Meeting topic: openstack-helm)"15:05
*** radeks_ has joined #openstack-meeting-515:05
mattmceuenFor anyone new around here, our documentation is a key area we want to enhance, grow, and groom15:05
portdirectfrom reading the etherpad, im not sure if he or jayahn are here today?15:05
*** yamamoto has quit IRC15:05
portdirect+++ to that mattmceuen :)15:05
mattmceuenoh, maybe I misunderstood "next week" -- maybe Robert meant next next week :)15:05
mattmceuenWe've been capturing some enhancements that should be made to the Multinode Install guide here: https://etherpad.openstack.org/p/openstack-helm-multinode-doc15:06
jayahni am here, but stuck in different meeting and task.15:07
mattmceuenI've started implementing a number of bits in this PS: https://etherpad.openstack.org/p/openstack-helm-multinode-doc15:07
mattmceuenHey jayahn!15:07
jayahnwe meant next week irc meeting.15:07
*** radeks has quit IRC15:07
roman_gI saw PS from Matt, would review today/tomorrow. One thing I would add is answer to the question15:07
mattmceuenCool thanks for clarifying :)  for now I'll just paste in your agenda note for general awareness at the end, since that's good to be aware of15:08
mattmceuen^ jayahn15:08
roman_g"What to do next once installation finishes?"15:08
mattmceuenThanks in advance roman_g15:08
rwellumWrong link mattmceuen ?15:08
portdirectroman_g: use openstack - and be happy?15:09
lamto/ running late15:09
srwilkersportdirect: hehe15:09
mattmceuenwhoops15:09
roman_gportdirect: Matt: add link to https://docs.openstack.org/openstack-helm/latest/install/developer/exercise-the-cloud.html to the bottom of Multinode install page.15:09
mattmceuenSomeday I will master the art of copy and paste15:09
mattmceuenhttps://review.openstack.org/#/c/576342/15:09
roman_g@sigit in Slack was asking on #openstack-helm on Thursday, where could he get openrc file15:11
roman_gand that exercise-the-cloud.html is the answer15:11
mattmceuenSo for that one roman_g, I think we could potentially copy a use-it script to the multinode install batch of scripts, rather than cross-linking from the multinode guide back to the dev scripts.15:12
roman_gor just move this file from /developer/ subdir to one level up to /install/15:13
portdirectroman_g: we just need to get better at refacoring what we have15:13
mattmceuenHowever, the use-it script makes some assumptions around network setup, etc... not sure if that's a good idea for a multinode setup or not.  What do you think portdirect15:13
portdirectfor example the openrc is at the bottom of here: https://docs.openstack.org/openstack-helm/latest/install/developer/exercise-the-cloud.html15:13
portdirectmattmceuen: totally agree15:13
portdirectat the end of the multinode guide - we should have a 'real' openstack deployment15:13
mattmceuenI suppose we could leave the scripts as-is and then just say at the bottom of the multinode guide something like, "for examples of how to exercise your new OSH cloud, please see <link> the developer guide"15:14
portdirectwhich means the dev-kick-the-tyres script wont be relevant in 99% of cases.15:14
portdirectwe should find actual openstack docs on cloud use - and link there15:14
portdirectif they dont exist, lets make them15:15
portdirectas it should be the same regardless of deployment system: osh, osa, kolla, trippleo etc15:15
*** yamamoto has joined #openstack-meeting-515:16
mattmceuen++15:16
mattmceuenI added that to the etherpad15:17
mattmceuenrwellum, I haven't gone through your additions to the etherpad yet in detail15:18
*** TuanVu has joined #openstack-meeting-515:18
mattmceuenDo you want to talk through them in brief here?15:18
rwellumYeah - for the most part it's digging out 'things' from the various playbooks that I was taking for granted when running the AIO15:19
rwellumI mainly have issues with ceph - I still think some fundamentals are missing15:20
mattmceuenWhat are you working toward with that - do you see the assumptions making it back into the multinode guide?  Or something standalone?15:20
*** yamamoto has quit IRC15:20
rwellumI am a little on the fence here - because to take the guide away from executing the gate scripts is quite a big step.15:21
*** TuanVu_ has quit IRC15:21
rwellumMaybe a third guide is in order?15:21
portdirectim not sure i quite agree15:21
portdirectin that once you had k8s setup, with hosts able to resolve k8s services dns, and ceph-common on the host15:22
portdirectwhat more is/was required, other than building the charts?15:22
rwellumNothing - agreed15:22
rwellumYeah I regressed :( - I don't know why ceph is acting up for me now.15:23
portdirectso from this point on: https://docs.openstack.org/openstack-helm/latest/install/multinode.html#deploy-openstack-helm15:23
portdirectthe guide should be totally agnostic of k8s deployment tooling15:23
portdirectprovided the above criteria are met15:23
mattmceuenSo after the PS above, the next PS I'll put in will split out the AIO setup entirely15:24
rwellumThere's a few assumptions regarding number of nodes etc - in the multinode scripts I believe - or?15:24
portdirectnice - i think getting this in  - even if we just have a stub for the above points will really help15:24
mattmceuenThe idea is that the new multinode guide will still link to it as an example way to set up a dev-grade k8s cluster, but also to link out to more legit ways to set up a k8s cluster for prod use15:25
portdirect++15:25
*** TuanVu has quit IRC15:25
rwellumok I can buy into that15:26
mattmceuenThat may help us refer out to good ways to stand up clusters independently of the rest of the guide, and avoid guide sprawl15:26
portdirectagreed - we are already at risk of that - eg the multiple places we tell you how to set up sudo :)15:27
mattmceuenI'll take a look at your material in there rwellum and chew on it as well with an eye toward how it can best fit in15:27
mattmceuenSo speaking of the troubleshooting doc - we also have an etherpad for that one https://etherpad.openstack.org/p/openstack-helm-troubleshooting15:28
rwellumOk - I also will document how I create the k8s cluster - as a potential example - but also on the fence because really - we don't need another k8s deployer :)15:28
rwellumAnd it's as portdirect says - osh is agnostic15:28
mattmceuenDef don't spend too much time on anything that you don't think would be valuable to others, though rwellum - some things are always going to be operator-specific15:29
portdirectrwellum: can we not just link to kubead, kubespray and other community proejcts?15:29
srwilkersportdirect: that would be the sane thing to do15:29
roman_glink to the kubernetes - the hard way ;)15:30
mattmceuenHAHA15:30
portdirect^^ actually we 100% should15:30
srwilkersi think it's important we reduce how much kubernetes specific documentation/support we offer15:30
srwilkersthe hard way is how i learned15:30
*** yamamoto has joined #openstack-meeting-515:31
srwilkersit's actually pretty good15:31
portdirectas virtually every deployment tool is based on it15:31
roman_gsame here15:31
rwellumYeah agreed15:31
mattmceuenYup I'm planning on linky-ing in the next PS, if you have any recommendations for good installers / guides etc for prod use please add them to the multinode etherpad!15:31
roman_git's nearly production install.15:31
mattmceuenSo for troubleshooting: https://etherpad.openstack.org/p/openstack-helm-troubleshooting15:32
mattmceuenAt the very very bottom I stashed a couple of errors I ran into during my recent multinode install adventure15:32
mattmceuenSymptom: one MDS started fine, but another died, complaining in the logs about a feature flags incompability.15:32
mattmceuenCause:  This error was caused by an old version of `docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04` being cached on one node, which was incompatible with a newer version on another node.15:32
mattmceuenResolution:  Pull the updated docker image on the node with the sad mds15:32
mattmceuenError message:  mds.mds-ceph-mds-65bb45dffc-qfqnr handle_mds_map mdsmap compatset compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2} not writeable with daemon features compat={},rocompat={},incompat={1=base v0.20,2=client writeable15:32
mattmceuenranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=file layout v2}, killing myself15:32
mattmceuenSymptom:  kube-system ingress (host networking) is running and openstack ingress (non-host networking) is failing15:32
mattmceuenCause:  The default calico pod subnet conflicts with a preexisting subnet in this environment15:32
mattmceuenResolution:  in the multinode-vars.yaml file, override the default via `kubernetes_cluster_pod_subnet: 10.25.0.0/16`15:32
mattmceuenError message: Readiness probe failed: Get http://192.168.23.131:10254/healthz: dial tcp 192.168.23.131:10254: getsockopt: connection refused15:32
srwilkerswell, i dont know if i'd say a production installation necessarily, but i like that it gives people exposure to whats going on, instead of just throwing kubeadm at the wall and getting a cluster15:33
srwilkersbut thats just me15:33
mattmceuenIRC rendered that sadly -- that's two different errors.  But the point is that I think this might be a straightforward approach to recording and helping people solve common issues15:33
rwellumThere's a 'WHAT'S NEXT!!' placeholder in that guide too - I added, my subtle way of asking for a core who knows osh to add to it.15:33
mattmceuenSymptom / Cause / Resolution / Error Message15:33
rwellum+1 mattmceuen15:33
mattmceuenshort & sweet & googleable15:33
portdirectsounds good15:34
portdirectif we can get a nice start on this15:34
mattmceuenInterested in feedback, is that what a troubleshooting guide should look like?15:34
portdirectthen we should try to answer questions via irc/slack as ps's15:34
portdirectso they are kept as reference15:34
mattmceuenI like that15:34
portdirecti find that 90% of support is answering the same things15:35
rwellumMy thought for the TS guide is that before someone opens a bug on osh, we guide them to this guide.15:35
mattmceuen++15:35
portdirectand its my bad for not documenting them, but this would lower the barrier to that loads mattmceuen15:35
*** yamamoto has quit IRC15:35
mattmceuenCool, I will add those couple errors as a start into the TS guide and we squint at it, and then continually add to it15:36
mattmceuenAlright - anything else on the Doc front before moving along?15:36
mattmceuenGood discussion guys and appreciate all the attention on this since last week15:37
mattmceuenAlrighty srwilkers you're up!15:37
mattmceuen#topic Logging Updates15:37
srwilkerscool15:37
*** openstack changes topic to "Logging Updates (Meeting topic: openstack-helm)"15:37
mattmceuen    fluentbit sidecar for ceph-mon and ceph-osd:  https://review.openstack.org/#/c/575832/15:37
mattmceuen    change to logging.conf for openstack services:  https://review.openstack.org/#/c/576001/15:37
srwilkersi've proposed adding fluentbit sidecars to the ceph-mon and ceph-osd charts, to allow us to gather the logs that get placed in /var/log/ceph in those pods15:38
*** wxy|_ has joined #openstack-meeting-515:38
rwellumMy question here srwilkers was why not make it a default?15:38
portdirectas that creats a hard dependency on fluentd15:39
srwilkersrwellum: it's set as a default currently as we don't deploy fluentd in the single or multinode gates for openstack-helm15:39
srwilkersand what portdirect said15:39
*** wxy| has quit IRC15:39
portdirectand we want these charts to be compose able simply15:40
srwilkersit follows what we've done with the prometheus exporters tied to things like rabbitmq and mariadb -- we leave them disabled by default so as not to create dependencies or assumptions15:40
srwilkersbut for those who want to use it, it provides additional insight into ceph logged events15:40
portdirectalso leaves the door open for people to add alternate log aggregators15:41
rwellumOk makes sense15:41
srwilkersin addition, we can also use the tags on the logged events to possibly add some sane fluentd filters in the future if we want15:42
mattmceuensrwilkers for the logging.conf change - looks awesome; would it prevent openstack logs from going to stdout by default?  I.e. breaking kubectl logs?15:43
srwilkersno.  you can define multiple handlers15:43
mattmceuenneat15:43
rwellumThere's a difference between fluentbit and fluentd though right?15:44
srwilkersbut the cool thing with that change is that for any version >= ocata, we can use the fluent formatters15:44
mattmceuenthat will be really great then15:44
portdirectsrwilkers: I'll get some new images published with that over the next 48 hours15:44
srwilkersrwellum: functionality is the same, but the big difference is that fluentbit has a much smaller resource footprint than fluentd15:44
rwellumYeah was confused because you said fluentbit sidecars15:45
srwilkersso we use fluentbit for the sidecar, then forward the messages to a fluentd serving as an aggregator15:45
rwellumYeah makese sense15:45
rwellumI'll go read the PS again15:45
srwilkersbut the fluent formatter and handler for the openstack services makes me happy, as we can send the logs directly to fluentd15:45
*** yamamoto has joined #openstack-meeting-515:46
srwilkersinstead of stdout > fluentbit > fluentd15:46
srwilkersthis also gets us something we've wanted for awhile15:46
srwilkersusing the fluent formatter and handler gets us full stacktraces when they're raised15:46
mattmceuenwoo hoo!!15:47
srwilkerscan also get us tags for things like the project name, host name, etc15:47
mattmceuenstacktraces - the most important part of logging :)15:47
srwilkersand since the services get a unique tag, we can define filters in fluentd in the future15:47
srwilkersie: do this for nova, do this for neutron, etc15:47
srwilkerscan also take the recent updates to the fluent-logging chart and create multiple indices in elasticsearch, and use the project tags to create indices per openstack service if your heart so desired15:48
srwilkersbut as mentioned, you do need at least ocata to use that formatter15:49
srwilkersbut even if you dont, i still feel this gives an operator greater control over what they want to see and how15:49
rwellumHave to bow out a little early today - will continue with deployment this afternoon and will bug the IRC :)15:49
mattmceuenThanks rwellum!  Looking forward to seeing the success and/or fallout! :)15:50
srwilkersthat's it for me15:50
mattmceuenThanks srwilkers - that's awesome15:50
mattmceuenLooking forward to playing with that :)15:50
*** yamamoto has quit IRC15:50
mattmceuen#topic topics for next time15:50
*** openstack changes topic to "topics for next time (Meeting topic: openstack-helm)"15:50
mattmceuenJust one thing to get in front of y'all -- a request for a discussion next week15:50
mattmceuenSupport multi versions of Rally - Let's have some time to think about it and discuss again next week.15:50
mattmceuenThere is a little more discussion / brainstorming in the agenda https://etherpad.openstack.org/p/openstack-helm-meeting-2018-06-1915:51
mattmceuen#topic Roundtable15:51
*** openstack changes topic to "Roundtable (Meeting topic: openstack-helm)"15:51
mattmceuenWe have 9 minutes left - anything else you would like to discuss?15:51
mattmceuenOr also - any PS urgently needing review?15:52
portdirectoh this is pretty cool: https://review.openstack.org/#/c/570658/15:53
portdirectto be honest, I'm kinda suprised it works at all15:53
portdirectbut would be really nice to have15:53
portdirectI'm not sure how we could gate it15:53
mattmceuenhttps://twitter.com/IanFromATT/status/100881539201655194115:53
portdirectand also really uncomfortable with some of the things it does15:54
mattmceuenI sense a third party gate hooked up to a macbook portdirect15:54
srwilkershttps://review.openstack.org/#/c/575157/15:54
portdirectas messing with a mac deployment (ie doeploying homebrew etc) is kinda bad i think15:54
portdirectas most people wont want to redeploy their mac to clean up an env...15:54
portdirectbut with some changes, could be super valuable15:55
srwilkersportdirect: yeah, im not a huge fan of that one15:55
mattmceuenAlright guys - unless there's anything else I can give you a full 3 minutes back15:56
mattmceuenThanks!15:56
mattmceuen#endmeeting15:56
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:56
openstackMeeting ended Tue Jun 19 15:56:49 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:56
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-19-15.00.html15:56
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-19-15.00.txt15:56
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_helm/2018/openstack_helm.2018-06-19-15.00.log.html15:56
*** gagehugo has left #openstack-meeting-515:57
*** jgu has quit IRC16:00
*** yamamoto has joined #openstack-meeting-516:15
*** jgu has joined #openstack-meeting-516:16
*** yamamoto has quit IRC16:20
*** jgu_ has joined #openstack-meeting-516:21
*** jgu has quit IRC16:24
*** yamamoto has joined #openstack-meeting-516:30
*** yamahata has quit IRC16:34
*** yamamoto has quit IRC16:35
*** yamamoto has joined #openstack-meeting-516:45
*** yamamoto has quit IRC16:50
*** yamahata has joined #openstack-meeting-516:57
*** derekh has quit IRC17:00
*** yamamoto has joined #openstack-meeting-517:01
*** wxy|_ has quit IRC17:04
*** yamamoto has quit IRC17:05
*** yamamoto has joined #openstack-meeting-517:15
*** yamamoto has quit IRC17:20
*** yamamoto has joined #openstack-meeting-517:31
*** yamamoto has quit IRC17:35
*** yamamoto has joined #openstack-meeting-517:45
*** yamamoto has quit IRC17:50
*** hongbin has quit IRC17:56
*** yamamoto has joined #openstack-meeting-518:00
*** yamamoto has quit IRC18:05
*** yamamoto has joined #openstack-meeting-518:15
*** yamamoto has quit IRC18:20
*** yamamoto has joined #openstack-meeting-518:30
*** yamamoto has quit IRC18:35
*** yamamoto has joined #openstack-meeting-518:35
*** yamamoto has quit IRC18:36
*** radeks_ has quit IRC19:07
*** jgu_ has quit IRC19:21
*** yamamoto has joined #openstack-meeting-519:36
*** yamamoto has quit IRC19:42
*** slaweq has quit IRC19:43
*** slaweq has joined #openstack-meeting-519:44
*** radek__ has quit IRC19:49
*** yamamoto has joined #openstack-meeting-520:38
*** yamamoto has quit IRC20:47
*** jgu has joined #openstack-meeting-521:18
*** jgu_ has joined #openstack-meeting-521:22
*** jgu has quit IRC21:26
*** yamamoto has joined #openstack-meeting-521:43
*** yamamoto has quit IRC21:48
*** slaweq has quit IRC22:04
*** slaweq has joined #openstack-meeting-522:18
*** slaweq has quit IRC22:33
*** yamamoto has joined #openstack-meeting-522:44
*** yamamoto has quit IRC22:49
*** slaweq has joined #openstack-meeting-523:11
*** slaweq has quit IRC23:15
*** yamamoto has joined #openstack-meeting-523:46
*** yamamoto has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!