Tuesday, 2017-10-17

*** sshank has quit IRC00:12
*** markvoelker_ has quit IRC00:17
*** yamamoto has joined #openstack-meeting-500:24
*** markvoelker has joined #openstack-meeting-500:25
*** yamamoto has quit IRC00:29
*** markvoelker has quit IRC00:29
*** markvoelker has joined #openstack-meeting-500:34
*** lukepatrick has quit IRC00:37
*** markvoelker has quit IRC00:39
*** markvoelker has joined #openstack-meeting-500:43
*** markvoelker has quit IRC00:48
*** markvoelker has joined #openstack-meeting-500:52
*** markvoelker has quit IRC00:57
*** markvoelker has joined #openstack-meeting-501:01
*** markvoelker has quit IRC01:06
*** markvoelker has joined #openstack-meeting-501:11
*** markvoelker has quit IRC01:15
*** markvoelker has joined #openstack-meeting-501:20
*** iyamahat has quit IRC01:21
*** yamahata has quit IRC01:23
*** yamamoto has joined #openstack-meeting-501:24
*** lukepatrick has joined #openstack-meeting-501:24
*** markvoelker has quit IRC01:53
*** markvoelker has joined #openstack-meeting-501:59
*** markvoelker has quit IRC02:03
*** markvoelker has joined #openstack-meeting-502:08
*** markvoelker has quit IRC02:12
*** markvoelker has joined #openstack-meeting-502:17
*** markvoelker has quit IRC02:21
*** markvoelker has joined #openstack-meeting-502:26
*** markvoelker has quit IRC02:30
*** markvoelker has joined #openstack-meeting-502:35
*** markvoelker has quit IRC02:40
*** TuanLA has joined #openstack-meeting-502:44
*** markvoelker has joined #openstack-meeting-502:44
*** TuanLA has quit IRC02:44
*** markvoelker has quit IRC02:49
*** TuanLA has joined #openstack-meeting-502:52
*** markvoelker has joined #openstack-meeting-502:53
*** daidv has quit IRC03:02
*** daidv has joined #openstack-meeting-503:08
*** markvoelker has quit IRC03:27
*** lukepatrick has quit IRC03:33
*** TuanLA has quit IRC03:45
*** yamahata has joined #openstack-meeting-503:45
*** TuanLA has joined #openstack-meeting-503:46
*** markvoelker has joined #openstack-meeting-504:17
*** markvoelker has quit IRC04:22
*** markvoelker has joined #openstack-meeting-504:27
*** markvoelker has quit IRC04:33
*** markvoelker has joined #openstack-meeting-504:34
*** daidv has quit IRC05:06
*** daidv has joined #openstack-meeting-505:52
*** markvoelker has quit IRC05:54
*** markvoelker has joined #openstack-meeting-506:50
*** ralonsoh has joined #openstack-meeting-507:49
*** ralonsoh_ has joined #openstack-meeting-507:50
*** ralonsoh has quit IRC07:54
*** ralonsoh_ is now known as ralonsoh07:55
*** daidv has quit IRC08:01
*** iyamahat has joined #openstack-meeting-508:10
*** iyamahat_ has joined #openstack-meeting-508:11
*** iyamahat has quit IRC08:14
*** iyamahat_ has quit IRC08:20
*** iyamahat_ has joined #openstack-meeting-508:21
*** iyamahat_ has quit IRC08:25
*** yamahata has quit IRC08:26
*** derekh has joined #openstack-meeting-508:34
*** yamamoto has quit IRC08:47
*** yamamoto has joined #openstack-meeting-508:55
*** yamamoto has quit IRC08:55
*** markvoelker has quit IRC09:03
*** markvoelker has joined #openstack-meeting-509:04
*** yamamoto has joined #openstack-meeting-509:15
*** TuanLA has quit IRC10:05
*** yamamoto has quit IRC10:18
*** yamamoto has joined #openstack-meeting-510:25
*** yamamoto has quit IRC10:30
*** yamamoto has joined #openstack-meeting-511:26
*** yamamoto has quit IRC11:32
*** yamamoto has joined #openstack-meeting-511:41
*** yamamoto_ has joined #openstack-meeting-511:42
*** yamamoto has quit IRC11:46
*** yamamoto_ has quit IRC11:56
*** yamamoto has joined #openstack-meeting-512:19
*** mjturek has joined #openstack-meeting-512:54
*** yamamoto has quit IRC12:54
*** lukepatrick has joined #openstack-meeting-513:10
*** felipemonteiro_ has joined #openstack-meeting-513:43
*** felipemonteiro__ has joined #openstack-meeting-513:44
*** felipemonteiro_ has quit IRC13:48
*** lukepatrick has quit IRC13:54
*** yamamoto has joined #openstack-meeting-513:55
*** yamamoto has quit IRC14:05
*** yamahata has joined #openstack-meeting-514:20
*** iyamahat has joined #openstack-meeting-514:41
*** iyamahat has quit IRC14:45
*** yamamoto has joined #openstack-meeting-514:47
*** yamamoto has quit IRC14:47
*** alraddarla has joined #openstack-meeting-514:55
*** mateuszb has joined #openstack-meeting-514:58
mattmceuen#startmeeting openstack-helm15:00
openstackMeeting started Tue Oct 17 15:00:37 2017 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack-helm)"15:00
openstackThe meeting name has been set to 'openstack_helm'15:00
mattmceuen#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: openstack-helm)"15:00
portdirecto/15:00
jayahno/15:01
mateuszbo/15:01
alraddarlao/15:01
mattmceuenHere's the agenda, all -- I'll give you a couple mins to add topics: https://etherpad.openstack.org/p/openstack-helm-meeting-2017-10-1715:01
*** gagehugo has joined #openstack-meeting-515:03
v1k0d3no/15:03
xeko/15:03
*** kiennt26_ has joined #openstack-meeting-515:03
mattmceuen#topic update to official project status15:03
*** openstack changes topic to "update to official project status (Meeting topic: openstack-helm)"15:03
mattmceuenJust an update -- we submitted OSH for TC governance last week15:04
mattmceuenhttps://etherpad.openstack.org/p/openstack-helm-meeting-2017-10-1715:04
srwilkerso/15:04
jayahnyeap.15:04
mattmceuenReview is ongoing, but it's looking very positive.  Seven +1 roll-call votes so far. :)15:04
mattmceuenThat's really all I have to say about that.  Hopefully good progress next week.  Any other thoughts on the topic?15:05
mattmceuen#topic log-based alerting15:06
*** openstack changes topic to "log-based alerting (Meeting topic: openstack-helm)"15:06
portdirectnothing other than congrats mattmceuen15:06
portdirect:D15:06
mattmceuenThanks @portdirect :)  back at you man15:06
mateuszbI've been investigating different methods for sending alerts based on logs15:06
mateuszbTat was discussed during last AT&T and Intel meeting15:06
v1k0d3nyeah, really happy to see this mattmceuen. great seeing you leading the charge.15:07
mateuszbBefore I proceed with an implementation I'd like to know your opinion ;)15:07
mateuszbAs for now I'm aware of 2 approaches:15:07
mateuszb1. Using ElastAlert15:07
mateuszb2. Using Prometheus plugin for fluentd (https://github.com/kazegusuri/fluent-plugin-prometheus )15:07
mateuszbThe idea behind both of them is the same: search for specified patterns (for example "Can't connect to MySQL") in container logs15:08
mateuszband if the pattern occurs - fire an alert15:08
mateuszbI'd be rather in favor of using ElastAlert as it's easier to configure and more intuitive for me15:09
mateuszbIt periodically queries ElasticSearch to retrieve a given pattern15:09
mateuszbIt has highly modular approach - each rule is a separate file15:09
jayahnI will look at ElastAlert. Just a question here. What is our purpose here? there can probably be many method to do log-based alarm. do we want to say "there is a reference implementation on log-based alert"?15:09
mateuszbYou can configure timeouts, thresholds and so on out of the box. I'd require, however, additional chart to be implemented15:09
mattmceuenHow does that compare to the Prometheus plugin, mateuszb?15:10
mattmceuenDoes it also support a modular approach, or more monolithic?15:10
mateuszbjayahn - that's a good question :) I'd like to know what the requirements from AT&T side are15:11
portdirectmateuszb: thats not to important here :)15:11
portdirectwhat does the community want - hopefully we align internally with that15:11
jayahnimportant stuff would be "what alert we want to have". this can be a shared list. how to implement it will highly depends on what tools you use.15:11
mattmceuenAgree.15:11
mattmceuen I would like srwilkers as our LMA SME to weigh in as well15:12
portdirect++15:12
jayahnso, IMHO, defining the basic set of alerts would be valuable.15:12
srwilkersi'd need to look at the differences between elastalert and the prometheus plugin, but i'd honestly prefer to handle it via prometheus if the end result is the same15:12
portdirectsrwilkers: how is the prometheus ps progressing?15:13
mateuszbI agree. I can imagine that for the beginning we can start with alerts which inform about DB/rabbitmq connection issues in specified pods15:13
portdirectits pretty clost to being out of WIP is it not?15:13
srwilkersportdirect: its out of WIP and ready for some initial reviews15:13
portdirectw00t15:13
srwilkersim currently working on getting jobs set up for the controller manager and scheduler, but thats roughly another ~10m worth of work15:13
srwilkersonce that's done, i'll be happy with where it stands15:14
mattmceuenwould it make sense to take this as a homework assignment:  srwilkers and jayahn to get acquainted w/ ElastAlert, and mateuszb maybe you could do a comparative example of a few rules, implemented by both tools, that you could share?15:14
jayahnif we know what alerts we want to create, then we can use that list to evaluate how "a specific alert implmentation tool" can satisfy our requirements.  that said, initiating an effort to defining a basic set of alerts would be also valuable.15:14
mateuszbThe result would be the same (or at least as for now I don't know any limitations)15:14
portdirectthat sounds like an awesome idea mattmceuen15:14
srwilkersif you call this homework, im going to be a typical CS student and wait until the last minute ;)15:15
portdirectand presumably there is nothing precluding both from being implemented?15:15
mattmceuenSorry, I meant to say "ice cream"15:15
portdirectif they serve diff use cases15:15
mateuszbOk, so I'll prepare 2 patchets (one for ElastAlert and one for fluentd-prometheus plugin)15:16
srwilkersportdirect: nope, there's not.  like jayahn said, it'd just be a matter of determining if we want to choose a reference or not15:16
jayahnwe should say "pizza". typical CS student can do anything for pizza.15:16
mattmceuen@jayahn that is another good homework assignment :) do we have any list in progress around specific alerts needed?15:16
mattmceuenha!15:16
srwilkersmateuszb: in regards to the fluentd plugin, it'd be nice to make sure it works with the fluentbit+fluentd set up that's in review currently15:16
jayahni can write up some initial docs regarding alert needed15:17
mateuszbsrwilkers: I'll do that on top of fluentbit+fluentd PS15:17
mateuszbsrwilkers: in fact, that is a requirement for fluentd-prometheus plugin to work15:17
mattmceuenjayahn mateuszb awesome, thank you!15:17
*** markvoelker has quit IRC15:17
mateuszbcause it enforces us to use only one fluentd instance per cluster which I think will be true when fluentd+fluentbit PS is merged15:18
mateuszbjayahn that would be great15:18
*** markvoelker has joined #openstack-meeting-515:18
mattmceuen#action mateuszb will prep comparison of ElastAlert and fluentd-prometheus15:19
mattmceuen#action jayahn will take a stab at initial alerting need documentation15:19
jayahnokay15:19
* portdirect notes a mech eng student only works for alcohol 15:19
mateuszb:)15:19
mattmceuenGood discussion guys, anything else on this topic?15:20
* jayahn an korean student regardless of major onnly works for alcohol15:20
*** iyamahat has joined #openstack-meeting-515:20
mateuszbthat's all from my side15:20
srwilkersahahaha15:20
mattmceuen@jayahn lol15:20
mattmceuen#topic cw's RFC on storage class name15:20
*** openstack changes topic to "cw's RFC on storage class name (Meeting topic: openstack-helm)"15:20
mattmceuenI'm not sure we have cw?15:21
srwilkersdoesnt seem so15:21
srwilkersthis was just to rename the general storage class right?15:21
mattmceuenWell let's get him some opinions offline.  He has an RFC out for whether we should use "general" or some other name.15:21
mattmceuenyup15:21
portdirectI have some feedback in there: https://review.openstack.org/#/c/511975/15:21
mattmceuenthx portdirect, copy/paste was failing me ;-)15:22
mattmceuenI'll leave it at that for now -- if you'd like to see the discussion or weigh in, please visit the review!15:22
*** markvoelker has quit IRC15:22
mattmceuen#topic Official project docs move (timing and process)15:23
*** openstack changes topic to "Official project docs move (timing and process) (Meeting topic: openstack-helm)"15:23
mattmceuenAfter we become an official project, I think we'll want to move our docs here, correct? https://docs.openstack.org/openstack-helm/latest/15:23
mattmceuen(away from readthedocs)15:23
portdirectyup15:24
portdirectI think this should be very simple to do15:24
mattmceuenWill we deprecate the readthedocs at that point, or leave it up?  Maintain it?15:24
portdirectI know lamt has insight into what needs done, but i think a theme change and a single ps to infra15:25
portdirectI vote for retiring the readthedocs15:25
portdirectthough it would be possible to have both15:25
portdirectand they would stay in sync15:25
jayahni also vote for retiring the readthedocs once we move to the official doc15:25
srwilkersi'd rather retire them than maintain them15:25
portdirectbut seems silly to have them in two places15:26
mattmceuen+115:26
mattmceuen#action mattmceuen to discuss readthedocs->docs.openstack.org with lamt15:27
mattmceuenAlrighty:  reviews time15:27
mattmceuen#topic reviews needed15:27
*** openstack changes topic to "reviews needed (Meeting topic: openstack-helm)"15:27
jayahnonce we move to docs.openstack.org, i can ask official translation to to take openstack-helm project.15:27
mattmceuengreat point jay15:27
jayahni talked with srwilkers on the review.15:28
mattmceuenawesome15:28
srwilkersyep :)15:28
portdirectjayahn: long as you promise not to write nasty things about us in korean ;)15:28
mattmceuensave any nasty comments for english please :-D15:28
*** renmak__ has joined #openstack-meeting-515:29
jayahnnothing to add. personally, i would like to have both prometheus and fluent working in our env. asap. :) .. let's get it done.15:29
jayahnha ha..15:29
mattmceuenyes15:29
srwilkersjayahn: +115:29
jayahnyou have google translation. so i will never wirte nasty stuff in korean.15:29
mattmceuenHere's the review for fluent: https://review.openstack.org/#/c/507023/15:30
jayahnhowever, we know how to avoid google translation, but still understand each other. tweaked version of korean writing. lol15:30
srwilkers;)15:30
srwilkersin regards to that review15:30
srwilkersim happy with it after discussing it with jayahn.  takeaway was that we can leave kafka as an opportunity for a future addition to OSH15:31
portdirect+215:31
srwilkerspotentially something for a new contributor to try their hand at15:31
jayahnyes.15:31
mattmceuenexcellent15:31
portdirectthat was also the one with some fun stuuf in the dep chack was it not?15:31
portdirect*check15:31
portdirectah yes: https://review.openstack.org/#/c/507023/9/fluent-logging/templates/deployment-fluentd.yaml15:32
portdirectwe should probably make a blueprint for extending the dep checking model to account for conditionals15:32
portdirectas this is starting to turn up more and more15:32
portdirectesp in neutron15:32
mattmceuenvery good point15:32
* portdirect why is it always neutron?15:32
srwilkersportdirect: i agree15:33
jayahn+115:33
srwilkersthere are somethings that are best handled via overrides, but conditionals may not be the best place for them15:33
srwilkerstook me chewing on that a bit to feel that way15:33
srwilkerswell, not all conditionals ill say15:33
portdirectI agree - I've had a few chats with alanmeadows on this15:33
portdirectand will try and summarise what we were mulling over and throw it out there15:34
* portdirect notes that hes should bring up somthing else in any other business 15:34
mattmceuen#action portdirect to create a blueprint for extending the OSH dependency checking model to account for conditionals15:34
mattmceuenbefore other business15:35
mattmceuenany other outstanding reviews we need more eyeballs on?15:35
mattmceuen(also thanks pete :) )15:36
portdirecthttps://review.openstack.org/#/c/457754/15:36
srwilkersnope, im good15:36
portdirecthttps://review.openstack.org/#/c/507628/15:36
portdirect^^ both of these ceph ones are right up on my list of things we want15:36
portdirecti think the disc targetting has a bit to go15:36
portdirectbut is getting close15:36
mattmceuenThat is great to hear15:37
srwilkersnice.  ill take a gander and see what's going on there15:37
mattmceuenTeam:  good work on reviews this week, I think we're going in a good direction there.15:37
mattmceuen#topic other business15:37
*** openstack changes topic to "other business (Meeting topic: openstack-helm)"15:37
mattmceuentake it away portdirect15:37
portdirectso - (and I'm gonna me a blueprint for this)15:38
portdirectwe have a need to be able to run multiple configs of some daemonsets15:38
portdirectie nova-compute and ovs-agent15:38
mattmceuenis "multiple" on the order of per-node, or a constant?15:39
portdirectwe can currently achive this though some footwork using the manifests: true/false, and multiple deployments of a chart, but I'd like a cleaner solution15:39
mattmceuengive me big-o notation, portdirect!15:40
portdirectmultiple for us could mean a lot of things unfortunatly - from groups of hardware to canary nodes15:40
portdirectso I'm tinking of extending the to_oslo_config logic for damonsets, and the damonset manifests themselves to allow this15:41
portdirectso an example config would be as folows:15:41
* srwilkers waits on the edge of his seat15:43
alraddarlahe probably got distracted while he was typing15:43
portdirecthttps://www.irccloud.com/pastebin/Fi1hpGbz/15:43
portdirectso most compute nodes would be debug=true15:44
portdirectones labeled with `arbitary_label` would be debug=false15:44
portdirectand if its hostname was `hostname` it would be debug=false regardless of what labels etc were applied15:44
portdirectthoughts before I do the initial write-up/proposal?15:45
*** lukepatrick has joined #openstack-meeting-515:45
srwilkershmm15:46
mattmceueninteresting - am I correct in assuming we would continue not to use this for e.g. disk targeting?  Or do you see it being used for that?15:46
portdirectdisc tragtting could potentially benifit from this15:46
portdirectthough thats solving a slightly diff problem15:47
mattmceuenmy only concern is that "node_groups" and "nodes" are only meaningful if you already know what they mean15:47
portdirectthis would need to be documented15:48
*** yamamoto has joined #openstack-meeting-515:48
*** mateuszb has quit IRC15:48
mattmceuenWould it make sense to put them under something (or call them something) more descriptive, like (but better than) per_thing_configs.node_group15:48
portdirectgood point15:49
portdirectso like this?15:49
portdirecthttps://www.irccloud.com/pastebin/FetsDzFf/15:50
mattmceuenthat is a good point, but not the point I was trying to make :)15:51
srwilkersi suppose i'd need to see what's written up and get the line of thinking behind it15:51
portdirectlol15:51
portdirectwill do - just putting it out there, for us internally this will be required15:51
mattmceuenI meant literally an id like "per_thing_conf" that made clear what a "node_group" or a "nodes" is -- what's the context15:52
portdirectbut would be great to have a solution that works well upstream15:52
mattmceuenI really like the flexibility15:52
portdirectgotcha15:52
jayahni will follow up on portdirect's upcoming write-up on this.15:52
mattmceuen+1 same, looking forward to it15:53
jayahnand. with my thinking hat.15:53
mattmceuenAright guys - any other topics?15:53
portdirectI'm good15:53
srwilkersoh, unrelated15:53
srwilkersbut TC voting is happening currently15:53
srwilkersdont forget to vote!15:53
mattmceuenthanks for the reminder!15:54
mattmceuenOk team - good meeting.  Have a great day, see you in the chat room!15:54
portdirecttheres this dude YamSaple15:54
portdirectwhatever you do dont vote for him.15:55
v1k0d3nhaha15:55
mattmceuenwe don't want to steal time away from him distracting you in our chat room portdirect -- excellent point15:55
portdirect;)15:55
*** yamamoto has quit IRC15:55
mattmceuen#endmeeting15:55
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"15:55
openstackMeeting ended Tue Oct 17 15:55:48 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:55
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_helm/2017/openstack_helm.2017-10-17-15.00.html15:55
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_helm/2017/openstack_helm.2017-10-17-15.00.txt15:55
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_helm/2017/openstack_helm.2017-10-17-15.00.log.html15:55
*** alraddarla has left #openstack-meeting-515:56
*** gagehugo has left #openstack-meeting-515:56
*** kiennt26_ has quit IRC16:07
*** ralonsoh has quit IRC16:33
*** yamahata has quit IRC16:34
*** iyamahat has quit IRC16:34
*** iyamahat has joined #openstack-meeting-516:53
*** derekh has quit IRC17:09
*** yamahata has joined #openstack-meeting-517:11
*** renmak__ has quit IRC17:13
*** sshank has joined #openstack-meeting-517:22
*** sshank has quit IRC17:28
*** ricolin has quit IRC17:37
*** markvoelker has joined #openstack-meeting-519:26
*** markvoelker_ has joined #openstack-meeting-520:15
*** markvoelker has quit IRC20:18
*** lukepatrick has quit IRC20:56
*** lukepatrick has joined #openstack-meeting-521:02
*** lukepatrick has quit IRC21:18
*** lukepatrick has joined #openstack-meeting-521:23
*** lukepatrick has quit IRC21:23
*** lukepatrick has joined #openstack-meeting-521:24
*** lukepatrick has quit IRC21:29
*** lukepatrick has joined #openstack-meeting-521:30
*** felipemonteiro__ has quit IRC22:17
*** lukepatrick has quit IRC22:49
*** mjturek has quit IRC22:54
*** kei-ichi has quit IRC23:47
*** kei-ichi has joined #openstack-meeting-523:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!