Tuesday, 2019-03-19

*** jamesmcarthur has joined #openstack-tc00:20
*** zhipeng has quit IRC00:27
*** zhipeng has joined #openstack-tc00:37
*** ricolin has joined #openstack-tc01:08
*** jamesmcarthur has quit IRC01:11
*** whoami-rajat has joined #openstack-tc01:25
*** lbragstad has quit IRC01:56
*** lbragstad has joined #openstack-tc02:09
*** jamesmcarthur has joined #openstack-tc02:20
*** jamesmcarthur has quit IRC02:25
*** dangtrinhnt has quit IRC02:25
*** dangtrinhnt has joined #openstack-tc02:30
mnaserasettle, jroll: just wanted to follow up if you had a chance to push up a review to slot in a place for the project review (wrt vision) in governance and update community afterwards over ML02:35
openstackgerritMerged openstack/governance master: Explicitly declare Train supported runtimes.  https://review.openstack.org/64227202:38
*** jamesmcarthur has joined #openstack-tc02:38
*** jamesmcarthur has quit IRC02:43
*** ricolin has quit IRC02:44
*** jamesmcarthur has joined #openstack-tc03:14
*** jamesmcarthur has quit IRC03:31
*** jamesmcarthur has joined #openstack-tc03:31
*** diablo_rojo has quit IRC04:15
*** jamesmcarthur has quit IRC04:40
*** dklyle has quit IRC04:44
*** dklyle has joined #openstack-tc04:45
prometheanfirejust saw the thing about golang, wondering if coinstallability was considered04:54
*** ricolin has joined #openstack-tc05:23
*** lbragstad has quit IRC06:38
*** ricolin has quit IRC06:43
*** jaosorior has quit IRC06:55
*** jaosorior has joined #openstack-tc06:57
*** Luzi has joined #openstack-tc06:59
*** tosky has joined #openstack-tc07:25
*** dtantsur|afk is now known as dtantsur08:18
*** e0ne has joined #openstack-tc08:25
asettlemnaser, sorry - just got back online. I have not done the thing, I'll check in with jroll08:58
*** ricolin has joined #openstack-tc08:58
*** Luzi has quit IRC08:59
*** jpich has joined #openstack-tc09:00
ttxohai09:01
ttxtc-members: quick reminder to add your name to https://framadate.org/dBXkYU4MUyROFSYB if you're interested in joining a TC dinner on Saturday evening of the summit09:02
asettleMorning o/09:02
ricolinhola!09:02
asettlettx, the saturday before?09:02
ttxasettle: yes09:02
asettleAh!09:02
ttxevrardjp: also need to update your answer to say whether 8:30pm would work for you09:03
evrardjpwill do ttx09:03
* ttx go grab coffee09:04
*** dklyle has quit IRC09:08
*** dklyle has joined #openstack-tc09:09
ttxI was reading https://twitter.com/mikal/status/1107787256012562433 and wondering if someone had more context ? I wish that discussion would happen on the ML instead of Twitter, but if we want to be friendly to part-time contributors I guess we can't force all discussions to happen on ML.09:11
ricolinttx to have a way to be friendly in social media maybe?:)09:20
evrardjpttx: If mikal wanders off due to comment on nova community that's quite opposite to the anti-nitpicking culture we have tried to build09:55
evrardjpthen I think we can say anti-nitpicking has failed09:56
evrardjpanyone has an idea on how to improve there?09:56
*** e0ne has quit IRC09:56
cmurphyI don't think it's fair to say anti-nitpicking has failed because one person complains09:59
*** jpich has quit IRC09:59
ttxyeah, I really need more context to form an opinion09:59
cmurphyI also don't think mikal's tweet is about nitpicking, it sounds like a legitimate disagreement about the value of a feature or design09:59
evrardjpcmurphy: that's not what I meant09:59
fricklerthis seems to be the review being talked about https://review.openstack.org/64068809:59
*** jpich has joined #openstack-tc10:00
evrardjpbut I am glad my controversive message got ppl to talk about it :p10:00
evrardjpfrickler: thanks for the context :)10:02
evrardjpcmurphy: it seems so10:02
ttxLooks like mikal was not ready/willing to discuss implementation... I don't see anyone saying that would be a bad idea here.10:05
evrardjpyeah10:06
evrardjpthat's totally different :)10:06
ttxI suspect there is more context though... since ni another tweet he also said people redirected him to os-profiler10:07
evrardjpMy point was -- this is a contributor, whoever this might be, known to us or not. We shouldn't shut him down, and he should listen to positive improvements. That's all I meant.10:07
evrardjpProbably terribly formulated though10:07
ttxThe os-profiler mention is what got me interested in that discussion -- as I don't think we should not do Prometheus support because we have os-profiler10:08
ttxBut no mention of os-profiler on that review so I suspect some of that discussion happened elsewhere10:08
*** cdent has joined #openstack-tc10:27
*** e0ne has joined #openstack-tc10:50
mugsiettx: I think it was IRC - but I can't find it anywhere ...11:01
mugsieand in this case it is really unfortunate, I know that these style of stats are very useful (and completely different to os-profiler)11:01
ttxmugsie: exactly11:10
jrollmugsie: +10011:21
jrollwe (and I assume other companies) have just built little binaries which run on the hypervisor and push similar data into $metrics_system_of_choice11:23
jrollwould be nice for it to be built in and standard (and I'd much rather just choose a tech rather than support a million systems)11:24
jrollasettle: mnaser: vision things is on my list to catch up on today - I'll dig around governance and see if I can find a place it belongs11:24
dtroyer(disclaimer: I've only read the review comments noted above but it raises a point in my head)  Many of our projects push back against anything that isn't fully-developed or designed.  I am fighting this in multiple places these days,… where does the community expect things to get worked out that need exposure to real deployments?11:24
asettlejroll, thank you :) I am *swamped* with non-stack work today11:24
asettledtroyer, good question11:25
jrollasettle: no worries11:25
cdentdtroyer: good question. Our development and release proceses aren't friendly to that kind of thing, which is a real bummer11:26
jrollI feel like we assume every deployment has patches on top of openstack. and we also assume everyone has a corporate sponsor. so asking people to prove things out downstream is typical11:29
jroll(which isn't always good)11:30
jrollI have personally found the community is quite receptive to helping design not-quite-baked features where there's a clear need. proving that the feature is valuable is harder without data. (in this case, probably showing off a dashboard built from these metrics)11:30
cdentI bet openstack-the-project would drastically improved if we hosted our own public cloud (with real customers)11:32
cdentnot that I expect that to happen11:32
cdentbut it would be interesting11:32
mugsiecdent: at one point I considered running a large Designate install on tip of master, but failed to get resources needed :/11:32
jrolla million times agree, cdent11:32
cdentdtroyer: I got pinged yesterday about whether or not I was intending to go any further with my ectd-compute stuff and thus far I haven't been motivated becauses there's little conversation about it11:33
mugsieI think it really helps11:33
cdentthat lack of solid feedback loops is a big problem (to me) in openstack11:33
jrollwhen infra-cloud was coming up, the infra team contributed a ton of feedback/code11:33
evrardjpjroll: I like what you said "would be nice for it to be built in and standard" -- this is what I believe the twitter rant is about.11:54
evrardjpofc I might be wrong11:54
* cdent missed a twitter rant?11:55
cdentlink?11:55
evrardjpwe have one feedback here, let's just try to convert the feedback with a built in feature11:55
evrardjphttps://twitter.com/mikal/status/110778725601256243311:55
evrardjpif we can call that a rant11:55
evrardjpyou know my english is not great :p11:56
evrardjpanyway11:56
cdentI find it utterly hilarious that mikal complains about nova's behavior when, from my standpoint, he is one of the major factors in why droves of people walked away from nova's toxic environment.11:57
cdentThat said, he's right11:57
evrardjpI don't judge/look at the messenger -- I just look at what could be interesting to learn from this11:58
evrardjpI totally would like this prometheus thingies :)11:58
cdentyes, me too11:58
evrardjpmore built in, less downstream please :D11:58
evrardjpthe metrics case is interesting, as I feel it's very close to healthchecking middleware11:59
evrardjpwhere do we stop, how do we implement things together?12:00
cdentI think it is _much_ different from healthchecking middlware12:00
evrardjpoh?12:02
cdentI guess it depends on how it goes, but I think metrics  and health are important to keep distinct12:03
cdentin part because of the granularity/frequency of the info12:03
mugsieI think they are in the same vein - using "cloud native" app features to help deployers / ops use standard toolchains and common app suites12:17
mugsiee.g.  would love to replace os-profiler with jeager / opentracing, so if someone does a k8s volume create, we could trace the entire call down to cinder and the backend to see what takes time or errors out12:18
evrardjpjaeger/open tracing is indeed a compelling use case from what I can see here12:23
mugsieit is12:29
fungithe prometheus/statsd functionality is compelling if ceilometer maintenance erodes further12:36
fungiso as to have a more integrated alternative12:36
*** ijolliffe has joined #openstack-tc12:37
mugsiefungi: I think this is orthangonal to celiometer - even if that was a really active project I would advocate for promethous12:39
mugsieit is a standard base level service for a lot of IaaS/PaaS/DCaaS software these days, and we should support the standard way people collect stats from a lot of other software12:39
fungiwell, yes, because 1. used by more tooling outside openstack, and 2. implementation would be actually integrated in services rather than some external codebase which tries to import code from services directly12:40
* fungi doesn't actually know whether ceilometer still imports nova and depends on details of private objects, but that was at least the case for a very long time12:41
fungioh, and by monkey-patching the services themselves too12:42
cdentAs I recall, the reason for that all that stuff was because ceilo was unable to get cooperation from the projects-being-measured12:44
cdentso ceilo has been surfacing fundamental issues in the openstack process for _years_12:44
fungiright, i'm not saying they made that choice lightly12:44
fungijust that it probably would have been far easier to maintain if the services were up for integrating stat emitting routines/hooks in their codebases12:45
* cdent nods12:45
fungihaving to work around the unwillingness to carry what effectively has to be integrated routines in the service implementations necessarily implies fragile design patterns and a need to constantly chase any changes made to related parts of the services12:48
*** mriedem has joined #openstack-tc12:54
mugsieYeah - things like changing the format of events doesn't help either - it forces projects to import nova.objects to read events from the notifications queue13:03
* dhellmann senses the makings of a community goal for the U cycle13:06
fungiopenstack unicorn: making the unpossible13:18
dhellmannI would wear that t-shirt13:25
fungiso... wrt the ad hoc goals meeting, whatever came of the idea that we could use the 14:00 utc thursday slot similar to our ~monthly meetings?13:26
fungithe poll ended up using thursday office hour instead13:27
fungiwhich even though it seems to be the most popular, i'll probably have to bow out of as i already have two other meetings going on at the same time13:28
*** lbragstad has joined #openstack-tc13:34
*** marst has joined #openstack-tc13:44
mnaser*honestly* i'm all for us adopting other projects into our ecosystem that have been more successful and thriving with contributions than our own little pet projects13:46
mnaserwe talk about being better open source citizens and part of that is integrating across *13:46
mnaserhaving said that, i'm pretty sure osprofiler != prometheus/exporters13:47
mnaserthat's more on the opentracing side of things13:47
mnaseri do have some disagreements on what mikal believes prometheus should be exposing but i didn't want to get into an in-depth discussion about implementation details13:48
mnaserceph has done a great job exposing things like this: http://docs.ceph.com/docs/mimic/mgr/prometheus/13:49
ttxthat's something that would benefit from openstack-level design. you would likely want those Prometheus metrics exposed across more than just Nova anyway.13:54
fungiyeah, but you really need code *in nova* to generate the events13:54
*** Luzi has joined #openstack-tc13:54
ttxoh for sure. But most of the objections (in that review at least) was more around how to expose the metrics, rather than which metrics or how to generate them13:55
fungi(and ideally not just flood the already overtaxed and broken external message queue)13:55
*** whoami-rajat has quit IRC13:55
fungibut yes, i'm not going to just assume this is a case of the nova reviewers rejecting the idea of stats emitting altogether, and merely taking exception to mikal's proposed implementation thereof13:56
fungier, rather than taking exception to13:56
evrardjpas  dhellmann said -- nice goal IMO13:57
TheJuliaSo speaking of metrics code, in some downstream discussions at least, we've been discussing exposing metrics to prometheus for hardware, so we actually have someone that is going to look at hooking into our existing metrics data stream that we had to send ceilometer, and be abel to redirect it to prometheus. As for measurement of internals, ironic could also change its internal metric measurement decorator to support14:05
TheJuliaadditional options.14:05
*** needscoffee is now known as kmalloc14:09
*** jpich has quit IRC14:09
*** jpich has joined #openstack-tc14:13
zanebI'd just like to take this opportunity to say that if the compute node had a ReST API then it would be very easy to expose metrics for monitoring tools14:19
zaneband then run away very quickly14:19
zanebbefore people start throwing things at me14:19
jrolltoo late, we now have automated catapults which are triggered on that phrase14:20
jrollI agree with that, though :)14:21
dhellmannmaybe adding a rest api for metric collection is the first step to unifying all of the team-specific agents that run on the compute host14:22
zanebis this the point in the discussion where we need to ping Nova people and invite them to throw stuff at us, to avoid them throwing stuff at us later when they find out we discussed it without pinging them?14:24
TheJuliadhellmann: are you speaking in only the nova context, or a larger context?14:25
* zaneb can never keep track of the rules14:25
mriedemdid someone say nova? i'll just drop this in here https://docs.openstack.org/nova/latest/contributor/policies.html#metrics-gathering14:26
cdentzaneb: I think you've been to enough nova ptg and forum sessions to know that throwing stuff is allowed at any point in the time space continium if you express the correct key phrases14:26
zanebcdent: I've been to maybe like 3?14:27
cdentthat should be enough14:27
mriedemzaneb: how is a rest api on the compute node itself going to make metric unicorns happen?14:27
mriedemthere are already compute REST APIs for hypervisor stats14:27
mriedemhttps://developer.openstack.org/api-ref/compute/?expanded=show-hypervisor-statistics-detail#show-hypervisor-statistics14:28
mriedemwould all of the people here that want to add metrics gathering and management to nova please sign this 4 year contract to test and maintain that stuff as well please14:28
mriedemthanks14:28
zanebmriedem: context https://review.openstack.org/64068814:29
mriedemwe also have old crap like this in the rest api https://developer.openstack.org/api-ref/compute/?expanded=#server-usage-audit-log-os-instance-usage-audit-log14:32
mriedemwhich was added for ceilometer long ago14:32
mriedemand some bandwidth notification stuff for xen which is just rotten at this point14:32
mriedemhttps://github.com/openstack/nova/blob/d3254af0fe2b15caff3990c965194133625b681d/nova/compute/manager.py#L751314:33
mriedemoh hi and this stuff https://github.com/openstack/nova/tree/d3254af0fe2b15caff3990c965194133625b681d/nova/compute/monitors14:33
mriedemwhich is not tested anywhere14:33
dhellmannTheJulia : the larger context14:33
*** jamesmcarthur has joined #openstack-tc14:39
*** cdent has quit IRC14:59
mnaseryeah nova does indeed have a lot of bit rotted metrics stuff that barely works15:00
mnaseri.e. instance usage audit log pretty much can kill your database server on a reasonbly sized cloud15:01
*** diablo_rojo has joined #openstack-tc15:02
mriedemi'd rather wipe the slate before re-inventing the wheel here, but since it's exposed in the REST API that's kind of hard to deprecate15:06
TheJuliadhellmann: I think one size fits all is a mistake. Each project has slightly different architectures and implementation details that do not map into the concept of compute nodes running services15:06
*** cdent has joined #openstack-tc15:10
*** Luzi has quit IRC15:12
gmannmriedem: well, if we are sure that's the thing not used much then, we can think of deprecating it via version bump. cleaning those are much helpful for maintenance. nova already have a lotss of APIs15:14
mriedemdeprecating the api with a microversion is just symbolic, the code remains essentially forever for backward compat15:15
gmannat least that indicate the deprecation and freeze code for that API.15:17
bauzasMHO on the Prometheus rant is that AFAIK Prometheus maintainers have a very strong opinion about push vs. pull metrics15:17
bauzasand they don't like push15:17
bauzason the other side, we do have many ways for Prometheus to pull our metrics thru our API15:18
bauzasso, I'm very disappointed by the rant since it's even not something I think Prometheus maintainers would love15:18
mriedemalways helpful for the conversation https://twitter.com/mikal/status/110778725601256243315:20
bauzasgosh dear Twitter15:20
mnaserprometheus style metrics => a rest endpoint that exposes key/values pretty much15:20
mnaserthats the whole 'exporter' idea15:20
bauzasI very know simon pasquier15:20
mnaserprometheus on it's own scrapes those exporters and then does $things with them15:20
bauzashe was an openstack dev before he moved to prometheus15:21
bauzasI can ask him to chime in15:21
bauzasbut I'm pretty sure they don't like push models15:21
bauzasoh, fun, actually, he replied15:21
bauzas"As stated in the documentation [1][2], the Pushgateway is targeted for  very specific use cases (mostly services that don't live long enough to  be scraped by Prometheus). I'd definitely recommend exposing the metrics  via HTTP."15:21
mnaserprometheus is pull based (exposing a rest api), statsd is push based (push to socket on event X)15:22
mnaserso they are inherently different architectures15:22
mnasernow if you ask me, a good exercise would be for us to gather around a bunch of operators, ask how you're doing metrics, and try to find the common denominator15:22
bauzasoh yeah15:22
*** whoami-rajat has joined #openstack-tc15:23
bauzasyup, but my take is that we have notifications for event-based metrics15:23
bauzasfor other metrics we have our API15:23
bauzasso, what are we missing ?15:23
mnaserand we use oslo.notifications for event based metrics which means anyone can easily write a driver if they want to use statsd15:23
mnaseri guess it narrows down to nova itself providing _native_ prometheus-consumable information, but that might start to creep scope15:24
bauzasyup, it's not I'm against stats push15:24
bauzasI'm more I don't think it should be nova's responsibility to push those metrics15:24
mnaserlike i cant just _wire_ prometheus into nova and pull info out, there's integration work to be done.  should that live inside nova or not is the question ..  i think there is strong reasons for both15:24
mnaseri found this the other day15:25
mnaserhttps://github.com/zhangjianweibj/prometheus-libvirt-exporter15:25
*** cdent has left #openstack-tc15:25
mnaser*SUPER* neat.15:25
bauzasagain, if we go with prometheus, it has to be a pull model15:25
TheJuliamnaser: we've been talking about creating a plugin that intercepts our event stream, and kind of sorts our data out by baremetal node, and then expect to have something read the data and transform it out into something exporter friendly since we have a curse of varying vendor data15:25
gmannyeah, i agree on consuming it via push driver and consume nova notification/API info and tell "this is we need and missing and how we can get"15:25
bauzasI have to go get my kids, but I'll reply on the change itself15:26
mnaseri think we need to also realize _what_ we need to instrument too15:26
bauzasI'm way off Twitter for discussing design points15:26
mnaserthis has a huge impact, if you want to instrument instance boot times vs # of instances on a hypervisor, those are very much different things15:26
mriedemheh https://github.com/zhangjianweibj/prometheus-libvirt-exporter looks like a mismash of existing stuff that is exposed in the compute API15:26
mriedemfor instance: https://developer.openstack.org/api-ref/compute/#servers-diagnostics-servers-diagnostics15:26
gmannbauzas: :) true.15:27
mriedemand ceilometer agent already pulls information directly from libvirt apis15:27
mnasermriedem: yeah, but this exposes it in 'prometheus'-friendly format15:27
TheJuliamnaser: ++, although we shouldn't front load too much because of the variability and if we decide to instrument too much then it will be perceived as too hard and too much work.15:27
mriedemmany years ago we had talked about ways to send that information out of nova via notification so ceilometer wouldn't need to ask libvirt for it15:27
mriedemmnaser: well anything can translate what nova exposes yeah?15:27
mnaserif i was to write something that hit the nova api directly, we'd get list of all vms and then call diagnostics on every single server, way slower overall i guess15:27
mriedemright, i'd have to dig (far) but i remember the ceilometer thing from years ago was a way to avoid that15:28
mriedembut still get to ceilometer what they were doing with libvirt15:28
mnaseryeah ceilometer actually does the same thing as this15:28
gmannyeah GET diagnostics if per VM only not as whole list15:28
mnaserbut ceilometer pushes notifications perodically, this just echos out info every time you hit the endpoint15:28
TheJuliamnaser: so for collecting data from redfish BMCs, we'll have to do much the same for components :(15:29
mnaseralso15:29
mnaseri'm in favour of bringing this problem on step up in some sense15:29
mnaserlike: a document on how to best instrument $foo using $some_tooling15:30
mnaserand depeloymnt tools can implement that, or not, but that stuff doesn't live inside the project itself15:30
mriedembtw https://etherpad.openstack.org/p/BER19-OPS-LOGGING-AND-MONITORING15:30
mriedemthis was just discussed in berlin15:30
mnaser*OR* we can have something like nova/prometheus-exporter under opendev and those interested in that can hack on it too :)15:30
mriedemwell, this == monitoring15:30
TheJuliamnaser: ++ I feel this is an easy discussion to sidetrack on because of different interpretations to words15:30
mnaseri do not feel that we're "cutting out" the casual contributor in this scenario going back to the original topic15:32
gmanni feel we can have under openstack/xyz-matrix means consuming all other service matrix too if needed. I agree on not being push these on project in-tree.15:32
mnaserthis is a fundamental part of a project and i don't think realistically a part time contributor could succeed with it, because it involves work, writing specs, getting people to agree15:32
mnaseri think this is just something we have to accept.  this isn't a small change.15:33
TheJuliaIt is a nice thought, but it is going to push users away if we only support our own thing15:33
mnaserfyi what mikal is bringing up is not accurate15:33
mnaserosprofiler != prometheus15:34
mriedemmnaser: found it https://openstack.nimeyo.com/49291/openstack-ceilometer-proposal-hypervisor-notifications15:35
mriedem201515:35
mnaserthese are two different solutions for two different problems.  one is cross-project / request tracing, the other is metrics exporting15:35
mnasermriedem: i think that's how it should be from the first place but who's gonna do the work is always the fun question15:36
mriedemhttp://lists.openstack.org/pipermail/openstack-dev/2015-June/thread.html#67283 is a bit nicer15:36
mnaseri.e. ceilometer is publishing info that is pretty much useless when it comes to network because i dont even know what port is publishing X usage15:36
mnaserit uses the port id generated by nova so 'qbr324342-324' which tells me nothing15:37
mnaserheck, ceilometer is in a great position to get a prometheus style api with all of its existing pollers15:37
TheJuliaI don't fully understand how they handle the data model or services, but it seems like a single point could work if there was no implication of what type of data was being provided by a general pull from ceilometer, but getting/sorting/selecting what is also kind of a conundrum because if I remember correctly, for Ceilometer, operators who used it with ironic also had to ensure ceilometer would pick the event_type out15:56
TheJuliaof the queue.15:56
bauzasmnaser: FWIW, that's the exact reason why I'll reply on the gerrit change and not on twitter15:57
bauzasmnaser: because when I see some other tweets, it's just "meh, nova cores are badboys"15:58
*** jaosorior has quit IRC16:02
jrolldansmith's comments on the review are A+, specifically:16:03
jroll> Nova exporting metrics for which it is authoritative is pushing things in the right direction of depending on something else for metrics, IMHO.16:03
jrollI see a screen or two here worth of discussion on implementation details, which is totally missing the point. mikal said:16:04
jroll> The previous answer was "we don't see the value in that, and have a ban on new implementations of (completely different other thing with similar name)", so I'd see that as a step forward.16:04
jroll> I wrote a experimental trivial implementation which I've now wandered off from because of these comments.16:04
*** jamesmcarthur has quit IRC16:04
*** jamesmcarthur has joined #openstack-tc16:05
jrollit appears he's looking to have a discussion on what he's trying to accomplish, bringing a POC along, I suspect the implementation details aren't important here16:06
TheJulia++16:06
jrollof course, I don't know if or where he attempted to start that conversation so it's hard to say what happened16:07
asettleMark Shuttleworth is doing the keynote FYI https://openinfradays.co.uk/16:08
jrollanyway, I feel like we all missed the point here (though it's certainly planted some seeds)16:09
asettleohffs wrong channel16:10
asettleSorry16:10
* mugsie shakes head at asettle16:11
asettle"Elect her to the TC" they said16:12
asettle"She's perfectly capable" the ysaid16:12
dhellmannTheJulia : I think we're mixing 2 unrelated comments. I think it makes sense to consider a goal to "expose useful internal metrics in a consistent way". I also think it makes sense to consider the compute agent as a source of those metrics, and to just add a REST API to it to provide them, rather than trying to collect them centrally and serve them some other way. As mriedem points out, there is a lot of existing work in this area to take into16:13
dhellmann account when considering implementation details deeper than what I've just said.16:13
jroll++16:15
dhellmannfor one thing, when we start talking about adding a similar rest endpoint to every service, I start wondering how we can take advantage of WSGI to provide middleware to do that16:17
dhellmannand of course, what does "useful internal metrics" even mean?16:18
TheJuliadhellmann: absolutely on all counts, I just want to us also avoid thinking that we can proceed down a path of one size fits all when I think we would be hurting ourselves if we dictated from the TC that this is the direction we think the community should head in. Ideally I think a couple projects should try and head in at least a POC direction if they see the need, and go from there. I know for internal process metrics,16:25
TheJuliathe original focus when the work was done in ironic was timing data... which also doesn't seem like the most useful thing in all cases.16:25
bauzasjroll: this isn't an implementation question16:30
bauzasjroll: IMHO, Nova could expose more if needed, for sure, but it's not the role of Nova to be responsible for sending specialized metrics to a very specific tool16:31
bauzasif so, we create a new API contract16:31
bauzaswhich I disagree16:32
*** e0ne has quit IRC16:32
jrollbauzas: that sounds like an implementation detail to me :)16:34
jrollthe point of the POC is to show that these metrics are useful, not to say this is how it should be done.16:34
mugsiebauzas: but if this "very specific tool" is used by most other projects in this space, we should look at it16:34
bauzascreating a new API contract to a specific 3rd-party tooling ?16:34
mugsienot just dismiss out of hand16:34
mugsieafaik there was a pluggalble point (or he was working on one) for swapping to statsd over /metrics16:35
bauzasmugsie: I'm not dismissing it, see my comment16:35
bauzasmugsie: there are many ways to collect nova metrics in prometheus already16:35
bauzasand I'm up for discussing what we can add more16:36
jrollbuilding a daemon which reads the notifications queue and emits metrics to $system is not a trivial thing that we can just handwave away as "this already works, we don't need to think about alternatives"16:37
jrollif for no reason other than the sheer volume of the message queue16:38
mugsiethat read as a dismissal. promethous (via a /metrics endpoint) is the defacto way people expose metrics for these system, not parsing rmq events or ceiolmeter16:38
bauzashere is the thing I can propose16:38
bauzasI very know a Prometheus maintainer16:38
bauzasI worked with him in the past, he also worked in the OpenStack community16:38
bauzashe knows ceilometer and all openstack things16:39
bauzasnow, he changed his role to some other job in Red Hat and is now a Prometheus maintainer16:39
bauzasso looks to me like the best person for discussing16:39
bauzasI can ask him to provide more thoughts in the change (he already did)16:40
TheJuliaI think one thing people are missing with this, we don't actually need to enforce sending over a message bus, a plugin can presently write the same data syslog, or with some work write it to... well... anything... which could be a service exposes a protected /metrics endpoint that is not on the same API surface as existing projects....16:42
*** cdent has joined #openstack-tc16:42
bauzasjroll: accepting a new time-based collecting thread for a specific 3rd party tooling isn't a trivial thing either fwiw16:42
bauzasand the nova community really cared about the support it needs16:43
bauzashttps://docs.openstack.org/nova/latest/contributor/policies.html#metrics-gathering16:43
jrollbauzas: again, implementation details. mikal is trying to show here that putting these specific metrics in prometheus is useful16:43
jrollwe have to agree on the "why" before the "how"16:43
mugsiebauzas: the metrics in that policy are different to the metrics that were proposed16:44
jroll+10016:44
jrollthis is about metrics on the nova-compute daemon, not the system it is running on (which that policy is about)16:45
bauzasand again, I'm not against discussing on how to expose some metrics that are useful16:45
bauzasor the why if you prefer16:45
*** cdent has left #openstack-tc16:46
bauzasjroll: there is already an openstack exporter https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config16:47
jrollbauzas: I was expressing that as a group, we tend to dive right into discussing implementation when people try to discuss general ideas. I wasn't saying anything about you specifically.16:47
dhellmannI'm less concerned with what metrics are collected than with getting teams to accept the idea that metrics collection is something we should be doing in the first place16:47
bauzasso, the question is : should we expose driver specific metrics to prometheus ? yes16:47
fungimaybe it's just me, but i can't help but think if we simply got all projects to provide an snmp callout hook and a nicely structured mib, then metrics would be solved in a way compatible with proper traditional tooling16:48
bauzasshould it be nova's responsibility to expose such metrics ? that's where I disagree16:48
jrollbauzas: I'm glad you agree - it appears mikal doesn't think the nova team as a whole agrees.16:48
fungii have a hard time imagining prometheus or any metrics collection system designed in the past 25 years lacks support for good old-fashioned snmp16:48
mugsiefungi: not really well16:48
bauzasjroll: with respect to mikal, the -2 was procedural16:48
fungimugsie: well, polling-oriented metrics collection system i mean16:49
jrollbauzas: it might not be nova's responsibility to emit the metrics, but it's nova's (or openstack's?) responsibility to make sure that getting the right metrics there is a good experience for operators16:49
bauzasjroll: I totally agree with that sentence16:49
jrollbauzas: the -2 was after the tweet; the tweet came from conversations.16:49
jrolldhellmann: +100016:50
fungimugsie: after all, your routers and switches and disk arrays and lights-out management interfaces and various other network-connected hardware are all providing snmp services already16:50
fungiyour power distribution units, your smart power strips, your hcav systems, your security systems...16:50
fungis/hcav/hvac/16:50
persia(not all: many of those run REST servers with various interesting things and don't provide SNMP anymore)16:51
fungieek, really? they've dropped their snmp services?16:51
persiaSome manuafacturers, yes.  Some are using derivatives of openBMC to provide data feeds.  Others custom embedded stuff.16:52
fungiwow, gross16:52
asettleOn another note for Twitter conversations today, anyone see this? https://twitter.com/jessfraz/status/110735838545908121816:52
asettleSpecifically the convo between Maish Saidel-Keesing and Steve Gordon16:53
bauzasI'm seriously considering to either grab popcorn everytime I go looking at twitter, or just stop looking16:53
fungiamusingly, i *just* popped some16:53
persiaasettle: Some of that comes from viewpoint rather than "cloud": http://www.openforumeurope.org/wp-content/uploads/2019/03/OFA_-_Opinion_Paper_-_Simon_Phipps_-_OSS_and_FRAND.pdf is a good recent read on different viewpoints16:54
asettlebauzas, I honestly don't look often for many reasons. It tends to be a cesspool of shit-human interactions and everyone touting their self-righteous bullshit.16:55
asettleNever-the-less, it has it's place for occasionally interesting debate16:55
jrollasettle: got a pointer to said conversation? I'm not seeing it16:55
asettlejroll, uno moment16:55
jrollta16:55
*** dtantsur is now known as dtantsur|afk16:56
fungiit's probably that i can't figure out twitter's ui, but i don't see any discussion between steve and maish there16:56
asettlejroll, probably from here: https://twitter.com/sparkycollier/status/110736518751290982516:56
asettleoh ffs fucking twitter16:56
jrolllol16:56
asettlehttps://twitter.com/maishsk/status/110738625811496550616:56
asettleTHERE we go16:56
asettleSorry folks.16:57
jrollfungi: twitter has threaded replies and it's super weird to navigate16:57
asettle^^ exactly that. You *think* you're clicking on the thread, but actually you're looking at a recipe for hot sauce16:57
jrolla recipe for hot sauce would be an improvement16:58
fungithis hot sauce recipe is entirely unsatisfying16:58
asettleHell yes it would16:58
asettleBut an improvement16:58
evrardjpMost of the discussion above should go into some kind of documentation for a future community goal, if we want to implement things across openstack in a common way -- we are just exposing "prior art" here16:59
fungiin another 6-ish hours when we close the nova and charms ptl elections you can follow up with the voter turnout for those17:00
asettle't would certain be interesting17:01
fungipreliminary numbers are 34% for openstack charms and 42% for nova17:01
*** ricolin has quit IRC17:01
asettleInnnnteresting17:01
bauzas42% is nice given our large community17:01
asettleYes ^^17:01
fungithe tc election turnout of 20% was less great, granted17:01
bauzasyeah, I've heard some friends unvoting for the TC17:02
asettleI do have some thoughts on that. But it ties into a lot of other conversations we've had in the past and I don't think it adds much. But it would be good to chat to you all in Denver.17:02
bauzasthey told me litterally "heh, I like you but I had no energy for reading emails and voting"17:02
asettleIs this currently an item on our agenda?17:02
mnaserasettle: i may or may not have asked for someone to volunteer to get our ptg etherpad started17:03
asettle>.> did ya now17:03
asettleI go away 8 days, I miss all the fun stuff17:03
fungialso, to maish's comment about "people that contribute on a regular basis to the code" the *majority* of our electorate is made up of people who have gotten a single patch merged to a repository17:04
mnaserasettle: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003916.html17:04
fungii expect if we had insight into *who* is and isn't voting, we'd see that the minority of people who contribute most of the code are roughly the same minority who participate in elections17:04
asettlefungi, I think you're right there17:05
asettlemnaser, sure. I'll grab it then. I'll put it as an AI for tomorrow as I'm absolutely shattered today and need to R&R a bit more.17:05
mnaserasettle: yay, thanks. :)17:05
asettleI'll reply to your email so it's public.17:05
asettleAnyone got last releases TC etherpad?17:07
asettleJust for a point of ref17:07
gmannasettle: https://etherpad.openstack.org/p/tc-stein-ptg17:09
asettleGracias gmann17:09
fungiokay, i take back what i said about the majority of the tc electorate having only one patch merged. it's 30% with only one change, 41% with 2 or fewer, 48% with 3 or fewer, and 54% with 4 or fewer17:12
fungiin the last election anyway17:12
fungi66% have up to 10 changes merged in the qualifying year for the last election17:13
*** jaypipes has joined #openstack-tc17:16
fungithe most active (by patch count) 20% of the electorate merged 29 or more changes that year17:16
fungiso we don't know which 20% voted in the tc election, but if it was the 20 most active by patch count then it was the people getting on average one patch merged every 13 days17:19
evrardjpIs it me, or the projects with little badges always look more serious than those without. I had one of the projects above with a proud badge of "tests passing" and "coverage 0%", and I was totally biased at first... "it has badges, it must be pro"17:19
evrardjpsneaky17:19
fungior phrased another way, the level of activity which results in getting a patch every couple weeks over the course of a year is roughly the same percent of our electorate who voted in the tc election17:20
evrardjpnice insights fungi17:21
*** jpich has quit IRC17:23
bauzasfungi: from the personal feedback I got (which is of course subject to debates), there is no clear relationship in between voting and contributing ratios17:32
bauzassome with high contribution levels just expressed some TC fatigue17:33
bauzas(which doesn't surprise me and which was one the reasons why I was running)17:33
bauzasanyway, time to wrap17:34
fungiyep, i don't necessarily expect them to be equivalent groups, more reacting to maishk's implication that the technical electorate are necessarily engaged contributors to the projects17:39
*** jamesmcarthur has quit IRC17:39
funginearly a third only every gave one change merged in a year, and the percent who voted are equal to the percent who get one change merged every couple weeks17:40
fungiso it depends a lot on what you consider "contribute on a regular basis" to mean17:41
fungiand one other point... where he compares it to participation for the osi board election, i don't know what their year-over-year turnout was like but the numbers they cite there are for this year's election which was wildly controversial. that sort of controversy definitely increases voter engagement, but... wow that was such a dumpster fire of a situation in their community17:45
fungii'd rather have a third the voter turnout they do and also a civil community who see elections as routine and not terribly "exciting"17:46
*** timburke has joined #openstack-tc17:48
mugsiefungi: yeah. OSI also had a lot of people joining just to vote in that election (due to its dumpster fire status), which would have shifted numbers17:51
*** jamesmcarthur has joined #openstack-tc17:54
*** jamesmcarthur has quit IRC18:12
*** jamesmcarthur has joined #openstack-tc18:14
*** jamesmcarthur has quit IRC18:14
*** jamesmcarthur has joined #openstack-tc18:14
*** efried has left #openstack-tc18:18
cmurphyframadate doesn't support time zones :( https://framagit.org/framasoft/framadate/framadate/issues/15818:25
evrardjpoh shucks18:33
evrardjpwhat is framadate based on?18:34
evrardjpit seems it's their own soft? wow I didn't know that18:35
evrardjplearning everyday18:35
fungilooks like php, so i'm not going to attempt to identify the time-handling bits in there18:36
fungithough neat that they're using zanata for translations18:37
*** diablo_rojo has quit IRC19:03
*** e0ne has joined #openstack-tc19:28
*** gouthamr_ has joined #openstack-tc19:34
*** gouthamr_ has quit IRC19:38
*** jamesmcarthur has quit IRC19:41
*** jamesmcarthur has joined #openstack-tc19:42
*** jamesmcarthur has quit IRC19:45
*** jamesmcarthur_ has joined #openstack-tc19:45
*** e0ne has quit IRC19:51
*** jamesmcarthur_ has quit IRC20:02
*** jamesmcarthur has joined #openstack-tc20:03
*** ijolliffe has quit IRC20:07
*** jamesmcarthur has quit IRC20:08
*** jamesmcarthur has joined #openstack-tc20:08
*** cdent has joined #openstack-tc20:19
*** diablo_rojo has joined #openstack-tc20:25
*** cdent has quit IRC20:26
*** tosky has quit IRC20:30
*** e0ne has joined #openstack-tc20:41
*** e0ne has quit IRC20:46
jrollasettle: I didn't get around to finding a place in governance for the team vision things today, but am going to prioritize it tomorrow morning21:01
*** cdent has joined #openstack-tc21:26
*** whoami-rajat has quit IRC21:52
*** diablo_rojo has quit IRC21:57
*** ijolliffe has joined #openstack-tc22:03
*** jamesmcarthur has quit IRC22:08
*** diablo_rojo has joined #openstack-tc22:30
*** jamesmcarthur has joined #openstack-tc22:30
* diablo_rojo puts election official hat on22:40
diablo_rojoSo.. its looking like at PTL election close we will have 6 PTLs with no IRC nicks provided. I know this is something that has been brought up in the past but I don't remember exactly what we decided in terms of handling this.22:40
* diablo_rojo puts community member hat on22:40
diablo_rojoI personally think that if you are a PTL, you should have a registered IRC nick.22:40
diablo_rojoProjects with PTLs missing provided IRC nicks: congress, freezer, karbor, senlin, watcher, zun22:42
cdentnice hat22:44
cdentseveral of those projects have ptls from china, yes?22:45
tonybcdent: correct22:45
openstackgerritMatt Riedemann proposed openstack/governance master: Document goal closure  https://review.openstack.org/64150722:45
gmanncongress PTL is on irc ekcs22:45
cdentfor whom irc has proven a roadblock?22:45
*** jamesmcarthur has quit IRC22:45
tonybgmann: Thanks.  It'd be good if Eric added that to his community profile22:45
*** jamesmcarthur has joined #openstack-tc22:46
gmannyeah, having mandatory IRC is not globally true22:46
gmanntonyb: yeah, i think he might have forgot or planning to move from IRC :). till now he conduct congress meeting regularly on IRC22:46
cdentI suspect a fair few people never know to even look at their foundation profile (although, one would hope a PTL would)22:47
cdentit's not something that is highly visible (at least from where I'm slouching)22:48
tonybcdent: Yeah, it's 'baked' into the election, summit, forum and PTG? planning process so I'd like to think that project leaders/influencers know about it and use it22:48
dtruongThe incoming Senlin PTL nickname is xuefeng .  But he is not on IRC much so I'm not sure it helps to point people to contact him that way.22:49
diablo_rojocdent, me many times. When I dont get email responses I go to IRC for many PTLs22:50
gmannML and email id is mandatory at least and we expect PTL to check the communication there actively22:50
cdentdiablo_rojo: If you're responding to "for whom..." what I meant there was that people in china have trouble accessing irc sometimes (either technically or socially), not who is hurt by people not being on irc22:51
diablo_rojoAhhhhh my bad22:51
diablo_rojomisinterpreted22:51
* tonyb gets outside the box what if we let/encouraged people set the NICK to 'weechat:$user_name' ?22:52
persiaI think occasional nonparticipation is better than explicit fragmentation.22:52
*** jamesmcarthur has quit IRC22:52
cdenttonyb: there have been suggestions like that in the past, but persia's comment hits one of the main objections22:53
persiaMost of the IRC "nonparticipants" seem to be folk who occasionally use a web gateway of some sort to attend meetings, but don't maintain presence (so chasing on IRC isn't a good alternative to email).22:53
cdentanother is some people being very -- on using wechat22:53
tonybWell isn't it just formalising the fragmentation we already have?22:53
diablo_rojoDo we want that formalization though? I feel like we shouldnt do that.22:54
persiatonyb: Not necessarily, unless there are a lot of formally organised openstack channels on weechat of which I'm unaware.22:54
tonyb*I* think that email and IRC are the correct tools and I don't want to fragment the community22:54
persiaIt's not just about nicks: it's also about project documentation specifying comms fora, etc.22:54
tonybWe can live with the status quo and just treat IRC as semi-mandatory but it won't help me reach $people22:55
diablo_rojogmann, I'll add their nicks to the results but we should get them to update their profiles.22:55
tonybGah /me has a meetign now :(22:55
gmannyeah, i (as TC liaison to congress) can ask him to add his nick in next congress meeting22:57
diablo_rojodtruong, thanks for the nick22:57
diablo_rojoSo with those two irc nicks we are now missing 422:58
*** marst has quit IRC23:00
fungiwith my tc member hat on, i think requiring them to read mailing list messages tagged with [ptl] or their project and to provide an e-mail address they will actually read regularly and respond from for private communication ought to be sufficient. i see irc as convenient and so nice to include, but i don't consider making it mandatory would really help23:02
fungijust requiring they provide an irc nick even if they don't ever use it is pointless23:02
fungiand if they are on irc regularly, they're likely to provide it already23:03
cdentthese things are true23:05
diablo_rojoIndeed. All true.23:06
*** mriedem is now known as mriedem_away23:07
*** jamesmcarthur has joined #openstack-tc23:14
*** jamesmcarthur has quit IRC23:30
*** tjgresha has joined #openstack-tc23:40
*** diablo_rojo has quit IRC23:46
*** jamesmcarthur has joined #openstack-tc23:51
*** tjgresha has left #openstack-tc23:52
*** penick has joined #openstack-tc23:56
*** diablo_rojo has joined #openstack-tc23:56
tonybfungi: so last cycle I proposed a patch to governance code to make IRC nic optional and at that time it was decided that it was mandatory.23:59
tonybso today if we add a ptl: block to projects.yaml without an irc_nick key/value the docs will fail to build23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!