Thursday, 2021-05-13

*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-tc02:33
*** ralonsoh has joined #openstack-tc04:31
*** vishalmanchanda has joined #openstack-tc05:06
*** e0ne has joined #openstack-tc05:31
*** e0ne has quit IRC05:54
*** e0ne has joined #openstack-tc06:05
*** e0ne has quit IRC06:10
*** e0ne has joined #openstack-tc06:15
*** lpetrut has joined #openstack-tc06:23
*** e0ne has quit IRC06:24
*** slaweq has joined #openstack-tc06:26
*** dklyle has quit IRC07:20
*** andrewbonney has joined #openstack-tc07:25
*** tosky has joined #openstack-tc07:47
*** lpetrut has quit IRC14:21
*** dklyle has joined #openstack-tc14:51
spotzo/14:58
gmanntc-members meeting time14:59
gmann#startmeeting tc15:00
openstackMeeting started Thu May 13 15:00:09 2021 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: tc)"15:00
openstackThe meeting name has been set to 'tc'15:00
gmann#topic Roll call15:00
*** openstack changes topic to "Roll call (Meeting topic: tc)"15:00
ricolino/15:00
gmanno/15:00
spotzo/15:00
dansmitho/15:00
jungleboyjo/15:02
gmannyoctozepto6 and Belmiro  would not be able to join as per "Apologies for Absence" section15:02
gmannso let's start15:02
gmann#link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions15:03
gmanntoday agenda ^^15:03
gmann#topic Follow up on past action items15:03
*** openstack changes topic to "Follow up on past action items (Meeting topic: tc)"15:03
gmanngmann to add SIG chair/co-chair info in sig doc site15:03
gmannI pushed the patch #link https://review.opendev.org/c/openstack/governance-sigs/+/79082715:03
gmannit is in governance-sig so good to merge with 2nd +215:04
gmannplease review and +A accordingly.15:04
gmanngmann to prepare the etherpad for draft proposal of PTL+TC periodic meeting15:04
gmannI have created the etherpad #link https://etherpad.opendev.org/p/tc-ptl-interaction15:05
gmannwhich we will discuss in later topic15:05
gmann#topic Gate health check (dansmith/yoctozepto)15:05
*** openstack changes topic to "Gate health check (dansmith/yoctozepto) (Meeting topic: tc)"15:05
gmanndansmith: any news you would like to share15:05
dansmithso, I've seen a bunch of cinder fails in the last week15:05
dansmithlots of rechecks due to volume related test failures,15:05
dansmithso I assume the lull in activity was masking a lot of that, but I think we've still got work to do there15:06
jungleboyj:-(15:06
dansmiththere's also the discussion about decommissioning the ELK infrastructure,15:06
dansmithwhich is well-justified but will significantly reduce our visibility into the frequency of these failures15:06
dansmithand limit our ability to work on the health as a system15:07
dansmithnot sure what to do about it, other than what has been communicated on the list at this point15:07
gmannhumm, that is valid concern15:07
jungleboyjdansmith:  Yeah, I was a little concerned with that discussion.15:07
dansmithbut clearly we either need someone to commit to a serious amount of time to rebuild it, and maintain it going forward,15:07
dansmithor we're just going to have to deal without it15:07
dansmithjungleboyj: me too15:07
dansmithso, that's pretty much it for me..not a particularly happy picture all said15:08
jungleboyjI was surprised by that discussion.15:08
dansmithI was surprised that more people didn't consider it critical infrastructure15:09
spotzIf there's issues and no one to maintain it it makes sense. Kinda goes with the frequent call outs for more opendev help needed15:09
dansmithbut also somewhat surprised at how much resource it consumes15:09
fungiit is not a lightweight service, no15:09
fungimostly because it's a massive amount of data our jobs are logging15:10
gmannsurly it will make hard for maintaining the CI/CD and tracking issue etc15:10
fungiit's a classic "big data problem" which needs experienced "big data people" to do it well15:11
dansmithfungi: well, that's really your fault15:12
dansmithas my wife says "I value X less because you make X look so easy" :)15:12
dansmithand ELK is definitely something I've taken for granted because it was just there and easy :)15:12
gmannAs this is no maintainer issues and kind of must-require for upstream CI/CD smooth maintenance, I feel this is one of the issue we should bring to broader audience. especially company or users.15:13
fungiplease do15:13
gmannand bring this to Board also which should add some resources.15:14
dansmithwell, they did on the list, and without a deadline15:14
gmannwe were asking for infra help since many years15:14
jungleboyjgmann:  ++  I think this is an issue that needs to be highlighted.15:14
spotzYeah it definitely needs attention and some help15:15
gmannfungi: any deadline for stopping those service?15:15
fungiit's one of many services we're decommissioning, it just happens to be a higher-profile one15:15
fungigmann: not decided yet at least, so no for now15:15
gmannok,15:16
gmannso let's take two action here15:16
fungiwe've in the past month or so taken out limesurvey (survey.o.o), asterisk (pbx.o.o) and our mqtt broker (firehose.o.o)15:16
spotzCan we move it to the bottom of the list to give us more time?:)15:16
fungiand there will certainly be more15:16
fungiand yes, like i said, clark started the discussion early so folks can come up with solutions15:17
fungiit's not going away yet, it just needs to15:17
gmann1. ML to especially highlight  the ELK services  and ask for help explicitly with ref to current thread15:18
gmann2. Bring this to Board meeting which is planned on June 29.15:18
spotzI'm pretty sure folks saw the email but not the importance15:18
dansmithspotz: because they don't know what ELK is doing for us, or because of some other reason?15:19
dansmithnot enough shouting?15:19
jungleboyjBased on the responses I am sure a lot of people are hoping the question will just go away.  :-)15:19
fungii can guarantee the question will eventually go away, just maybe not in the way those people are hoping15:20
spotzdansmith: don't know what it's doing for us15:20
dansmithokay15:20
jungleboyjfungi:  ++15:20
dansmithunfortunately a minority of people work on gate issues (which is an issue itself), so those are the ones likely to feel strongly15:20
spotzKinda like we're retiring asterisk, well I don't use it so no big deal. But it could be tied into an aleerting system whiichh I'm not aware of...15:20
dansmithremoving it will raise the bar for difficulty on that already unsung and unpopular task15:20
gmannagree15:21
dansmithso it's kinda self-fulfilling that it's not a big deal to more people15:21
gmanndansmith:  do you mind doing 1st one, raising critical alarm on ML for ELK services ?15:21
fungii'm curious how the current discussion on the ml could have reached a broader audience, personally15:22
dansmithgmann: well, I can, but I'm a bit confused about what the current email is lacking15:22
dansmithright what fungi said15:22
dansmithwe can send another email with "ELK: why you should care" but I'm not sure it will really garner more attention, but if people think it's important then I can15:22
fungiit sounds like, based on dpawlik's recent response, that rdo might be interested in running something15:22
gmannI am fine with only current but it has few more services listed also so in case we want to highlight ELK explicitly15:22
*** clarkb has joined #openstack-tc15:23
dansmithfungi: yeah but I'm worried they don't realize the 1TiB/day requirement :P15:23
dansmithgmann: ah, okay15:23
spotzfungi I can check on the RDO side15:23
gmannespecially what we must continue for day to day debugging15:23
gmannwhich is what we can highlights in various platform like newsletter, Board etc15:24
dansmithokay well, I'll work on a draft and look for some reviews15:24
gmann+1 thanks15:24
gmann#action dansmith  Draft the ML to explicit Alert/highlight  the ELK services maintenance help with ref to current thread15:25
jungleboyjdansmith:  ++ Thanks.15:25
gmannI will bring this on Board agenda15:25
gmann#action gmann to bring the ELK/infra help to Board15:25
gmannany other things to discuss in Gate check topic?15:26
dansmithnot from me15:26
gmannthanks for brining this and keeping eyes on gate things15:26
gmann#topic Planning for TC + PTL interaction (gmann)15:26
*** openstack changes topic to "Planning for TC + PTL interaction (gmann) (Meeting topic: tc)"15:26
gmannI composed a draft in #link https://etherpad.opendev.org/p/tc-ptl-interaction15:27
gmannI would like to ask each members to add your opinion in etherpad and then we can discuss it in next meeting15:27
gmannor feel free to add more discussion point too if you want15:28
gmann#topic Xena cycle tracker status check15:29
*** openstack changes topic to "Xena cycle tracker status check (Meeting topic: tc)"15:29
spotzAdded a comment for if we hold it AT the PTG:)15:29
gmannspotz: cool thanks15:29
gmann#link https://etherpad.opendev.org/p/tc-xena-tracker15:29
gmannthis is tracker's tacking week15:29
gmannas this is still initial of cycle, we can just discuss the one having progress instead of tracking all item's progress15:30
gmannone is from dansmith on charter change on ATC->AC #link https://review.opendev.org/c/openstack/governance/+/79009215:31
gmannwe have discussion on gerrit on this15:31
gmannanything to discuss here or continue on gerrit itself ?15:32
ricolinI think we can keep the discussion in review:)15:32
ricolinI mean for https://review.opendev.org/c/openstack/governance/+/79009215:33
gmannon fungi's point on Bylaw, i left comment there for a way to comply Bylaws. basically what we did for "OpenStack Technical Committee Approved Release" term15:33
gmannhope that should be fine from Bylaws perspective15:33
gmannmoving next15:34
gmannAudit and finish the previous cycle pending community-wide goal work15:34
gmannI started few patches for PTL guide15:35
gmann#link https://review.opendev.org/q/topic:%2522project-ptl-and-contrib-docs%2522+status:open15:35
gmannnot many but just to start15:35
fungiyes, as long as the tc indicates that whatever it's producing is the atcs list the bylaws requires, then it's fine. the bylaws already has a carve-out for adding people who aren't code contributors, just need to have some way to bridge the naming divide15:35
spotzOh in regards to PTL elections, diablo_rojo mentioned it's based on ATC status. Could it be AAC status plus a contrib in that project?15:35
gmannyeah15:35
spotzAC not AAC15:35
fungiit's technically "apc" for ptl elections, not atc15:35
gmannyeah, PTL are APC always15:36
spotzOk so reword:) The APC list build off of AC + project contrib?15:36
fungiptl elections also aren't mentioned in the bylaws, so the tc can do whatever it likes there as long as the requirements for editing the charter are observed15:36
gmannmeans ATC which can be contributor in x project would have vote for project y pTL election15:36
gmann*would not15:37
gmannand yes, we in TC repo will keep list of ATC (renamed as AC )15:37
ricolinIsn't the ATC less for a year?15:38
gmannspotz: APC is project contrib only. AC = APC+ other active contributors15:38
fungiatc is actually for "365 days" according to the bylaws, i think15:38
ricolinfungi, that's what I remember:)15:39
gmannyeah15:39
gmann" The term shall be for three hundred and sixty five days after the date of approval of the application by the Technical Committee."15:39
gmannsection 3.a.ii in https://www.openstack.org/legal/technical-committee-member-policy/15:40
fungifor election purposes we've basically considered that to be interpreted as two release cycles15:40
gmannright15:40
spotzOk shoot:(15:40
gmannanyways let's review the patch and continue discussion there15:42
gmannmoving next15:43
gmannTC members to drive the Y cycle community wide goal15:43
gmannricolin: any updates after you sent the ML15:43
ricolingmann, No much really, mainly checking around with backlogs for goals ans make some survey15:44
gmanngreat. may be we need to cleanup backlog also15:44
gmannwe did in Ussuri cycle at some extend15:44
ricolinI think so15:44
gmannthanks for working on this.15:45
ricolinI can make some clean15:45
ricolinPersonally, I like the container image support one15:45
gmannI do not see any other progress in etherpad please bring here if there is any and i missed that15:45
gmannricolin: +115:45
ricolinwondering if mnaser have any idea how can we push that one forward15:46
gmannI think there was patch also for proposing it as goal15:47
ricolingmann, if RBAC is proposed, we only need one goal to search for15:47
gmannricolin: #link https://review.opendev.org/c/openstack/governance/+/72010715:47
ricolingmann, that was abandoned for an year15:47
gmannyeah15:47
gmannricolin: yeah I will compose RBAC one, most probably next week or so15:48
dansmithI think a number of projects are working on rbac stuff,15:48
dansmithso that seems like a no-brainer right?15:48
ricolinIf we allow some project skip from the container goal15:48
gmannyeah, many are left with less attention15:48
ricolinit might get easier to be accepted15:48
gmannas per discussion with lbragstad we could do goal this cycle and close the popup team but as we are not doing any goal in this cycle we could do in Y15:48
spotzlbragstad is out on paternity:)15:49
gmannand to make sure all project are done and we can try out the default enabling new rbac via oslo or so15:49
gmannyeah15:49
jungleboyjspotz:  Oh wow,  Didn't know that.  Good for him!15:49
gmannanything else on goal or tracker status?15:50
ricolintc-members please help to suggest Y-cycle goal if you see any potential fits15:51
ricolinnope15:51
gmann+1.15:51
ricolinbut we need to do it asap15:51
*** e0ne has joined #openstack-tc15:52
gmann#topic Open Reviews15:52
*** openstack changes topic to "Open Reviews (Meeting topic: tc)"15:52
gmann#link https://review.opendev.org/q/project:openstack/governance+is:open15:52
gmannwe have two open review15:52
gmannthis is eligible for approve #link https://review.opendev.org/c/openstack/governance/+/79058215:52
gmannother in discussion.15:53
gmannthat's all for today I think15:53
gmannthanks all for joining15:53
gmann#endmeeting15:53
*** openstack changes topic to "OpenStack Technical Committee office hours: Tuesdays at 09:00 UTC, Wednesdays at 01:00 UTC, and Thursdays at 15:00 UTC | https://governance.openstack.org/tc/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-tc/"15:53
openstackMeeting ended Thu May 13 15:53:45 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:53
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-05-13-15.00.html15:53
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-05-13-15.00.txt15:53
openstackLog:            http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-05-13-15.00.log.html15:53
spotzThanks gmann and everryone15:53
jungleboyjThank you!15:54
gmannricolin: spotz jungleboyj can any of you +A this, as 2nd reviewer can +A in governance-sig review https://review.opendev.org/c/openstack/governance-sigs/+/79082715:55
ricolingmann, done15:56
gmannthanks15:56
dansmithgmann: fungi: so, looking at the current thread, I'm worried about starting a new one to highlight ELK and losing or splintering the discussion that has already happened and will come out of it15:56
spotz+A or +W just making sure I have access:)15:56
dansmithgmann: fungi: So what if I did that thing one should never do, and reply but change the subject a bit to kinda start a sub-thread about importance and resources?15:57
fungidansmith: creating subthreads by changing the subject is plenty fine by mailing list etiquette standards15:58
gmanndansmith: +1. that will work15:58
fungijust keep the in-reply-to header untouched15:58
dansmithfungi: when properly done to create a subthread, sure, but it's abused too often for me to consider it a best practice :)15:59
fungiso that sane mail user agents will still thread it together with the original discussion15:59
fungii would consider this a great example of when a subthread is appropriate15:59
dansmithyup16:00
dansmithclarkb: did you see the tc meeting where I was asked to summarize and put out a call for help?16:05
dansmithclarkb: just want to make sure you realize the reply coming to the thread (later) will be not intended for you, but to provide a linkable summary of specifically the ELK concerns that we can use when going door to door looking for handouts :)16:06
*** openstackgerrit has joined #openstack-tc16:09
openstackgerritMerged openstack/governance master: Retire puppet-glare  https://review.opendev.org/c/openstack/governance/+/79058216:09
clarkbdansmith: ya fungi pointed out it was being discussed here so I joined (late) and also read scrollback via eavesdrop logs16:09
dansmithack16:10
clarkbthat reminds me I wanted to point out the spotz  that part of the motivation with removing asterisk is that it never had huge uptake but jitsi meet is seeing good use. Basically we deployed a better tool for the use case that people actually seem to use16:12
clarkband survey was only used twice which made it an easy candidate16:13
dansmithyeah asterisk was deployed in like 2012 or something before there was as wide of adoption of non-IRC communication anyway I think16:14
openstackgerritMerged openstack/governance-sigs master: Add email and irc nick for SIG chair/co-chair  https://review.opendev.org/c/openstack/governance-sigs/+/79082716:19
spotzclarkb we used to run it at the house but it was ages ago. Didn't know it was still a thing:)16:25
* dansmith still does16:26
dansmithgmann: fungi: clarkb: https://termbin.com/69e9 <- comments please16:27
fungidansmith: lgtm. there's a lot more in there than elk and e-r (the 80 worker threads which preprocess and feed log files to the indexer, also openstack-health and the underlying subunit sql db), but that's probably already waxing into a lot of detail16:33
dansmithfungi: I snipped out the subunit2sql bit because I didn't realize that was related16:34
clarkbdansmith: subunit2sql uses the same ingestion infrastructure but to a different backend (mysql vs elasticsearch)16:34
dansmithokay, so subunit-worker, and the trove one.. health is something else rtight?16:35
fungiopenstack-health is basically the dashboard for that16:35
fungiand api16:35
gmannyeah16:35
fungihttp://status.openstack.org/openstack-health/#/16:36
fungino idea if that's been maintained lately16:36
gmannopenstack-health is fine to shutdown as we in QA team also does not much maintainer for that but  I can bring this in QA office hour too16:36
clarkbgmann: thanks16:36
dansmithokay I'll just throw those resources back in, but I was trying to avoid the word salad of topics in the subject to resonate with more people (if that makes sense)16:36
gmannI used to search it for sometime like check test health or so but I have not used in recently past (1 year ot so)16:37
gmannor16:37
gmannand I am not JS developer so cannot help in health dashboard :(16:38
mnaseri'm sorry, i missed todays meeting16:38
mnaserit's eid today so i'm a bit out of it :)16:38
gmannah, forgot about that. happy Eid mnaser16:39
mnaserthanks gmann :)16:39
gmanndansmith: fungi clarkb if I understand correctly for this sub thread, we will only highlight help for ELK and leave rest like o-h as OK-TO-SHUTDOWN right?16:40
dansmithgmann: I thought that was the goal of your suggestion.. to specifically explain what ELK means to the project and the impact of taking it down16:41
dansmithI wanted to focus on that, and hence the snipping out of anything that isn't critical to understanding that impact and the requirements we have16:41
dansmithI'm not convinced this email is necessary, so I don't want to send it unless you all think it will be helpful, even if it's just an executive summary people can link to when asking for resource16:42
fungiwhich bits to focus on is a question of which ones people are relying on, but as far as what opendev will shut down i still expect it's all of that, at this point we're mainly looking for people to build something similar maybe but i doubt we can accommodate it directly within opendev's remaining fleet of services16:42
gmanndansmith: +1, sorry many thing at same time, so did not read the full logs16:42
dansmithspotz asserted that it probably wasn't well understood so ... /cc spotz ^ :)16:42
gmanndansmith: yeah for executive etc to know that "what is this and we want help not just shutting down as no use"16:43
fungithe opendev sysadmins are currently at the point of turning things off so we can focus on keeping gerrit, zuul, mailing lists, etherpad, et cetera up and running16:44
spotzYeah if no one realizes the purpose it serves they don't know it's important to keep it16:44
gmanndansmith: this lgtm overall - https://termbin.com/69e916:45
*** timburke_ has joined #openstack-tc16:45
dansmithspotz: ack, I've tried to do that summary in ^ .. does that look good to you as well?16:46
*** timburke has quit IRC16:48
spotzdansmith: reading then I've got a live stream:)16:51
spotzdansmith: looks good!16:53
dansmithcool, thanks16:53
*** andrewbonney has quit IRC17:28
zanebI've wondered for a while if https://prometheus.io/docs/practices/instrumentation/#logging would fulfil some of the functions of ELK a lot more efficiently. not clear how you would then trace that back to individual jobs though17:38
*** ralonsoh has quit IRC17:43
fungipresumably the same way, you tag every indexed object with relevant metadata17:45
zanebyou'd end up with billions of timeseries; effectively that would reduce to the same thing we're doing now17:49
clarkbyou would be able to see trneds but no identify specifics I think17:54
clarkbif you want specifics that requires disk :/17:54
fungior more likely disks17:55
clarkbwhere that gets tricky for us is we do a lot of pre merge testing. You want a way to avoid mixing pre merge with post merge trneds. You could potentially split in half that way instead and get a lot of value18:00
clarkbrather than doing it by job/change/etc18:00
*** e0ne has quit IRC19:22
*** e0ne has joined #openstack-tc19:23
*** e0ne has quit IRC19:24
*** e0ne has joined #openstack-tc19:25
*** e0ne has quit IRC19:26
*** e0ne_ has joined #openstack-tc19:26
*** johnthetubaguy has quit IRC20:09
*** johnthetubaguy has joined #openstack-tc20:09
-openstackstatus- NOTICE: Zuul is in the process of migrating to a new VM and will be restarted shortly.20:41
*** vishalmanchanda has quit IRC20:43
*** slaweq has quit IRC20:49
*** e0ne_ has quit IRC21:21
*** corvus has quit IRC21:58
-openstackstatus- NOTICE: We are cautiously optimistic that Zuul is functional now on the new server. We ran into some unexpected problems and want to do another restart in the near future to ensure a revert addresses the source of that problem.22:05
*** corvus has joined #openstack-tc22:56
*** tosky has quit IRC23:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!