Thursday, 2021-09-30

*** diablo_rojo is now known as Guest137304:33
*** ykarel_ is now known as ykarel06:07
*** ykarel is now known as ykarel|lunch09:13
*** ykarel|lunch is now known as ykarel10:05
*** tobberydberg_ is now known as tobberydberg11:51
*** pojadhav- is now known as pojadhav13:22
*** pojadhav- is now known as pojadhav13:54
gmanntc-members: meeting time15:00
gmann#startmeeting tc15:00
opendevmeetMeeting started Thu Sep 30 15:00:10 2021 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'tc'15:00
yoctozeptoo/15:00
jungleboyjo/15:00
gmannfungi: clarkb dpawlik ping15:00
gmann #topic Roll call15:00
gmanno/15:00
dpawliko/15:00
fungimnaser is still presenting on openinfra.live15:00
belmoreirao/15:00
*** diablo_rojooooooo is now known as diablo_rojo15:00
clarkbhi15:01
gmannonly four tc-members, let's wait for 1 more min. 15:02
mnasero/ kinda?15:02
gmannlet's start15:03
gmann#topic Follow up on past action items15:03
gmanngmann to remove 'TC tags analysis' from agenda15:03
gmanndone15:03
gmann#topic Gate health check (dansmith/yoctozepto)15:03
gmanndansmith: yoctozepto go ahead15:03
yoctozeptogates are unbelievably happy our I'm just oblivious to gate issues15:03
yoctozeptoor*15:04
yoctozepto:-)15:04
jungleboyj:-)15:04
gmannafter lot of fixes :)15:04
yoctozeptoyeah, but it was much smaller than previous similar events15:04
yoctozeptoat least how I remembered them15:04
gmannapache2 on devstack uwsgi setup is fixed which blocked gate for almost 1.5 days15:04
gmannwe also faced two more issue: 1. nova ceph on stable and 2. nova grenade on master failure15:05
fungimit's fedora 34 mirror we sync from in opendev was broken, we just moments ago completed a switch to a mirror at uh.edu15:05
gmannboth are fixed now15:05
gmann+115:05
clarkbdid the dstat replacement issue ever get addressed?15:05
fungialso we've got another zuul restart coming up today for some more stabilization fixes in advance of xena release week15:05
yoctozeptoclarkb: no, unfortunately15:06
gmannrelease is planned for next week so let's keep eyes on gate.15:06
yoctozeptoguess it's not bad enough15:06
yoctozeptoI'll look at it at some time for sure15:06
gmannthanks15:07
gmannanything else on gate?15:07
gmann#topic Place to maintain the external hosted ELK, E-R, O-H services15:08
gmannAs background and to refresh the memory:15:08
gmannAs there is/was no maintainer to maintain ELK services in OpenDev and these services were proposed to shutdown, we raised the ELK service help in community as well as to Board of Directors.15:08
gmannAllison(BoD chair) putting a great effort on this and it seems, there are good possibility (still under discussion) of getting hosted ELK service from OpenSearch (an open source fork of Elasticsearch).15:09
gmannin addition to hosted ELK, dpawlik volunteer to maintain it.15:09
gmannthis etherpad has more details #link https://etherpad.opendev.org/p/elk-service-maintenance-plan15:10
dpawlikexactly gmann15:10
gmannNow two open questions on us:15:10
gmannlet's go one by one15:10
gmann1. is OpenSearch can fulfill our requirement for ELK service usage?15:10
clarkbNote OpenSearch doesn't do the L in ELK. Only the E and K15:10
gmannok15:10
gmannclarkb: fungi dpawlik you can give more details on us15:11
yoctozeptocan we use fluentd instead of logstash then?15:11
gmanns/us/this15:11
fungiand there's a bunch more tech in that suite openstack is relying on, elasticsearch is merely the queryable storage backend15:11
yoctozeptogmann: perhaps on us as well, who knows15:11
wendaryoctozepto: yes, and also, OpenSearch is working on an alternative to logstash, it just isn't ready yet15:11
gmannwendar: hi15:12
wendarhi15:12
fungiyoctozepto: whoever's going to run it can presumably design it to use whatever ingestion software they want, just note that there's already a pile of logstash grok parsers15:12
gmannfungi: yes, basically "ELK, Elastic-Recheck, and OpenStack-Health services"15:12
fungigrok parser rules15:12
yoctozeptofungi: yeah, I was wondering how much investment was done already15:12
yoctozeptothen a logstash clone would be better15:12
wendarand, there is also an existing plugin for logstash to write to OpenSearch, so that might be the simplest in terms of immediate migration15:12
yoctozeptoor that ^15:13
gmannyeah15:13
gmannany other challenge we see on OpenSearch option?15:14
* mnaser actually here15:14
gmannmnaser: hi15:15
fungialso opendev's sysadmins don't want to be responsible for custom notification bits for this, so we need to be able to drop the gearman event bits from our base job and instead the ingestion should work on polling available apis15:15
mnaseringestion rules are a thing, fluentd stuff is a thing, there's a few other ways that we can get data into opensearch which is going to have to probably be dealt with by whoever decides to pick this up tbh15:15
yoctozeptomhm15:16
dpawlikgmann: on disabling security plugin, we can have elasticsearch like it is running right now on opendev, but the proxy rules what operations like PUT/POST/DELETE needs to be filtered15:16
mnaseri figure whoever will be pushing for this will probably get to be the authoritive answer at the end of the day15:16
dpawlikby disabling*15:16
diablo_rojoSorry am late, was hosting an Open Infra Live episode15:16
gmannmnaser: yeah we will be discussing that as next item which is important 15:16
fungidpawlik: however, according to the opensearch folks at amazon we talked to, kibana isn't going to work correctly through a read-only proxy. they suggested switching to a shared read-only account for queries15:17
ricolinsorry I'm late15:17
dpawlikfungi: on rdo we share the read only user account15:18
gmannshared read-only should work though 15:18
clarkbright one of the biggest things that kept us from upgraded prior to losing help was new ES needs new Kibana and new Kiabana can't operate anonymously15:18
fungisomething like what we do for the openstack-health subuinit database15:18
clarkbopensearch addresses this by open sourcing AAA tooling15:18
fungis/subuinit/subunit/15:18
fungi(though we actually have a query proxy in front of the mysql api for that database too, now that oi think about it, i should add that to the list in the pad)15:19
dpawlikclarkb: that's why we do a trick to inject header with user credentials to automatically  login into the kibana15:20
dpawlikclarkb: by saying "operate" you mean just to make a query and check visualizations, right?15:21
fungiright, they alluded to that on the opensearch call... basically embed http basic auth creds15:21
clarkbdpawlik: no, it needed write access to the database which we've removed with the proxy15:21
clarkbanyway it sounds like their authentication and authorization tooling allows sufficient fine tuning to make kibana work15:22
fungifrom what they were saying, new kibana sort of "drops privileges" when a user authenticates to it, but needs essentially administrative access through the api for other functions15:22
dpawlikfungi: right, there are some indexes that the user should be able to have write access to it15:23
clarkbI don't think kibana will be an issue. And if it is it sounds like there may be sufficient interest on the opensearch side for potentailly improving things15:24
gmanntrue15:24
dpawlikclarkb: +115:24
clarkbThe bigger issue is figuring out updating ingestion so that it works against a modern system. Then operating and maintaining that system15:24
clarkbSince OpenSearch as a service leaves that to the user15:24
gmannwendar: clarkb how is their contract going to be, yearly?15:24
wendarTom described it as "perpetual", and also said to chat with him if the current amount/year isn't enough (it probably will be)15:25
gmannclarkb: yeah. let's figure out where to place those which we will discuss next.15:25
gmannwendar: ok15:26
gmannI think we seems ok on opensearch on our requirements and later we can see what all we can truncate  the things15:27
gmannnow next question is on operating it15:27
gmann2. Where to maintain the hosted ELK? OpenStack?  or OpenDev?15:27
gmannDue to less/no long term maintainers for ELK and more focus on key services, the OpenDev team decided to cease maintenance for the ELK, Elastic-Recheck, and OpenStack-Health services.15:28
TheJuliaI think there is a third option, spin up a new namespace on opendev because its use can be more generally applicable.15:28
gmannI added few option in L16 https://etherpad.opendev.org/p/elk-service-maintenance-plan15:28
fungii think we're well past the time for opendev to continue being responsible for these services. we tried, we asked for help, we finally decided it was too far gone and have planned to shut it down15:28
gmannfungi: agree15:29
TheJuliaso the determination then becomes, where does it need to live and the tooling operate, and much of that would be implementation detail based on re-engineering right?15:29
fungiyes, i believe so15:30
fungiat the moment, if something happens and the servers it's running on suffer a catastrophic failure, we have no way to rebuild what's there15:30
gmanncurrent options: 15:30
gmannOption1: Add it under TACT SIG 15:30
gmannOption2: Add it in OpenStack QA. 15:30
TheJuliaso in essence, we're kind of spinning an entirely new project then15:30
gmannOption3: Start a new project or SIG. 15:30
gmannor Option4: (for completeness) Stop relying on these services.  but we want these so let's keep it for later15:31
yoctozeptoOption2 is not feasible, we are already too small15:31
clarkbre this tooling being generally applicable, I think it totally is and tehre is a lot of neat stuff that can be done in the psace. But we've not seen interst from outside of openstack really15:31
gmannyoctozepto: yeah, same opinion from me too and listed in etherpad too15:31
yoctozeptook15:31
TheJuliaclarkb: could that also be the connoation that comes with the openstack name?15:31
TheJuliaconnatation15:32
clarkbTheJulia: maybe? What I've heard from the zuul devs is that it just isn't necessary for their workflows because Zuul's logging is much easier to sfit through15:32
yoctozeptogmann: but I can't see this in the etherpad, you sure you wrote it down?15:32
gmannyoctozepto: ah, missed. 15:32
clarkbI think a certain scale is necessary before you cross the threshold where developer time is better spent learning lucene instead of grep15:33
* yoctozepto glad it's not him being blind15:33
gmanndone15:33
yoctozeptothanks15:33
fungithe reason it's been useful for openstack is a matter of scale, there are numerous failures which occur a fraction of a percent of the time, but you need to run tens or hundreds of thousands of times to get a large enough sample size to isolate and classify them15:33
gmannyeah, scale is all key thing here15:33
clarkbZuul being a much smaller project also has a stronger sense of ownership for issues that pop up. If my tests fail due to some random reason I go and debug it and hopefully fix it and many of the zuul devs do that here15:34
fungithe other projects in opendev just don't have the change volume openstack does, and don't run similar integration tests nearly as many times, so they lack sample size to make good use of this sort of solution15:34
clarkbWith many more devs in openstack not everyone can be expected to do that and rechecks happen which allow a larger percentage of these things through long term15:34
yoctozepto++ on all the thoughts15:34
gmannditto, +1. and with the integration scale openstack has it is natural. 15:35
gmannok so we left with two option now15:35
gmannTACT SIG or new project/SIG under openstack15:35
TheJuliadoes it *have* to be under openstack?15:36
gmannfor TACT SIG fungi its again you need to give opinion. 15:36
gmannTheJulia: it is better if openstack is only user of it and no one else want it15:36
fungiTheJulia: it doesn't have to be an openstack project or sig, no15:36
gmannfungi: then new open infra project ?15:36
wendarTheJulia: It doesn't have to be under openstack, but openstack is currently the only user, so it makes some sense, at least as a starting point.15:36
yoctozeptoit would look better under opendev to gather more folks around the solution15:37
fungithe software could still be generally applicable even if the instance of the service being run isn't used beyond openstack projects15:37
TheJuliagmann: it will never be able to gain external usage or adoption outside of openstack if we brand it openstack upfront15:37
yoctozeptoeven if only for naming sake15:37
yoctozeptoTheJulia ++15:37
clarkbTheJulia: wow that is an extremely pessimistic take15:37
gmannopendev seems not option so I think openstack is default then15:37
jungleboyjTheJulia:  ++15:37
gmannor what other option we have?15:37
jungleboyjclarkb:  It is pessimistic but probably realistic.15:38
wendarWe can always move it out of openstack later, if/when we start to generalize the code.15:38
clarkbsure but OpenDev the non openstack location for doing this work has said "hey lets try and figure this out" for a few years now15:38
gmannwendar: exactly.15:38
clarkband literally we have had negative interest (the team has shrunk over that time)15:38
fungithe tact sig doesn't really have any "resources", it's effectively an umbrella to cover things people happen to be working on which didn't fit as opendev services but also didn't really make sense as an openstack project team15:38
clarkbI don't think calling it something other than openstack helps either is kinda my point there15:38
TheJuliaclarkb: the name comes with a lot, good and bad unfortunately15:38
gmannfungi: we have maintainer now its just we need a place it acn better fit for now15:38
wendarThe OpenSearch team even said they'd be willing to consider taking it on as a project, if it becomes generally useful.15:39
TheJuliawendar: that sounds like a great possibility15:39
gmannfungi: and if we again face no maintainer issue then it goes away15:39
wendar(But, there's quite a bit of work to do, before that would make sense anyway.)15:39
fungiwe've been winding down all the services the opendev sysadmins were running outside of opendev itself, in order to be able to keep our core services in good shape. the only one which we don't currently have plans to discontinue (but would like to) is wiki.openstack.org15:39
clarkbat least putting it in openstack whee people seem to care would give it potential to survive15:39
gmannclarkb: indeed 15:39
gmannand if more project care and we have more maintainer then we can things of opendev or at external place again15:40
gmannso let's go with the current situation 'only openstack need/care about it'15:40
gmannand openstack would not object if it goes at some central place for others to use15:41
fungiwe have said in the past that we'd manage server resources at a low level for any project-specific efforts which needed them, but we stopped considering that viable some years back when it became apparent that people would build things which weren't long-term manageable, users would start to rely on them, then the services would be abandoned and the sysadmin group would be pressured to15:42
fungikeep them running (ask.openstack.org as a prime example)15:42
gmannso we have (will have) hosted service and maintainer to maintain it so place hardly matter for now and starting in openstack make sense to me and later we can again re-think on place based on interest of other projects.15:43
yoctozeptook, so a SIG?15:43
gmannnew SIG or TACT?15:44
yoctozeptoDebuggability SIG15:44
fungiwiki.openstack.org is perhaps an even better example. we had volunteers from wikimedia foundation running it, they left, it wasn't really set up compatible with how we did our configuration management, so now it's in a really precarious state where basically nobody knows how to tend to it, its very outdated (probably rife with security holes) and we'd be hard pressed to rebuild it if15:44
fungisomething happened to the server it's on15:44
yoctozeptofungi: good to know15:44
TheJuliafungi: sounds like everyone just needs to begin moving off of it (projects wise)15:44
fungiTheJulia: well, it's an example of what i don't want whatever this replacement is to become15:45
spotz__Hey all15:45
* TheJulia adds to ironic's PTG topic list15:45
TheJuliafungi: ++15:45
gmannI am fine on new SIG if fungi think it is ok and leave TACT SIG as it is as per its current scope ? 15:46
gmannbut for me TACT SIG seems more close and even fungi there helping on migration or any other question, sharing same IRC channel or so15:46
fungieach time we've proposed discontinuing the wiki server over the years, there's been uproar but no new volunteers to fix the problem. and the elastic-recheck/openstack-health discontinuation is at risk of going down the same path (though hopefully not, as we've learned those lessons i think?)15:46
fungii'm fine with the coordination happening through the tact sig, as a peg to hang the sign on, but just be aware that doesn't come with any guaranteed resources15:47
gmannopenstack-health we can again re-think if needed or not. QA is not using it so much as it used to be15:47
TheJuliafungi: yeah, we need to adapt and change processes, and it sounds like getting rid of the gearman bits would help, I hope at least.15:47
clarkbgmann: in fact I think the health services has been dead for a few months now and no one noticed :)15:48
gmannfungi: true, and same thing if there is maintainer issue then yes we re-think on it. Will not put everything on you.15:48
gmannclarkb: :)15:48
gmannlet me bring it again in QA meeting on 'who need it' ? i raised it few month back but we did not decided 15:49
fungito be fair, hald of elastic-recheck has been broken for months/years so i don't think all of its functionality is being relied on, and sometimes the backend is down for a week with nobody noticing15:49
fungis/hald/half/15:49
clarkbfungi: yup it tends to get more use during periods of time like now with release candidates and stabilization and people caring about the gate15:50
gmannyeah15:50
clarkbthen no one cares during the 3 months of merge everything as quickly as possible15:50
gmannok so are out of time for other agenda15:50
yoctozepto(btw, I think TC should now concern itself with wiki going down to help TACT SIG too)15:50
gmannso as summary 1. OpenSearch is good option 2. TACT SIG is new place for hosted ELK maintenance 15:51
yoctozepto(as I doubt it's common knowledge)15:51
gmannany more things to discuss on this, though I will keep it in agenda for next meeting too15:51
clarkbnote it isn't just the wiki and the ELK stack. It is all the services right? This is why I continue to try and remind people that helping OpenDev ensures this doesn't happen to a growing list of services. But I think many have sort of accepted that ship has sailed at this point?15:52
gmannthanks for joining and great discussion on this  clarkb fungi dpawlik wendar fungi 15:52
gmannmoving next15:52
clarkbNow I don't think OpenStack should be the only project to help either. Zuul does a bunch of help but primarily on the CI side. Airship and Starlginx could probably be involved more15:52
gmann#topic Project Health checks framework15:52
gmannclarkb: indeed. 15:52
gmannricolin: I think as we discussed in previous meeting, we are going to work on documentation in parallel to current tool right?15:53
gmann#link https://etherpad.opendev.org/p/health_check15:53
ricolinIndeed we have that action15:54
gmannricolin: ok, any more updates or so you would like to share today?15:54
ricolinnot yet starts it15:54
gmannok, no issue. if we can have somthing by PTG then it is good to discuss there15:54
ricolinNP15:54
gmannmoving next15:54
gmann#toppic OpenStack newsletter15:55
gmannwe have two items there for newsletter, please add more if you think to highlight #link https://etherpad.opendev.org/p/newsletter-openstack-news15:55
gmannmay be by tomorrow as deadline? diablo_rojo ?15:55
clarkbYou should probably have something about the new release?15:56
clarkbOr maybe that is implied15:56
gmannclarkb: yeah not sure when it is going to publish before or after release ?15:56
gmannwe usually add it explicitly if i remember correctly 15:57
diablo_rojogmann, actually its tuesday15:57
diablo_rojofor me to write the content15:57
diablo_rojoso preferably Monday15:57
diablo_rojolol15:57
gmanndiablo_rojo: ok then we can add it if it is released after Xena then add 15:58
gmann#topic Open Reviews15:58
gmann#link https://review.opendev.org/q/projects:openstack/governance+is:open15:58
gmannthis one to review and provide early feedback if you have any #link https://review.opendev.org/c/openstack/governance/+/81072115:58
diablo_rojoI expect there is going to be a focus on Xena in the next one if not this one. We can mention it but I don't think I will be needing to spend a lot of time on that particular topic. 15:59
gmanndiablo_rojo: +115:59
gmannone last thing, next meeting is going to be Video call.16:00
gmannthat's all from me today.16:00
gmannthanks all for joining16:00
gmann#endmeeting16:00
opendevmeetMeeting ended Thu Sep 30 16:00:25 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-30-15.00.html16:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-30-15.00.txt16:00
opendevmeetLog:            https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-30-15.00.log.html16:00
jungleboyjThanks!16:00
dansmithgmann: sorry again16:01
dansmithgmann: we were having a tree taken down.. appointment was at noon and they show up at 8am. imagine our surprise :/16:01
dansmithactually before 8am.. chainsaws were running by 8.16:01
gmanndansmith: np!, I will try to summarize it over email in case. otherwise ELK topic we will continue in next week call too16:02
diablo_rojoThanks gmann!16:02
dansmithgmann: okay I think the openinfra live thing overlaps with the tc meeting next week, so I might also be unable to attend that one, depending on how long it goes16:02
gmanndansmith: yeah it is exactly after the openinfra let's see. I am going to ask hosting help to yoctozepto 16:03
dansmithack16:03
gmanndiablo_rojo: added a note there in etherpad. 16:05
diablo_rojogmann, thank you!16:05
yoctozeptogmann: thanks, and ok - I will only be watching it, not participating, can host the meeting16:07
gmannmnaser: fungi please review this project-config for new charm https://review.opendev.org/c/openstack/project-config/+/80901216:07
gmannneeded by governance patch16:07
gmannyoctozepto: thakns, I will add he details of call and agenda and let you to take care next. I will join but again depends on what time we finish the openinfra sessions 16:08
gmannthanks 16:08
gmann*the details16:08
yoctozeptosure thing16:09
fungidansmith: gmann: technically the tc meeting starts the same time openinfra.live ends, but the show has a tendency to "run long" sometimes and can end up overlapping16:09
dansmithfungi: typ16:10
dansmither, yup16:10
gmannyeah16:10
dpawlikthanks gmann!16:25
gmanndpawlik: your welcome, and thanks again to you for helping us on this.16:31
gmannricolin: I added 'documentation part as ' TODO in project health check etherpad in case we do not forget #link https://etherpad.opendev.org/p/health_check16:32
*** diablo_rojo is now known as Guest142117:02

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!