15:00:24 #startmeeting tc 15:00:24 Meeting started Thu Jun 23 15:00:24 2022 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:24 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:24 The meeting name has been set to 'tc' 15:00:28 #topic Roll call 15:00:29 o/ 15:00:31 o/ 15:01:02 p/ 15:01:05 o/ 15:01:10 o/ 15:01:10 lol nailed it 15:01:14 o/ 15:01:22 o/ 15:02:41 o/ 15:02:49 In absence section: rosmaita (will miss 23 June and 30 June meetings) 15:03:36 let's start 15:03:52 today agenda #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions 15:04:01 #topic Follow up on past action items 15:04:19 no action item 15:04:27 #topic Gate health check 15:04:57 stream9 is broken with some qemu or libvirt problem -- sounds like everyone has removed it from voting on their queue 15:05:02 yeah 15:05:17 I'd have to go back and look, but I think it was "working" for _almost_ two months or so? 15:05:43 Do we have any errors we can report back? 15:05:47 the fips jobs also needed some fix to the base jobs, but that is merged now 15:05:50 at least yes for qemu things but it was not stable 15:06:15 since I made it voting, some issue keep coming at least detach volume 15:06:45 gmann: right but we don't know that those were really stream's fault vs just differences in behavior 15:07:34 spotz: it's already reported up the stack yes 15:07:37 yeah, we need some volunteer from c9s team side to help in debugging so that we can fix our side things and they can do their side things 15:07:44 Thanks dansmith 15:08:21 also on gate health, 15:08:22 gmann: From the Virtualation SiG or RDO? 15:08:25 just to update other tc-members, we discussed about c9s job stability in qa channel and clarkb mentioned there was some discussion with c9s maintainer in berlin summit . there was some talk on making it as party CI. 15:08:49 spotz: c9s maintainer 15:08:56 ok 15:09:34 yes, they're apparently also using zuul to do their centos stream testing, so they may be able to run some of the same jobs we do directly in their ci as well 15:09:39 we have many distro where we could not get much help/volunteer to debug things like opensuse, openEuler and we stopped their CI recently or last year 15:10:11 yeah, using us as a heavy test case for things they want to merge makes a lot of sense 15:10:13 anyways we need to work on such collaboration if we have c9s in our testing runtime 15:10:26 us using an unstable platform for our own testing is super frustrating 15:10:35 true 15:11:04 anyways I will add it as a separate topic and if we can call c9s folks during meeting. we can get some next steps 15:11:15 we also discussed using a stream9 periodic which, if it fails three times in a row, rings a bell for someone would be a good canary for "something has changed, which might be a bug or might be an issue on our end" 15:11:26 yeah 15:11:35 but it's going to take some commitment from folks who have interest and ability in connecting those dots 15:11:44 we did talk with bookwar at length about adjusting their workflow to be able to fast-track fixes and reverts/rollbacks rather than batching those up with normal platform updates 15:12:19 fungi: that's cool, but it would still require halting the pipeline for a lot of people for however long the notification and fast-track takes 15:12:21 they acknowledge that resolving regressions a few weeks or a month after they're reported is suboptimal 15:12:30 so I expect most people will still fall back to making it n-v immediately 15:13:02 Merged openstack/governance master: Add cinder ibm storwize charm https://review.opendev.org/c/openstack/governance/+/846900 15:13:06 #action gmann to add c9s testing collaboration topic in next meeting agenda and call c9s folks to help it to make stable distro to test in OpenStack 15:13:12 maybe some kind of solution would be to use some snapshots of c9stream images which we know that works fine? 15:13:17 so on another gate health topic: 15:13:23 yeah 15:13:29 gorka has been looking at the c-bak memory usage in the gate, and seems to have made some progress on some obscure libc thing that might be preventing it from releasing memory back to the system 15:13:40 it was something what bookwar already adviced to do on production for someone in the Summit 15:13:42 #link https://review.opendev.org/c/openstack/devstack/+/845805 15:13:53 so that's awesome, and he's been using the performance.json stuff to measure differences 15:14:09 would be good to get this in so that is easier: https://review.opendev.org/c/openstack/devstack/+/846198 15:14:30 slaweq: the downside to freezing the version of centos stream we use in testing is that it doesn't reflect how end users are consuming the same platform 15:14:35 still yet to be decided if we should apply the libc fixes globally or just to c-bak 15:14:38 +1, that is nice improvement 15:14:46 unless we want to start recommending to users that the freeze centos stream themselves and not get security fixes 15:15:01 fungi good point :) 15:15:32 eek 15:16:35 dansmith: libc fixes, link if there is any? 15:16:50 gmann: it's the link you provided 15:17:03 dansmith: ohk got it. 15:17:10 "fixes" might have been too strong.. "tweaks" perhaps :P 15:17:28 let's see how it goes and if we see more oom issues 15:18:04 there is a new ovs issue too aiui ykarel debugged it on a held node and looks like bad cpu feature testing 15:18:16 oh yeah, I saw that one too 15:18:40 manifests as a bunch of kernel stack traces, IIRC 15:18:53 ok 15:19:15 dansmith: ya I think they may be querying cpuid without checking for errors and the blazing ahead assuming the register values are all valid 15:19:25 wow 15:19:28 the errors can happen if you request a cpuid leaf value that is larger than what the cpu supports 15:19:35 o/ sorry i'm late. 15:19:49 but that was half awake and sick with what is apparently not covid digging this morning 15:19:53 so I could be completely wrong 15:20:38 ok, anything else on gate health ? 15:21:15 let's move then. 15:21:21 one thing 15:21:26 ok 15:21:34 about rechecks 15:21:52 I recently started doing some script which will check "naked" rechecks 15:21:58 and I have some initial data 15:22:05 nice 15:22:06 it's in https://etherpad.opendev.org/p/recheck-weekly-summary 15:22:15 it's not 100% accurate yet for sure 15:22:39 for example Cinder isn't good as there are some 3rd party comments with "recheck ..." there 15:22:54 those shouldn't be catched probably but they are 15:23:05 so I will have to improve my script for sure 15:23:08 this is nice data 15:23:09 still, pretty nice thanks! 15:23:18 slaweq: ++ Yeah, there are a number of CIs that give the recheck command. 15:23:23 slaweq: when you're more confident will you send that to the ML? 15:23:30 QA is 68.42 % :) I will add this recheck thigns in our weekly meeting to improve it 15:23:31 but in general it seems that we have many rechecks without reasons 15:23:37 dansmith yeah, I will 15:23:46 yeah sending it on ML will be good trigger to such projects 15:23:47 sweet! 15:23:49 thanks 15:23:54 but I need to verify it a bit more and fix issues which I saw there 15:24:02 that's all from me 15:24:12 nova is good in that stage only 4.8% 15:24:28 slaweq: thanks a lot for such data. 15:24:56 #action slaweq to send the recheck data on ML asking projects doing more bare recheck to start working on it 15:25:14 Kendall Nelson proposed openstack/governance-sigs master: Create the Environmental Sustainability SIG https://review.opendev.org/c/openstack/governance-sigs/+/845336 15:25:29 moving to ELK topic first as dpawlik have some appointment 15:25:35 #topic New ELK service dashboard: e-r service 15:25:40 dpawlik: go ahead 15:26:10 Changes in ci log processing: created account on docker hub for ci log processing project (credentials are in AWS secret service); enabled pushing logscraper and log gearman container images into the docker hub. Still waiting for "good looking" performance.json to push the metrics to the Opensearch. About elastic recheck: I know that there was 15:26:10 something done, but it is still not deployed. 15:26:35 * dpawlik sorry for log message, in 3 min I need to go 15:26:44 dpawlik: note zuul has very good support for managing that credential and building and uploading the image for you. 15:27:19 clarkb: indeed. The same secret is also encrypted in zuul 15:27:34 * clarkb built and tested a gerrit 3.6 installation as well as tested a 3.5->3.6 upgrade yesterday before anything merged using the zuul docker image management tools 15:27:35 dpawlik: what do you mean "good looking" ? 15:27:46 dpawlik: thanks for updates. on elastic recheck, do you know if rdo and master branch merge is done? 15:27:54 dansmith: I mean there is no bug like service: "v2" 15:28:45 dpawlik: that's more a bug in devstack itself.. do we need to wait before we can be indexing those things? It would have been handy to have it indexed for something last week.. things like the v2.0 problem will filter out once it's fixed right/ 15:28:51 I think we want to have some neutron folks to look into that, right dansmith ? 15:29:05 gmann: not sure anyone is actively doing it, but it'd be nice yeah 15:29:20 but same issue in other service also like aodh? 15:29:34 I can bring that topic in the neutron meeting next week 15:29:40 gmann: could be, I'm not sure.. I don't look at any jobs with aodh in it 15:29:52 dansmith: I can "filter" temporary such information and start pushing the metrics to the separate index, if it's a prio. If it can wait, I suggest to wait 15:30:10 I still think it's useful to have them indexed even if we have some inaccurate bits like that 15:30:36 agree 15:30:42 dpawlik: problem is, the data is useful as-is, and fixing these little things are not as high-prio as having the rest of the data available 15:31:16 Kendall Nelson proposed openstack/governance-sigs master: Create the Environmental Sustainability SIG https://review.opendev.org/c/openstack/governance-sigs/+/845336 15:31:23 dansmith: allright. Will do a fix for that. I will catch you on #openstack-infra 15:31:28 * dpawlik need to go. sorry 15:31:31 ack 15:32:15 thanks dpawlik 15:32:31 #topic Checks on Zed cycle tracker 15:32:41 #link https://etherpad.opendev.org/p/tc-zed-tracker 15:33:20 there are some progress but few items are still not started, correct me if I am wrong 15:33:31 1. Technical guidlines for logging levels with more example or scenarios 15:33:32 2. Drive the OSC as community-wide goals, be or find champion 15:33:45 3. Renovate translation SIG i18 15:33:53 4. Recognize the new contributor work in some way: 15:34:10 I think that "3. Remove release naming instead just use the number" can be marked as done now 15:34:25 slaweq: yes, I will do thanks 15:34:31 thx gmann 15:34:42 these 4 are not yet started, please check if it is assigned to you. 15:34:45 "2. Drive the OSC as community-wide goals, be or find champion": we wanted to wait for the corresponding forum session 15:34:57 arne_wiebalck: ok 15:35:08 diablo_rojo and I were there to discuss with Artem and Stephen 15:35:25 and also asked how the TC could help 15:35:30 Seems like we are getting to a good place with some larger projects to hold up as examples 15:35:37 which I think is why we had held off last time 15:35:56 Nova for example is nearly at parity. 15:36:24 what the SDK team needs would be have people fixing "little" tasks in the various projects 15:36:49 something for interns, students .. for instance 15:36:55 Yeah, more project involvement 15:37:03 I can bring the students if people are willing to mentor 15:37:10 as per last status i remember we still not got agreement from glance team on this? 15:37:19 one suggestion was to have burn-down charts to see where the projects are 15:37:37 arne_wiebalck: yeah that will be great and good data to check if we can make it goal or not 15:37:41 gmann: AFAIK, there is no desire from the (rest of) glance team to support OSC/SDK 15:37:41 gmann: this is correct, I don't think anyone from the glance team was ther 15:37:44 *there 15:38:03 dansmith: ok 15:38:04 but I'm also not sure there's much missing really 15:38:11 but I'll be glad to bring it back up again, 15:38:12 Yeah noone was there from glance 15:38:20 dansmith: the general suggestion was to "increase the community pressure" 15:38:21 dansmith: +1 15:38:24 but I think we likely need some sort of "thou shalt do this" in order to push it along 15:38:37 dansmith: ++ 15:38:43 arne_wiebalck: yep, certainly more requests from users would be very helpful as well :) 15:38:51 dansmith: ++ 15:39:01 I think having some chart on what left for what projects will give good idea for next step 15:39:09 I think noone objected that the overall goal makes sense 15:39:13 dansmith, shoot! I was going to bring that up to the ops meetup but I got distracted by other topics. 15:39:20 arne_wiebalck, +1 15:39:32 diablo_rojo: bring up "go nag glance about osc"? yeah, that'd be helpful :) 15:39:33 The users/operators in the room at the forum session seemed all for it 15:39:40 yeah, there is no doubt that operators want it its just us not finishing it 15:39:56 at least they want it since many years :) 15:39:57 that's good 15:40:03 dansmith, well not exactly, more the OSC topic in general, but I maybe if the request came from another source? :D 15:40:03 in neutron we should be more or less good - I recently though that we have support for everything in OSC but recently gtema discovered that some advanced services like e.g. vpnaas are still using neutronclient python bindings 15:40:14 amotoki was going to check that with gtema AFAIK 15:40:14 ah okay 15:40:18 gmann: right, this is where ops may also be able to help 15:40:23 gmann: with interns etc 15:40:27 +1 15:40:40 The more the merrier 15:40:42 gmann: if there would be an easy to access list of low-hanging fruits 15:41:02 so should we start it as popup team like RBAC and proceed or all pre-work on-boarding intern etc before we can shape it as goal 15:41:05 i remember attempting a spreadsheet matching client with osc commands about 2 years ago https://docs.google.com/spreadsheets/d/1C5f0BQcfD8czzhKJmWQ-nnDq_6_D9W2xsWsEh-zHIAo/edit#gid=0 15:41:21 arne_wiebalck, stephenfin has generated that in the past per project, but I think its a bit tedious, maybe we can talk to him and work on generating them together? 15:41:48 diablo_rojo: the low-hanging-fruits list? 15:42:26 diablo_rojo: or the discrepancy list? 15:42:50 anyway, let's talk to him and see what he can provide :) 15:42:55 arne_wiebalck, kinda both? 15:43:00 so what is next step 1. popup team 2. prepare the fresh current state and speed up the work 15:43:03 yeah, related ofc 15:43:21 but for ops the easier to see what needs to be done the better 15:43:25 having some regular set of people driving it with regular meeting is much needed i think 15:44:47 * arne_wiebalck hears silence when more meetings are mentioned :-D 15:45:02 actually, whoops, the above link is not my document. google drive search just randomly popped up a similar one :/ 15:45:12 ohk :) 15:45:23 arne_wiebalck, I feel the same. 15:45:31 arne_wiebalck: we need some regular work from some dedicated group here otherwise we know how it is going since many years 15:45:42 Lets keep things as process less as possible until we have decided to make it a goal 15:45:55 we have been asking project to finish the work for many years and we are not able to complete it 15:46:06 I'm not anti-meeting for this, I think the policy one has been quite useful in that regard 15:46:18 I dunno who is likely to be that chair though 15:46:26 popup team is not process overahead 15:46:53 yeah, RBAC has some good progress at least with popup team. 15:47:17 otherwise we end up saying "we need that much work to do so please do it and no one do it :)" 15:47:21 gmann, its not a matter of no meetings == no progress. Its no people == no progress. 15:47:50 how about we check with artem and stephen if they would like to drive this? 15:47:52 meeting i mean drive the things in actual discussion/doing work than just status things 15:48:02 arne_wiebalck: sure. that will be great 15:48:19 ok, let me check with them then 15:48:32 if not, then we re-assess 15:48:34 and if we will have some regular work/meeting/people on it, it can attract more people to help 15:48:41 thanks 15:48:42 gmann: ++ 15:49:10 #action arne_wiebalck to check with stephen and artem about driving the OSC work as popup team or any other dedicated group way 15:49:32 moving next? 15:49:49 #topic RBAC community-wide goal 15:49:51 #link https://etherpad.opendev.org/p/rbac-zed-ptg#L171 15:50:25 so also, the sdk team already has regular meetings once a month anyway 15:50:26 in tuesday policy popup call, we discussed the operator feedback from ops meeting, KDDI (japanese telco) 15:50:53 diablo_rojo: yeah in any ways if a dedicated set or people can work on it 15:51:14 and we have the new direction based on that feedback on RBAC, its on etherpad 15:51:43 I need to update the same in community wide goal docuement. I have started that and almost 80% done but not up yet 15:52:03 I will finish that new direction write up and there we can discuss the same 15:52:22 gmann: can you please update the meeting information on https://meetings.opendev.org/#Secure_Default_Policies_Popup-Team_Meeting 15:52:28 So I think a meeting is fine once we have a lay of the land, but I think we need to get the list of disparities before we start doing those because up till now, all the meetings we've had have just been status which is not all that productive. 15:52:28 key agreement there was to postponed the 'scope' which is what operator want 15:52:48 i keep missing those meetings :/ 15:52:52 knikolla: yeah that is in my list, I will do it. sorry got distracted on that 15:53:10 knikolla: sorry about it, I keep this wiki up to date https://wiki.openstack.org/wiki/Consistent_and_Secure_Default_Policies_Popup_Team#Meeting 15:53:17 sent invite calander on ML too 15:53:31 but I will update the irc-meeting repo also 15:53:58 dansmith if I remember your availability, you will be out tomorrow right? 15:54:18 checking if I can push goal document update today how soon you can review it 15:54:21 ping me when you push the irc-meetings update and i'll be sure to expedite approval 15:54:45 gmann: correct.. I'll be ducking out a little early today, but if it's soon I can look 15:54:53 it is not just update but adding support of non irc meeting in irc-meeting repo so more work 15:55:07 dansmith: ok, let me try how fast i can 15:55:19 oh, yeah i don't think irc-meetings is a place to document meetings which don't happen in irc 15:55:22 that is all on RBAC, let's wait for the goal do up 15:55:41 fungi: we really need a place where we can document it so it's in the ics files automatically 15:55:45 if we want somewhere to track non-irc meetings, that's probably a bigger discussion 15:55:48 fungi: yeah or remove that thing from wiki and add some other calendar 15:56:16 anyways let's discuss that separatly, as we are running out of time 15:56:18 #topic Create the Environmental Sustainability SIG 15:56:38 there is new SIG proposal #link https://review.opendev.org/c/openstack/governance-sigs/+/845336 15:56:49 I think that is discussed in forum sessions. 15:56:59 this seems to answer a question i was asking diablo_rojo about in #opendev, namely that the sig is an openstack sig not some foundation-level effort? 15:57:18 I commented there about its scope. as it scopes is not just openstack but openinfra [projects 15:57:33 ahh, thanks 15:57:38 I think to make it productive we should start small 15:57:38 so doing it as openstack SIG is not that suitable place for it 15:57:44 and its should be openstack first 15:57:51 because thats where most of the efforts are 15:58:07 Makes sense. 15:58:09 make progress and then push it to a higher level and hold up the progress we made from the openstack scope first 15:58:17 diablo_rojo: we do have working group as one possible place to start like diversity group, interop group 15:58:31 I think if we start too high level first, we risk a lot of sprawl and not much progress. 15:58:38 if it's an openstack sig, i would expect the irc channel to be #openstack-envirosig instead of #openinfra-envirosig, unless there's a foundation sig for which the openstack sig participants are a subset 15:58:41 gmann, yeah I know 15:58:56 fungi, I wanted room for growth? 15:59:15 diablo_rojo: my concern is if we see it as openinfra level then it is better to do it now instead of changing all latter which is more work 15:59:28 like irc channel as fungi mentioned 15:59:34 at the foundation level we don't have any concept of "sigs" that i'm aware of 15:59:45 fungi, that too 15:59:48 its working groups 15:59:55 fungi: Working Group 16:00:07 so openstack-envirosig or openinfra-envirowg 16:00:12 and it can work same way as we want as SIG, like interop WG doing 16:00:22 sig is an openstack thing, wg is a foundation thing 16:00:29 they own code repo, marketting things, connect community or so 16:00:48 or what diversity group is doing which is close example of this new env SIG 16:01:18 I think we need other input from people interested in the efforts. 16:01:21 I feel converting things later make more confusing which include the supporting channel, communication ways change 16:01:26 I am just one opinion. 16:01:50 Which is why I set up the IRC channel to be openinfra and not openstack 16:01:52 and changes to those things end up with losing people helping there 16:02:15 my concern is more around overloading organizational terminology. reusing the "sig" term for foundation level organizational concepts is going to increase confusion 16:02:25 I think it is more confusing to change in the future. 16:02:37 why not we can do all things as WG if we have interested people can help three too instead of SIG 16:03:11 I do not think any difference there to do the things as WG or SIG if we have people to help here 16:03:49 fungi: we do not need to say it as SIG but we can sao Working Group like Diversity WG and drive the things to all OpenInfra projects 16:04:17 we are running out of time 16:04:27 gmann: agreed. i'm referring to the proposed changes to add the irc channel for it, which uses "openinfra" and "sig" together in teh name 16:04:37 fungi: ah right 16:05:12 I think the people that want to participate in the group should decide what level it goes at. I am more than happy to do it either way. I just want to see us move forward. 16:05:26 let's continue the discussion in next meeting. and meanwhile tc-members and other please input your feedback on gerrit #link https://review.opendev.org/c/openstack/governance-sigs/+/845336 16:05:34 sure 16:06:02 ++ 16:06:03 diablo_rojo: I will keep it in meeting agenda till we get clear direction on this 16:06:18 thanks for working on it. 16:06:30 that is all for today, we are out of time now 16:06:46 thanks everyone for joining 16:06:53 #endmeeting