15:00:19 <gmann> #startmeeting tc
15:00:19 <opendevmeet> Meeting started Thu Mar 10 15:00:19 2022 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:19 <opendevmeet> The meeting name has been set to 'tc'
15:00:23 <gmann> #topic Roll call
15:00:27 <jungleboyj> o/
15:00:28 <dansmith> o/
15:00:33 <rosmaita> o/
15:00:38 <slaweq> o/
15:00:38 <dmendiza[m]> 🙋
15:01:08 <gmann> o/
15:02:17 <spotz_> o/
15:02:21 <arne_wiebalck> o/
15:02:44 <gmann> #topic Follow up on past action items
15:03:05 <gmann> - gmann to add the FIPs testing direction in next meeting agenda
15:03:21 <gmann> added in agenda and we will discuss in end of today meeting
15:03:30 <gmann> - slaweq to send the ML to call for PTL for adjutant and trove project
15:03:35 <gmann> slaweq:  sent on ML
15:03:47 <slaweq> I did :)
15:04:05 <slaweq> and there are some volunteers potentially
15:04:05 <gmann> #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027557.html
15:04:12 <gmann> #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027558.html
15:04:28 <slaweq> thx gmann for links
15:04:36 <gmann> +1 yeah, let's discuss those in leaderless project topic
15:04:44 <gmann> #topic Gate health check
15:04:59 <gmann> there is more traffic yesterday and took more time to merge patches but not worst.
15:05:03 <gmann> any other news
15:05:09 <dansmith> ugh, I meant to do some digging,
15:05:45 <dansmith> but I was chasing down what seemed like a common failure in some patches related to volume resize (I think)
15:05:56 <dansmith> and it seemed like maybe it was a problem with the worker instead of any of our stuff,
15:06:17 <dansmith> but then I applied updates and restarted my system and lost some state (like a noob)
15:06:43 <jungleboyj> :-(
15:06:52 <gmann> oh
15:06:58 <dansmith> that was the only thing I was chasing at the time.. seemed like I've also seen a bunch of blind rechecks lately, have others?
15:07:28 <gmann> slaweq: you have the script for recheck for neutorn. right?
15:07:40 <gmann> can we extend that for all project and we can see how much we are doing?
15:07:48 <gmann> script to count the recheck
15:07:59 <slaweq> gmann: I have script which can count number of rechecks in every project
15:08:19 <dansmith> ah, that'd be interesting
15:08:36 <gmann> yeah and will be good to check here weekly
15:08:36 <slaweq> I can prepare such data for patches from all projects
15:08:46 <gmann> +1
15:08:58 <jungleboyj> slaweq:  That is interesting.
15:08:59 <slaweq> sure, I will prepare it for next week
15:09:07 <gmann> perfect, thanks.
15:09:10 <dansmith> agree, especially in light of losing some capacity, we might need to lead a charge to get people to actually look for the reasons before they recheck
15:09:27 <gmann> I remember neutron CI meeting discuss/monitor sich data in very nice way.
15:09:40 <slaweq> dansmith: yeah, that's why I did it and we are checking it every week
15:09:46 <dansmith> slaweq: ++
15:09:52 <slaweq> now it's good as we are below 1 recheck in averega
15:10:01 <gmann> nice
15:10:05 <slaweq> but we had weeks where we had e.g. 10 rechecks to get patch merge :)
15:10:10 <slaweq> 10 in average
15:10:17 <gmann> slaweq: let's add a etherpad where we can put weekly data there and check in meeting.
15:10:30 <slaweq> gmann: sure, I will do that
15:10:33 <gmann> thanks
15:11:25 <gmann> for centos9-stream failure on detach. this can help but I have not debugged the failure on this yet #link https://review.opendev.org/c/openstack/tempest/+/831608
15:11:48 <gmann> I thought of doing before rc-1 but could not finish.
15:12:11 <gmann> anything else on gate things?
15:13:13 <gmann> #topic Z cycle Leaderless projects
15:13:54 <gmann> we are left with two project and slaweq got success to get the volunteer for trove
15:13:56 <gmann> #link https://etherpad.opendev.org/p/zed-leaderless
15:14:03 <gmann> #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027654.html
15:14:19 <gmann> wu.chunyang volunteer to maintain it as they are using it
15:16:01 <gmann> checking their contribution in Trove or so
15:16:14 <jungleboyj> ++
15:16:51 <gmann> I find these #link https://review.opendev.org/q/owner:wuchunyang%2540yovole.com+project:openstack/trove
15:17:27 <slaweq> I looked here https://www.stackalytics.io/?release=all&user_id=wu.chunyang&metric=commits
15:17:34 <gmann> yeah
15:17:37 <gmann> I think we can go ahead and assign the PTL, any other opinion
15:17:45 <slaweq> +1
15:17:46 <gmann> especially there is no maintainer in that project
15:18:03 <gmann> if no objection here, I will propose the PTL assignment patch
15:18:22 <jungleboyj> +1
15:18:47 <rosmaita> no objection
15:19:17 <gmann> ok, I will propose patch after meeting
15:19:25 <gmann> 1nd project Adjutant
15:19:26 <gmann> Braden Albert asked to wait until March end #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027626.html
15:19:48 <gmann> which we can wait and we can check in PTG. based on situation that time we can take call.
15:20:04 <gmann> that is all on leaderless projects, anything to discuss on this topic?
15:20:22 <jungleboyj> Well, that isn't too bad.
15:20:53 <gmann> yeah, more leaderless project this time but fastest to fill them.
15:21:10 <jungleboyj> ++ Focus on the positive.
15:21:30 <gmann> which indicates we need to adjust our election process or so. but anyways we will discuss it in PTG
15:22:06 <gmann> #topic PTG Preparation
15:22:17 <gmann> we selected the time slot of TC discussions #link http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027584.html
15:22:41 <gmann> Please plan accordingly. and topic wise schedule we will prepare later
15:23:10 <gmann> keep adding the topics you would like to discuss this etherpad #link https://etherpad.opendev.org/p/tc-zed-ptg
15:23:23 <gmann> that is all for PTG for now, anything else?
15:23:41 <gmann> and this etherpad for TC+PTL sessions https://etherpad.opendev.org/p/tc-ptl-interaction-zed
15:23:44 <gmann> #link https://etherpad.opendev.org/p/tc-ptl-interaction-zed
15:23:55 <ade_lee_> is there a specific time to give updates on community goals like fips?
15:24:09 <ade_lee_> or should I add to the above agenda?
15:24:44 <gmann> ade_lee_: we have not prepared the exact topic wise schedule but please add your preference in etherpad and we can try for that L39 in #link https://etherpad.opendev.org/p/tc-zed-ptg
15:24:59 <gmann> arne_wiebalck: oh or you mean for today meeting?
15:25:07 <gmann> ade_lee_: ^^
15:25:21 <ade_lee_> ah gotcha -- I just saw that line
15:25:25 <gmann> cool
15:26:02 <gmann> ade_lee_: if time conflict you can add the details in etherpad also but I  can try to schedule as per your availability
15:26:33 <ade_lee_> gmann, ok great thanks
15:26:44 <gmann> and yes, if anyone has preference on any topic to avoid conflict with any specific time/day please mention that in etherpad
15:27:12 <gmann> I do not promise to avoid 100% conflict but we can try :)
15:27:40 <gmann> moving to next topic...
15:27:59 <gmann> #topic Check on Yoga Tracker progress
15:28:02 <gmann> #link https://etherpad.opendev.org/p/tc-yoga-tracker
15:28:18 <gmann> let's go by one by one as we are very  close to Yoga cycle end
15:28:34 <gmann> 1. Technical Guidelines: dansmith
15:29:10 <dansmith> yeah I suck
15:29:12 <gmann> dansmith: I think you are already doing lot of things, but just to know any plan for unified limit like next cycle?
15:29:24 <dansmith> yeah, nova merged theirs now,
15:29:30 <gmann> yeah
15:29:32 <dansmith> so we have two examples to go by
15:29:47 <dansmith> I still want to do this, it was just always the easiest thing to push aside
15:30:39 <gmann> sure, no worry. if you have time but having nova completed is good progress overall
15:30:46 <dansmith> ack
15:30:54 <gmann> 3. TAG framework removal proposal
15:31:28 <gmann> for this few tags are pending specially release script patch and VMT one which needs to wait for OpenStack website to be updated by foundation
15:31:43 <gmann> so this is good progress and can be completed based on these deps
15:32:08 <gmann> 4. Improvement in project governance - diablo_rojo_phone
15:32:23 <gmann> for this we discussed to have a 'Tech pre-review'  process proposal
15:32:30 <gmann> but I did see anything up yet
15:33:01 <gmann> 5. Release name process change -belmiro
15:33:04 <gmann> #link https://review.opendev.org/c/openstack/governance/+/829563
15:33:20 <gmann> Belmiro proposal is in good shape and with more +1 there
15:33:49 <gmann> and as discussed in last or this week, we will discuss the release name-removal things separately in PTG as it need lot of discussion and deps
15:34:48 <gmann> jungleboyj: diablo_rojo_phone arne_wiebalck spotz knikolla ^^ if you would like to review it as I am planning to merge it early tomorrow (early in CST)
15:35:03 * jungleboyj just opened a tab for it.  :-)
15:35:04 <gmann> if no strong objection from yoru review or here
15:35:32 <gmann> 6. Community-wide goals: ricolin gmann
15:35:33 <diablo_rojo_phone> I can review today
15:35:47 <gmann> jungleboyj: diablo_rojo_phone thanks
15:36:01 <gmann> we have done the community restructure but left with the checklist
15:36:18 <gmann> I will try if I can push something up next week
15:36:32 <gmann> 7. User Survey Feedback Analysis/Responses - jungleboyj
15:36:42 <jungleboyj> I too suck.  :-)
15:36:53 <jungleboyj> I have started on that.  Will have things ready for the PTG.
15:37:11 <gmann> jungleboyj: perfect, thanks. and before PTG is good time
15:37:30 <dansmith> jungleboyj: suck buddies? wait, that.. nevermind :)
15:37:40 * jungleboyj laughs
15:37:47 <gmann> anything else on Yoga tracker items progress?
15:37:59 <jungleboyj> Well, that was something I never expected from dansmith .  :-)
15:38:23 <slaweq> dansmith: You made my day with "suck buddies" :D
15:38:31 <dansmith> slaweq: yw :P
15:38:32 <gmann> :P
15:38:48 <gmann> #topic Adding FIPs gate jobs and direction on how to run them in optimize way w.r.t. infra resource shortage.
15:39:09 <gmann> ade_lee_: can you please brief about what is your plan about adding FIPs job in gate ?
15:39:28 <gmann> and then we can discuss what best we can do to utilize the infra resource in best way
15:39:49 <gmann> I remember you were trying to add as experimental in Tempest gate
15:40:04 <gmann> which is not merged yet
15:40:10 <ade_lee_> gmann, well - its been really project by porject at this point - and only in checks .
15:40:31 <ade_lee_> many projects have a fips gate
15:40:48 <dansmith> I think the concern is more about us having to test it on centos, and that not going very well
15:40:54 <ade_lee_> and some (like cinder) are still working on getting gates in
15:41:01 <dansmith> AFAICT, the centos jobs we've tried have been too unstable to keep voting
15:41:15 <dansmith> so is there a plan to be able to test these on ubuntu at some point in the future?
15:41:18 <gmann> so it has to be on centos and adding in project check pipeline as non voting?
15:41:20 <ade_lee_> understood, centos has been trying as of late
15:41:48 <gmann> and is centos one passing and stable?
15:41:55 <ade_lee_> dansmith, I've contacted canonical about fips on ubuntu but there are issues
15:42:13 <gmann> dansmith: if it is stable then we can add but again if stable which I doubt on any centos job
15:42:21 <ade_lee_> mainly because fips on ubuntu is a premium feautre -- ie. one that requires a subscription
15:42:27 <dansmith> lol
15:42:29 <gmann> :)
15:42:35 <fungi> ubuntu doesn't provide fips support "for free" (it's a paid feature which requires obtaining and applying license keys)
15:42:43 <ade_lee_> if we figure a way around that we can switch to ubuntu
15:42:45 <dansmith> maybe that's the problem with the centos jobs? we forgot to put quarters in the slot?
15:42:58 <jungleboyj> Bwah ha ha!
15:43:13 <ade_lee_> dansmith, I've been trying to move the contos jobs to centos-9 which appears to be getting more stable
15:43:28 <jungleboyj> Based on experience it would have to be a huge stack of quarters.
15:43:28 <fungi> canonical seems like they might be willing to give us license keys, but we have no way to prevent people from accessing those in jobs
15:43:30 <dansmith> really? that's surprising, but good if there's a horizon
15:43:55 <dansmith> fungi: I would have thought the license would be for support and not "a license key to turn off a bunch of ciphers"
15:43:56 <ade_lee_> the latest problem there has been changes to openssl to disallow sha1 in signatures by default
15:44:16 <ade_lee_> and thats been causing all sorts of headaches, but we're finding them
15:44:16 <gmann> but should we try them as periodic or experimental and on few projects and if they are stable then try to add in all other projects?
15:44:17 <spotz_> Is there anything the CentOS community can do to help? I’ve been promoting working with our neighbors
15:44:40 <dansmith> spotz_: suck less? (/me continues today's theme)
15:44:45 <fungi> dansmith: right, it's the customizations they're gating behind the license key, someone could certainly roll their own fips support for any distro if they have the time and interest
15:44:55 <gmann> spotz: yeah, help in fixing the centos jobs, and for devstack support which is broken regularly and we do not have many centos people to help there
15:45:07 <dansmith> spotz_: the problems seem to come from libvirt, qemu, and all over, so it's not really a single thing to fix AFAICT
15:45:21 <clarkb> dansmith: to put it more diplomatically I would say putting mor eemphasis and reverting broken changes and/or fixing them. The whole ping sysctl perms thing was broken for like a month and was a trivial revert that could've been done instead
15:45:22 <spotz_> Ok
15:45:46 <dansmith> clarkb: yeah I was just trying to say: it's not one bug we need fixed, it seems systemic
15:45:46 <clarkb> they appear to be changing things aggressively via pulls from upstream rhel and when it breaks they don't fix centos they wait for rhel to fix
15:45:51 <clarkb> dansmith: agreed
15:46:04 <dansmith> which also makes me not really want to try to fix it and just work on being able to run it on something stable
15:46:15 <clarkb> basically when thing sgo wrong they are not addressed in a timely manner directly in centos and instead deferred to rhel if I read between the lines in the bug tracker properly
15:46:20 <clarkb> and that leads to long periods of broken in centos
15:46:39 <fungi> we've added rocky linux very recently, which is a rhel clone (currently rhel 8 as there's still no rhel 9)
15:46:42 <dansmith> clarkb: yeah and I kinda think that's the "new way" we should expect from here on out
15:46:49 <spotz_> That is something being worked on
15:46:55 <slaweq> maybe stupid question but what about fedora? Would it be better than centos maybe?
15:47:17 <gmann> not sure if that is tried for FIPs
15:47:23 <fungi> slaweq: not for long-term testing since we can't keep running it for stable branches
15:47:38 <clarkb> ya rocky seems like a good option to look at if stablitiy is the issue
15:47:46 <slaweq> k
15:47:47 <fungi> our stable support lifetime is several times that of fedora support for a given release
15:47:55 <gmann> back to FIPs jobs running: ade_lee_ do you have any rough list that how many projects run FIPs jobs in check or gate pipeline?
15:48:12 <ade_lee_> counting ...
15:48:59 <ade_lee_> keystone, barbican, glance, nova, cinder (in progress), ceilometer, tripleo , manila , neutron
15:49:16 <ade_lee_> horizon (in progress)
15:49:41 * ade_lee_ getting my spreadsheet
15:49:55 <slaweq> just to be clear: neutron has those jobs in periodic, not check nor gate pipelines
15:49:56 <gmann> octavia, designate also
15:50:15 <gmann> and in tempest experimental but that is not merged yet
15:50:23 <gmann> slaweq: +1
15:51:01 <gmann> ade_lee_: can we add them as perioidic for all those projects and collect more data on stability and ion next cycle or so where we select this goal then we start adding in each pipeline
15:51:02 <ade_lee_> I think thats it , but I may have missed one or tow
15:51:09 <johnsom> Yeah, Octavia and Designate are both on plan for FIPs jobs. Octavia even has FIPs enabled in the amphora instances today.
15:52:01 <ade_lee_> gmann, we can -- seems reasonable until centos is more stable
15:52:02 <gmann> especially when infra resource shortage coming soon..
15:52:26 <ade_lee_> how often do periodic jobs run?
15:52:37 <gmann> ade_lee_: yeah or until anyone gift  us ubuntu preimum  :)
15:52:52 <gmann> ade_lee_: you can configure, like weekly or month
15:52:59 <clarkb> currently daily
15:53:08 <dansmith> what about debian? do they have a fips knob?
15:53:10 <gmann> weekly also you can configure
15:53:21 <slaweq> in neutron periodic those jobs are pretty stable: https://zuul.openstack.org/builds?job_name=neutron-ovs-tempest-fips&job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack%2Fneutron&pipeline=periodic&skip=0 :)
15:53:47 <gmann> ade_lee_: like this https://github.com/openstack/placement/blob/master/.zuul.yaml#L64
15:53:52 <ade_lee_> ok - that seems like a reasonable compromise -- and yeah, some of the jobs are quite stable ^^ see neutron/ cilometer
15:53:56 <slaweq> of course it's just few days as we run them like that but seems pretty ok IMHO
15:54:07 <gmann> ok
15:54:10 <ade_lee_> and barbican/keystone
15:54:18 <fungi> the crypto-policies package on debian has a fips-mode-setup tool
15:54:27 <gmann> but whatever you like daily weekly or depends on projects and how stable they are
15:54:46 <ade_lee_> sure thats fair
15:55:08 <ade_lee_> is it possible to trigger a periodic job manually in a review?
15:55:08 <fungi> getting "ubuntu premium" isn't so much the problem, as i mentioned, it's securing the key when run against arbitrary code which has root access to the nodes
15:55:37 <slaweq> ade_lee_: I think You can add same job to periodic and experimental pipelines
15:55:45 <gmann> ade_lee_: for that you can add in experimental queue also
15:55:45 <fungi> ade_lee_: if you want to trigger a job manually you put it in experimental, yeah
15:55:50 <fungi> you can have the job in both pipelines
15:55:51 <gmann> slaweq: +1
15:55:52 <slaweq> and then trigger it in review with "check experimental"
15:55:59 <ade_lee_> ok cool - I can do that
15:56:05 <fungi> you can't "trigger" the periodic pipeline, that's what makes it periodic
15:56:14 <fungi> it's triggered by a timer
15:56:20 <dansmith> fungi: uh I disagree. "wait" is an action :P
15:56:32 <gmann> ok, let's convert the existing FIPs job to run periodic/experimental for all projects they have been added or going to be added
15:56:35 <jungleboyj> :-)
15:56:35 <gmann> ade_lee_: ^^ ok for you
15:56:36 <ade_lee_> suuL: advance time
15:56:54 <ade_lee_> works for me
15:57:10 <gmann> perfect, any objection from anyone on above proposal
15:57:22 <dansmith> to recap,
15:57:30 <dansmith> it's fips jobs go in experimental or periodic?
15:57:31 <ade_lee_> fyi, on the tripleo side of things, things are progressing enough, that we make fips testing the default
15:57:46 <gmann> dansmith: yes
15:57:50 <ade_lee_> as we're testing on centos 9 in any case
15:57:53 <dansmith> no objection from me :)
15:58:23 <gmann> ade_lee_: ack. and in PTG we will check the goal status and see if we can run more or so or other distro FIPs nobs
15:58:25 <ade_lee_> and we can exampine data on stability at the ptg
15:58:31 <gmann> yeah
15:58:43 <gmann> cool, thanks ade_lee_ for joining and discussion
15:58:45 <gmann> #topic Open Reviews
15:58:47 <gmann> 2 min left
15:58:48 <ade_lee_> and I'lltry sync again with canonical
15:58:49 <rosmaita> is there a way you can see all the fips jobs builds in zuul?
15:58:55 <gmann> ade_lee_: +1
15:59:07 <rosmaita> i can see this one for glance: https://zuul.openstack.org/builds?job_name=glance-multistore-cinder-import-fips&skip=0
15:59:22 <rosmaita> but if i wanted to look at all of them, is there a way to do that
15:59:28 <fungi> if they have known names, you can add them all to a query yes
15:59:31 <ade_lee_> ack -- I was wondering about that too
15:59:48 <fungi> i don't think you can regex that unfortunately
15:59:51 <fungi> or glob
16:00:04 <gmann> yeah by name, it does not work as regex https://zuul.openstack.org/builds?job_name=*fips&skip=0
16:00:10 <fungi> but if someone makes a list of them, we could make a query and link it somewhere
16:00:16 <gmann> anyway we are out of time
16:00:23 <rosmaita> fungi: ty
16:00:31 <ade_lee_> fungi, I can find the names and provide to you to help make a query?
16:00:32 <gmann> we have discussed about open reivews, please check those
16:00:50 <gmann> that is all for today meeting, anything else?
16:01:24 <gmann> ok, thanks for joining everyone
16:01:27 <gmann> #endmeeting