14:00:06 <hberaud> #startmeeting releaseteam
14:00:06 <opendevmeet> Meeting started Fri Sep 17 14:00:06 2021 UTC and is due to finish in 60 minutes.  The chair is hberaud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:06 <opendevmeet> The meeting name has been set to 'releaseteam'
14:00:11 <hberaud> Ping list: elodilles armstrong
14:00:14 <ttx> o/
14:00:16 <hberaud> #link https://etherpad.opendev.org/p/xena-relmgt-tracking
14:00:21 <hberaud> We're way down on line 349 now.
14:00:27 <hberaud> Will just wait a couple minutes for folks.
14:01:14 <yoctozepto> o/
14:01:21 <elodilles> o/
14:02:50 <hberaud> ok let's go
14:02:52 <hberaud> #topic Review task completion
14:03:41 <hberaud> The first task was "Process any remaining library branching exception". IIRC I didn't see exception
14:03:55 <hberaud> 2) On the Monday, generate release requests for all deliverables that have do not have a suitable candidate yet.
14:04:11 <hberaud> this was done with 2 series of patches
14:04:57 <hberaud> the first one was for CWI projects => https://review.opendev.org/c/openstack/releases/+/808782
14:05:14 <hberaud> sorry wrong link https://review.opendev.org/q/topic:%22xena-c-a%22+(status:open%20OR%20status:merged)
14:05:33 <hberaud> only one patch remain unmerged yet, I'll approve it now
14:06:00 <elodilles> thanks!
14:06:17 <hberaud> Thanks to you elodilles
14:06:48 <hberaud> and the second series was for RC deliverables => https://review.opendev.org/q/topic:%22xena-rc1-deadline%22
14:07:59 <hberaud> deadline is over and almost all PTLs approved their part so I think that we are good to go with the patches which remain opened
14:08:18 <hberaud> ttx, elodilles feel free to validate these ones
14:08:27 <elodilles> I'll continue to review these patches after the meeting,
14:08:38 <hberaud> +1 thanks
14:08:53 <elodilles> now that the gate is working :)
14:09:21 <hberaud> We should only notice that even if the PTL validation badge is not present those patch are almost validated by current PTL
14:09:54 <hberaud> s/are all almost/
14:10:34 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for freezer-web-ui  https://review.opendev.org/c/openstack/releases/+/808618
14:10:36 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for adjutant  https://review.opendev.org/c/openstack/releases/+/808593
14:10:38 <hberaud> And that's all for task completion
14:10:46 <ttx> will also do reviews after meeting
14:10:53 <hberaud> thanks
14:10:57 <hberaud> #topic Assign R-2 tasks
14:11:23 <hberaud> Any volunteer for "After all the projects enabled in devstack by default have been branched, we can engage with the QA, I18n and Requirements PTLs to finalize the stable branch setup" ?
14:11:25 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for freezer  https://review.opendev.org/c/openstack/releases/+/808619
14:11:28 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for murano-agent  https://review.opendev.org/c/openstack/releases/+/808674
14:11:30 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for murano-dashboard  https://review.opendev.org/c/openstack/releases/+/808677
14:11:32 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for murano  https://review.opendev.org/c/openstack/releases/+/808680
14:11:42 <hberaud> and "Ensure that all projects that are publishing release notes have the notes link included in their deliverable file."
14:12:03 <elodilles> i can help with those though i might need help
14:12:07 <hberaud> I took that one
14:12:21 <gmann> ping me once branches are ready and QA will take care of devstack/grenade things
14:12:34 <hberaud> then let me put your name on engaging QA, i18n etc...
14:12:41 <gmann> sure, +1
14:13:00 <hberaud> gmann: I mean elodilles's name :)
14:13:15 <gmann> ok :) either is ok
14:13:21 <hberaud> elodilles: this one is easy
14:13:28 <elodilles> and I'll let you know gmann when we are there :)
14:13:33 <hberaud> do not hesitate to ping me if needed
14:13:39 <gmann> elodilles: ack.
14:13:39 <elodilles> hberaud: +1
14:13:47 <hberaud> excelllent
14:14:18 <hberaud> and that's all
14:14:32 <hberaud> #topic Review countdown email contents
14:14:40 <hberaud> #link https://etherpad.opendev.org/p/relmgmt-weekly-emails#L10
14:14:53 <hberaud> let me know if it LGTY
14:16:30 <ttx> lgtm
14:16:41 <ttx> only fixed a Xena capitalization
14:17:03 <hberaud> we all use the same color :)
14:17:09 <hberaud> thanks
14:17:23 <ttx> boom
14:17:34 <hberaud> ahah
14:18:22 * yoctozepto has to go, will read later the discussion on patching the relationship with governance repo
14:18:29 <hberaud> yoctozepto: o/
14:18:46 <fungi> yoctozepto: i've followed up about it on the ml anyway, we can summarize further there
14:19:54 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for nova  https://review.opendev.org/c/openstack/releases/+/808706
14:19:58 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for senlin-dashboard  https://review.opendev.org/c/openstack/releases/+/808727
14:20:01 <opendevreview> Merged openstack/releases master: Proposing Xena RC1 for solum-dashboard  https://review.opendev.org/c/openstack/releases/+/808729
14:20:39 <elodilles> (email lgtm, too)
14:20:44 <hberaud> thanks
14:20:47 <hberaud> #topic PTG time slot
14:20:52 <hberaud> We need to schedule our TZ for the PTG, so which will fit well for you?
14:21:24 <hberaud> As it will be a 1 hour PTG for us, I was thinking reusing our meeting time, is it ok for you all?
14:21:56 <elodilles> it could work :)
14:22:10 <ttx> hmmm sure
14:22:18 <ttx> Ideally we should avoid conficting with TC sessions
14:22:24 <ttx> since that gets horizontal attention
14:22:38 <elodilles> oh and that's also usually friday
14:22:56 <ttx> how about we wait to see where they place themselves
14:23:05 <elodilles> usually relmgmt was on Tuesdays, wasn't it?
14:23:30 <ttx> i would not say that was a tradition
14:23:33 <hberaud> ttx: Ashlee and diablo_rojo_phone wait for our reply today (and of the week)
14:23:46 <hberaud> elodilles: yes it was
14:24:02 <ttx> maybe we can answer that we are flexible and want to avoid TC slots
14:24:19 <elodilles> actually I didn't give any project that we shouldn't collide in the survey :X
14:24:38 <hberaud> Concerning myself I don't have a great preference
14:24:40 <elodilles> if i remember well
14:24:57 <hberaud> elodilles: do you want to handle that point?
14:25:17 <hberaud> and reply about our flexibility as suggest by ttx
14:25:29 <clarkb> fungi: thank you for that followup email I was very confused as to why siblings wasn't doign that already
14:25:35 <clarkb> ok /me needs to do morning errands now
14:26:07 <elodilles> ok, sure, then i will propose preferably Tuesday but avoid TC
14:26:17 <hberaud> sold
14:26:26 <hberaud> thanks
14:26:54 <hberaud> #topic PTG etherpad
14:27:06 <hberaud> #link https://etherpad.opendev.org/p/relmgmt-yoga-ptg
14:27:14 <elodilles> i've just added a draft
14:27:21 <elodilles> no content yet
14:27:28 <elodilles> but feel free to add any topic
14:27:35 <hberaud> thanks I'll
14:27:53 <elodilles> maybe the "things to change" from the end of the tracking etherpad?
14:28:06 <hberaud> yes it's a starting point
14:28:30 <hberaud> Usually we did that
14:28:49 <opendevreview> Merged openstack/releases master: Release glance-tempest-plugin for Xena  https://review.opendev.org/c/openstack/releases/+/808782
14:29:12 <hberaud> #topic double check unreleased client-libraries
14:29:19 <hberaud> #link https://review.opendev.org/q/topic:%22xena-mileston-3%22
14:29:45 <hberaud> So I missed to propose these patch during R-5
14:30:12 <hberaud> These projects are quiet
14:30:29 <fungi> is that change topic intentionally misspelled?
14:30:49 <hberaud> nope
14:30:56 <hberaud> It's a nit
14:30:59 <hberaud> good catch
14:31:19 <fungi> no worries, i was trying to work out if the url was wrong or the topic was just set with a typo
14:31:20 <hberaud> I'll update it all over the patches
14:31:29 <hberaud> typo only :)
14:31:39 <fungi> pro tip: gertty can batch set change topics
14:31:58 <hberaud> Oh!?!? How to do that?
14:32:39 <fungi> query it for a list of changes with the old topic, then process mark the changes listed (% key by default) and then set the topic while you're on the change list
14:32:48 <hberaud> ah sorry I misread gertty
14:33:18 <hberaud> I see, thanks for the tips
14:34:21 <hberaud> so these deliverables don't have a lot of changes to publish but it could worth to reserve a range of version for them for stable xena
14:34:30 <hberaud> this is the reason behind this batch
14:35:20 <hberaud> I don't think that we will receive validations from PTL/liaison for these deliverables and the opened patches, so I'll force them after the meeting, is it ok for you?
14:35:37 <elodilles> these are what you discussed with ttx, that despite they don't have release content we should release to avoid confusions?
14:35:46 <hberaud> yes
14:36:06 <elodilles> ok
14:36:16 <elodilles> I can help with the review here as well
14:36:19 <hberaud> if they need to fix something at some point during Xena it will be possible
14:36:29 <hberaud> elodilles: thanks
14:36:33 <elodilles> np
14:36:39 <hberaud> much appreciated
14:37:02 <hberaud> #topic pydot2 and the governance repo
14:37:15 <hberaud> So now we are here :)
14:38:29 <fungi> updated the topic on those 11 changes to xena-milestone-3 (with an e) now
14:38:31 <hberaud> As I said in my email I would avoid to rely on pypi for our own deliverables in the requirements of openstack/release to avoid future catch-22 situations
14:38:37 <hberaud> fungi: thanks
14:39:28 <hberaud> for now we fixed the problem but fungi proposed a solution which seems to be better
14:39:45 <gmann> +1, I think we should stop releasing governance itself
14:39:58 <gmann> I will check election scripts/deps which i know use it
14:40:23 <gmann> ttx: fungi do you know if any one else, foundation site etc need governance in PyPI ?
14:40:27 <hberaud> +1
14:40:45 <fungi> the only reason we tagged the governance repo originally was to mark the point in time where the state of the projects list was used in determining contributors for the electorate, yes
14:41:04 <gmann> fungi: ok, and we can use hash for that also
14:41:09 <fungi> it could be done with an arbitrary commit id instead (and i've even done it that way for special elections)
14:41:11 <elodilles> for local use/run pypi is more convenient, isn't it?
14:41:17 <gmann> but anyways do not want to hijack this meeting on this, will discuss in TC meeting or so
14:41:37 <gmann> fungi: +1
14:43:00 <elodilles> my point is if we just run 'tox -e validate' for example, locally, then it will use the old openstack-governance 0.10.0 from pypi and similarly could fail
14:43:42 <elodilles> or do i miss something?
14:44:01 <fungi> elodilles: i suppose you could include some helper which grabs the necessary content from governance if it's not available already
14:44:04 <hberaud> Do we have any volunteer to handle that point and propose the changes? yoctozepto maybe?
14:44:35 <gmann> hberaud: what we need now?
14:45:03 <fungi> the way projects handle obtaining constraints files from the requirements repo for local tox invocation is an example
14:45:21 <hberaud> gmann: at least "declare openstack/governance in"
14:45:22 <hberaud> the required-projects lists for the jobs which are using it, and
14:45:24 <hberaud> then the tox role from zuul-jobs will know to consume it from
14:45:26 <hberaud> locally supplied source instead of fetching it from PyPI
14:45:49 <hberaud> I suppose it will require update in project-config
14:46:02 <hberaud> at least
14:46:18 <gmann> ah, I thought that is done. I can help
14:46:23 <fungi> yes, i think the key release jobs which use the files from governance are probably defined in project-config
14:46:51 <hberaud> gmann: much appreciated
14:46:54 <fungi> best to scope that addition just to the jobs which actually use the governance files though
14:47:11 <hberaud> I added this topic to our PTG agenda to ensure that we don't forget it too
14:47:24 <fungi> so that we're not unnecessarily installing openstack/governance in every single release-related job whether it's needed or not
14:47:31 <gmann> yeah
14:48:07 <fungi> i have a feeling only the reference/projects.yaml file from governance is used, right?
14:48:24 <hberaud> yes
14:48:31 <hberaud> at least
14:48:50 <fungi> so just jobs which iterate over or do lookups in that file should need governance listed as a required project for the job
14:49:29 <fungi> and then it can be removed from the (test-?)requirements.txt file
14:49:41 <hberaud> gmann: ^
14:49:55 <gmann> ack. thanks
14:50:16 <hberaud> let us know if you need help
14:51:28 <hberaud> #topic Open Discussion
14:51:47 <hberaud> Anything else to discuss today?
14:51:59 <fungi> looks like we import from governance in doc/source/_exts/deliverables.py so it might still need to be included in doc/requirements.txt
14:53:00 <hberaud> ack
14:54:10 <fungi> also anything which uses find_gerrit_acl_issues.py, get_contacts.py, list_changes.py, validate.py, deliverable.py, aclissues.py...
14:54:32 <hberaud> Yeah I was thinking about acl
14:54:43 <fungi> so it's not consuming/parsing the projects.yaml directly, it's using the public api provided by the openstack_governance python module
14:54:55 <hberaud> yep
14:55:45 <fungi> continuing to release openstack_governance would therefore make sense, so if the concern is just with tagging releases of the governance repo itself i suppose that module could be split into its own separate governance-tools repo or something
14:56:14 * yoctozepto is back
14:56:43 <yoctozepto> so, what's the consensus on governance-in-releases fix?
14:56:54 <fungi> gmann: in summary, openstack/governance currently contains a public api in the form of a python package, which is consumed by other projects. treating it as a proper library would make sense because that's how it's already consumed
14:57:03 <hberaud> for now follow fungi's proposition on the ML thread
14:57:19 <yoctozepto> I must say I'm unsure we can easily patch the tox jobs as they rely on global definitions
14:57:27 <yoctozepto> so we would switch it for all the projects
14:57:33 <yoctozepto> and also ignore local git checkouts
14:57:50 <gmann> fungi: humm, if those are used in release or election then we can ask to use directly from source  otherwise lib make sense
14:58:29 <yoctozepto> as far as I understand both, governance and releases repos only really make sense when considered from the very tip
14:58:31 <fungi> yoctozepto: which tox jobs specifically are you concerned about? you can add required-projects similar to how we do for, e.g., tox jobs run by horizon plugins which need horizon installed from source
14:58:57 <yoctozepto> fungi: all jobs were tox and all were failing, so e.g. pep8, py36, py38, cover, whatever
14:59:08 <yoctozepto> we would need to inherit all
14:59:15 <yoctozepto> and control which python vers to test
14:59:27 <gmann> in project-config, I only find this job using the releases/tools/check_approval.py (which is governance/project.yaml) and it has governance in required_projects - /  https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L1078
14:59:28 <yoctozepto> which might not be *that* bad considering the repo's purpose
15:00:22 <yoctozepto> mayhaps it's just an overkill that it's included in requirements.txt and should be moved to tox's definition - for endusers rely on git path and zuul could substitute its own one? wdyt?
15:01:08 <hberaud> gmann: this one was the only one who works previously, I think because "required-project" is used
15:01:23 <fungi> gmann: the problem comes from things which call modules in the openstack_releases package or scripts in openstack/releases tools directory, since they import from the openstack_governance package which must be installed
15:01:39 <yoctozepto> I see many files use openstack_governance
15:01:43 <gmann> yeah
15:02:34 <fungi> so openstack_releases has numerous places where openstack_governance is treated as a python library and needs it installed via pip from somewhere
15:02:44 <yoctozepto> mhm, precisely
15:02:55 <hberaud> agreed, the problem was that we use governance as an API ans so we need to install and pull its dependencies and this is where we faced the pydot2 problem
15:02:56 <yoctozepto> and we want that to be direct from git
15:03:10 <yoctozepto> the current solution is only incapable to do depends-on
15:03:24 <yoctozepto> and slightly less efficient
15:03:35 <yoctozepto> than with required-projects
15:03:54 <fungi> yoctozepto: well, also it results in running git clone from opendev every time the project is installed, that's inefficient and fragile, so will lead to a lot of new random job failures
15:04:18 <yoctozepto> fungi: yeah, that was included in "slightly less efficient"
15:04:29 <yoctozepto> I do wonder how bad that would be but yeah
15:04:36 <yoctozepto> for releases we should strive for best stability
15:04:44 <yoctozepto> so perhaps really control all the jobs locally?
15:04:49 <yoctozepto> releases-tox etc.
15:04:55 <hberaud> fungi: however I'm not sure to see how splitting the python module part of the governance repo will protect us from similar situation?
15:04:58 <yoctozepto> releases don't have to gate on the same python versions tbh
15:05:02 <fungi> unfortunately, even a modest increase in random failures for release jobs means me or another opendev admin getting involved to reenqueue failures because many of these jobs aren't idempotent
15:05:17 <yoctozepto> fungi: yeah, I feel you totally
15:05:35 <gmann> hberaud: yeah not sure that will solve these kind of issues
15:05:43 <yoctozepto> ok, so how about this
15:05:51 <fungi> hberaud: splitting the python module part of the governance repo to a separate repo means that the tc can stop worring about releasing the governance repo and can only have to release the governance-tools repo
15:06:03 <fungi> separating the software from the data
15:06:07 <yoctozepto> 1) releases control their jobs by themselves
15:06:25 <yoctozepto> 2) they add required-projects to the base tox job
15:06:43 <gmann> fungi: yeah but those tools still face same issue as currently we are facing, its tool broke us not data
15:06:54 <yoctozepto> 3) governance also adds gating on a relevant test job
15:06:55 <hberaud> yes
15:07:06 <hberaud> the problem come from the env
15:07:13 <yoctozepto> then we have more control AND more reliability
15:07:21 <fungi> hberaud: gmann: i suggested splitting the library out of the governance repo as an alternative so that people can still continue to consume it from pypi for running locally
15:07:21 <gmann> yoctozepto: on 2nd, base tox job you mean install gov for everyone ?
15:07:36 <yoctozepto> gmann: no, I mean custom base tox job for releases
15:07:41 <gmann> +1
15:08:11 <fungi> how do horizon plugins do things like pep8 since they need horizon installed from source?
15:08:15 <gmann> yoctozepto: and thing is if governance ship any tool, it should test in their gate which is missing key here
15:08:36 <hberaud> yoctozepto: seems a good idea
15:08:39 <gmann> and if no one use those tool like universal* then just delete
15:08:40 <yoctozepto> gmann: we test it; we had our gate broken
15:08:43 <yoctozepto> fungi: good question
15:08:55 <yoctozepto> fungi: I bet they just use release lol
15:09:08 <yoctozepto> but let me check something sensible
15:11:23 <yoctozepto> yeah, they seem to be installing latest release
15:11:29 <yoctozepto> which is clumsy
15:11:41 <yoctozepto> (to avoid saying it's utterly wrong in general)
15:11:58 <yoctozepto> but it seems we just lost track of the needs of tox jobs
15:12:16 <yoctozepto> I am unsure if we want to try to develop a generic approach
15:12:38 <fungi> clarkb: ^ do you know of a general approach to adding required-projects for every single job run for a particular project?
15:13:06 <yoctozepto> anyhow, releases have an outdated set of template jobs: "openstack-python3-victoria-jobs"
15:13:09 <fungi> the idea is that any time openstack_releases is installed, openstack_governance needs to be installed but from source not from a release on pypi
15:13:18 <yoctozepto> thus it makes sense to change the process
15:14:10 <fungi> it looks like we would probably have to use child jobs of all the generic ones and then add required-projects to each one that way
15:14:11 <yoctozepto> meh, and the jobs are overridden in the pipeline already lol
15:14:23 <yoctozepto> so it can just be added to these overrides
15:14:32 <yoctozepto> there is a lot of duplication, ok
15:15:42 <fungi> it's possible we can set required-projects on variants instead of using job inheritance
15:15:47 <fungi> yeah
15:16:05 <gmann> how many jobs we have ? may be let's go with like check-release-approval
15:16:10 <gmann> simple way
15:16:20 <fungi> it just means the project-templates are no longer useful
15:17:32 <hberaud> gmann's proposition could be a first step
15:17:53 <gmann> and continue releasing governance for its public APIs for local use or so
15:18:22 <hberaud> nothing prevents us from improving it behind
15:19:54 <hberaud> WFY?
15:20:23 <yoctozepto> loving that keystone also stayed on testing victoria https://opendev.org/openstack/keystone/src/branch/master/.zuul.yaml
15:20:23 <yoctozepto> meh
15:20:33 <hberaud> I'll have to grab kids at school
15:20:45 <yoctozepto> hberaud: make sure to take only yours
15:21:00 <hberaud> will try :)
15:21:02 <yoctozepto> otherwise they may report a KIDnapping
15:21:16 <gmann> :)
15:21:40 <yoctozepto> I'll propose something and let's see how it's going to work
15:22:01 <hberaud> +1
15:22:03 <hberaud> We could continue the discussion on the thread
15:22:13 <hberaud> I'll close the meeting
15:22:20 <yoctozepto> or just in this channel ;-)
15:22:32 <hberaud> As you want :)
15:23:39 <hberaud> Thank you everyone for joining. I added a record of this discussion to our PTG etherpad
15:24:04 <hberaud> #closemeeting
15:24:19 <hberaud> #endmeeting