14:00:01 <marios|ruck> #startmeeting tripleo
14:00:01 <opendevmeet> Meeting started Tue Aug  3 14:00:01 2021 UTC and is due to finish in 60 minutes.  The chair is marios|ruck. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:01 <opendevmeet> The meeting name has been set to 'tripleo'
14:00:15 <marios|ruck> #topic agenda
14:00:15 <marios|ruck> * Review last minutes & action items
14:00:15 <marios|ruck> * One off agenda items
14:00:15 <marios|ruck> * Bugs & Blueprints
14:00:15 <marios|ruck> * Projects releases or stable backports
14:00:18 <marios|ruck> * Specs
14:00:20 <marios|ruck> * open discussion
14:00:23 <marios|ruck> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:00:25 <marios|ruck> Hello folks, who is around today? o/ good morning good afternoon good evening
14:00:43 <bshephar> \o hey marios|ruck
14:00:43 <fultonj> o/
14:00:46 <owalsh> o/
14:00:52 <slagle> hello
14:01:00 <tkajinam> o/
14:01:18 <spotz> o/
14:01:27 <marios|ruck> k lets get started others can catch up as usual o/ hello all
14:01:32 <weshay|ruck> 0/
14:01:35 <marios|ruck> #topic review last meeting logs & action items
14:01:37 <eliadcohen> o/
14:01:45 <marios|ruck> #link https://meetings.opendev.org/meetings/tripleo/2021/tripleo.2021-07-20-14.00.html
14:01:54 <marios|ruck> anything to be mentioned from the last meeting? /me checks
14:02:00 <marios|ruck> was a bit quiet last one
14:02:06 <fmount> o/
14:02:27 <marios|ruck> k moving on in a sec if nothing here
14:02:43 <rlandy|ruck> o/
14:02:51 <marios|ruck> #topic one off agenda items
14:02:52 <marios|ruck> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:03:07 <marios|ruck> taking them in order as usual - feel free to add your stuff ^^
14:03:16 <marios|ruck> #info  * [TripleO] Yoga PTG timeslots vote http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023750.html       * 1300-1700 UTC       * Mon 18th October + Tuesday+ Wednesday and Thursday 21st.
14:03:41 <marios|ruck> booked the 13-1700 timeslot for the meetup
14:03:45 <marios|ruck> same one we used other times
14:03:54 <marios|ruck> there was no alternative preferred from the doodle poll
14:04:00 <marios|ruck> any comments ?
14:04:37 <marios|ruck> gonne leave the independent one last skipping will revisit cos it may have discussion
14:04:43 <marios|ruck> #info * Releases: victoria https://review.opendev.org/c/openstack/releases/+/801353           ussuri https://review.opendev.org/c/openstack/releases/+/802043
14:04:50 <marios|ruck> #info stable/stein now EOL  https://review.opendev.org/c/openstack/releases/+/798925
14:05:04 <marios|ruck> some releases made and stein is also (with queens) now EOL no more patches here please they will be dropped
14:05:07 <marios|ruck> cannot merge
14:05:14 <marios|ruck> any comments on these ones?
14:05:28 <weshay|ruck> nope
14:05:33 <marios|ruck> ;) thanks weshay|ruck
14:05:35 <weshay|ruck> :)
14:05:38 <marios|ruck> #info * [tripleo] Changing TripleO's release model http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023809.html spec https://review.opendev.org/c/openstack/tripleo-specs/+/801512
14:05:40 <odyssey4me> o/
14:05:42 <marios|ruck> k this one
14:05:50 <marios|ruck> so we have brought up this subject many times over the last ~2 months ish
14:05:53 <marios|ruck> latest is we now have a spec
14:06:04 <marios|ruck> so we can get specific and try to converge on a decision
14:06:08 <marios|ruck> wether to proceed or not
14:06:11 * Tengu adds it to his pile of reviews
14:06:14 <marios|ruck> thanks to folks that have commented already on that
14:06:25 <Tengu> thanks for pushing that spec up, marios|ruck !
14:06:31 <marios|ruck> any comments on this item on the proposal anyone has no idea what we're talking about?
14:06:44 <marios|ruck> Tengu: :)
14:06:52 <cloudnull> o/
14:07:09 <marios|ruck> my main concern is as outlined in the spec, we're effectively going to say that 'tripleo no longer supports all openstack releases'
14:07:15 <marios|ruck> i don't know if that is an ok thing to say
14:07:22 <opendevreview> Jiri Podivin proposed openstack/validations-common master: DNM  https://review.opendev.org/c/openstack/validations-common/+/803316
14:07:31 <marios|ruck> if there are no comments here then consider commenting on the spec.
14:07:43 <marios|ruck> at the moment there is no big -1 or -2 on that so it is leaning more towards happening than not
14:07:47 <cloudnull> +1 I think that 's ok to call out.
14:07:48 <marios|ruck> if you disagree SPEAK UP :D
14:07:50 <marios|ruck> weshay|ruck: ^^^
14:07:56 <Tengu> dropping support for old things is more than probably OK... at some point, ppl should upgrade -.-. supporting pike is a non-sense for instance.
14:08:08 <Tengu> DROP'EM'ALL!
14:08:09 <marios|ruck> Tengu: well, not just about old things
14:08:13 <cloudnull> support == long term support on branches.
14:08:17 <marios|ruck> Tengu: we may choose not to create xena
14:08:28 <Tengu> w00t
14:08:31 <marios|ruck> Tengu: so as soon as xena is created as a new thing in all the projects, we'll say 'no we don't do that'
14:08:42 <marios|ruck> Tengu: ther is some suggested mitigation in the spec, about using tags
14:08:47 <marios|ruck> but it isn't ideal
14:09:03 <Tengu> I'll check it out
14:09:11 <weshay|ruck> ya.. I think it's a choice between what is critical vs. important.. and focusing on what's critical will be better for the community
14:09:11 <cloudnull> re: we may choose not to create xena +1 I think that's OK
14:09:14 <weshay|ruck> imho
14:09:43 <marios|ruck> k any more comments here or we will move on in a sec
14:09:51 <marios|ruck> please checkout spec https://review.opendev.org/c/openstack/tripleo-specs/+/801512
14:09:55 <marios|ruck> comment ther ^
14:10:17 <marios|ruck> Tengu: next item is yours please go ahead?
14:10:24 <Tengu> yep, thanks
14:10:48 <Tengu> So, following a spec dropping every single container healthchecks due to the issues we hit, and the podman way of doing them...
14:11:09 <Tengu> I've pushed a new spec, about "how to implement healthchecks without podman", with actual, real healthcheck endpoints for the APIS
14:11:15 <Tengu> #link https://review.opendev.org/c/openstack/tripleo-specs/+/803152
14:11:36 <Tengu> so it's still on WIP, but I'd happily get some feedback, especially since this is a cross-team effort
14:12:02 <Tengu> namely: teams developping an API or webservice would need to provide a /status, /healthcheck or related if they want monitoring
14:12:15 <Tengu> end of message :)
14:12:25 <marios|ruck> Tengu: thanks for proposing that checking if i have already seen it cant remember
14:12:33 <marios|ruck> ah is new new ;)
14:12:35 <Tengu> ah, and it's a collaboration with the CloudOps folks.
14:12:36 <marios|ruck> adding to list
14:12:44 <Tengu> :)
14:13:00 <marios|ruck> any comments/questions ?
14:13:07 <bshephar> Tengu: How about for things that aren't API endpoints though? nova_compute being an example
14:13:28 <Tengu> bshephar: for that kind of "non-api service", they are actually consumed by an API
14:13:36 <Tengu> so those API end-point should ensure they are able to communicate properly
14:13:57 <Tengu> that's the case for DB, message queueing (rabbitmq for instance) and so on
14:14:23 <Tengu> as said, that spec is still a WIP, and I'd love to get comments/ideas/pointers.
14:14:39 <Tengu> bshephar: for instance, your comment would be really nice on the spec to start a discussion :)
14:14:45 <bshephar> I'll spend some more time reading the spec and add additional comments there. :)
14:14:47 <Tengu> (and engage Compute ppl)
14:15:05 <Tengu> as said: cross-team effort - DF can't handle all the things ;)
14:15:18 <slagle> yea honestly, i think we need to describe what is needed and let the nova team figure out how to implement it
14:15:26 <Tengu> exactly
14:15:34 <opendevreview> Merged openstack/tripleo-heat-templates master: Change default for NovaNfsVersion to NFSv4.2  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/801428
14:15:50 <Tengu> get a canvas, and let ppl act within it
14:16:06 <bshephar> Yeah, I mean, if the Nova API is up, you can verify nova_compute is working using the nova API. Maybe just cover the basics and the rest will figure itself out
14:16:11 <opendevreview> Jiri Podivin proposed openstack/python-tripleoclient master: Replacing MD5 use to comply with FIPS  https://review.opendev.org/c/openstack/python-tripleoclient/+/801460
14:16:22 <Tengu> :)
14:16:44 <Tengu> we have here an opportunity to make an actual monitoring.
14:17:01 <Tengu> until now, the healthchecks are... well. we NEVER could trust them enough to act on the services based on the failures.
14:17:04 <slagle> well i still think we need a way to ask each individual container if it's doing what it's "supposed to do"
14:17:29 <slagle> we need to be able to check nova-compute, without asking the api
14:17:31 <Tengu> slagle: I'd rather avoid using the podman native healthchecks, and anything "podman exec" related, frankly.
14:17:42 <Tengu> systemd might provide info though
14:17:45 <marios|ruck> Tengu: did you end up removing the 'old' healthchecks?
14:17:49 <odyssey4me> would it not be possible to just have the /healthceck access point work for those that have a proper one, and not there for those that don't?
14:17:49 <Tengu> as in "systemd status ...."
14:17:53 <marios|ruck> Tengu: or not fully or did we change direction on that?
14:17:55 <opendevreview> Sorin Sbârnea proposed openstack/tripleo-repos master: Add ansible-test sanity checks job (nv)  https://review.opendev.org/c/openstack/tripleo-repos/+/803353
14:18:07 <marios|ruck> Tengu: hmm guess not i see them in logs from today jobs
14:18:09 <Tengu> marios|ruck: it's on hold right now, in order to get that spec done. We can't really stay without any replacement
14:18:15 <opendevreview> Francesco Pantano proposed openstack/tripleo-heat-templates master: DNM - Test Pending Ceph Pacific daemons in CI  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/778915
14:18:28 <marios|ruck> Tengu: i see so removal depends on this
14:18:41 <Tengu> marios|ruck: had to sync with CloudOps, they missed the first spec. So now I'm really closely collaborating with them
14:18:55 <marios|ruck> Tengu: cos they rely on those i guess ack
14:19:02 <Tengu> apparently, some ppl are actually relying on the healthchecks (that's carzy ;))
14:19:06 <marios|ruck> haha
14:19:24 <marios|ruck> k any other comments will move on in a sec please review the spec https://review.opendev.org/c/openstack/tripleo-specs/+/803152
14:19:27 <bshephar> Yeah, +1 on avoiding podman specifc implementation there
14:20:26 <marios|ruck> next one item from tripleo-ci
14:20:36 <marios|ruck> #info     https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 wire up tripleo_get_hash ansible module in repo-setup role (paused/behind major refactor at https://review.opendev.org/c/openstack/tripleo-repos/+/801029 and https://review.opendev.org/c/openstack/tripleo-repos/+/801621
14:21:00 <marios|ruck> this was discussed at ptg we had a session on tripleo-repos .. we now have two independent sub modules there for get hash and yum repos
14:21:08 <marios|ruck> some pointers there if you are interested in following that work
14:21:18 <marios|ruck> any comments or questions about that?
14:22:03 <eliadcohen> no comments marios|ruck
14:22:06 <marios|ruck> #info     reminder https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-08-03-Community-Call every tuesday community call 1330 UTC
14:22:10 <marios|ruck> thanks eliadcohen :D
14:22:16 <marios|ruck> just a reminder about the ci community call
14:22:22 <marios|ruck> every tuesday 1330 UTC
14:22:35 <marios|ruck> ping us on #tripleo or #oooq if you want to join we can add you to the google meet
14:22:40 <zbr> cloudnull: let me know when you have few minutes to chat about sanity checks.
14:22:49 <marios|ruck> zbr: we are in the tripleo meet ;)
14:22:53 <marios|ruck> moving on
14:23:00 <marios|ruck> storage folks o/ please go ahead fultonj
14:23:02 <cloudnull> happy to me at the top of the hour
14:23:16 <fultonj> fmount: care to comment about promotion?
14:23:21 <cloudnull> to **meet -cc zbr
14:23:24 <fmount> sure thanks fultonj
14:23:39 <fmount> we are working on the promotion of -pacific v16.2.5
14:23:56 <fmount> but we had an issue w/ ceph-nfs because of the Ceph container build
14:24:16 <fmount> and we're looking for a new rebuild of the ceph daemon image before moving forward w/ it
14:24:28 <fmount> the new bits allow us to re-enable the ceph dashboard
14:24:32 <fmount> in the cephadm context
14:24:35 <opendevreview> Martin Schuppert proposed openstack/tripleo-heat-templates stable/wallaby: Change default for NovaNfsVersion to NFSv4.2  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/803285
14:24:39 <fmount> and finalize the migration to cephadm
14:24:48 <fmount> #link https://review.opendev.org/c/openstack/tripleo-common/+/797226
14:24:58 <fmount> the review to make the promotion when it's ready ^^^
14:25:15 <fmount> fultonj: would you cover the deployed ceph part?
14:25:23 <fultonj> the deployed_ceph feature basically working
14:25:25 <fultonj> #link https://review.opendev.org/q/topic:%22deployed_ceph%22+(status:open%20OR%20status:merged)
14:25:42 <fultonj> it has 2 python unit tests ATM and I'll be adding a few more
14:25:46 <fultonj> i'll add molecule test too
14:26:04 <fultonj> adding it to scenario001 will be trickier
14:26:11 <fultonj> i'd try a POC of that in a separate review
14:26:24 <fultonj> comments on the testing for that?
14:26:35 <fultonj> it's a new cli option to deploy ceph before overcloud
14:26:43 <fmount> yeah not sure, we should make some work in quickstart I guess
14:26:46 <fmount> to add ceph first
14:26:51 <fmount> before deploying the overcloud
14:27:38 <marios|ruck> fmount: maybe you can join one of the ci community calls when you're ready to discuss wiring into scenario1 if you have questions/doubts
14:27:43 <marios|ruck> and fultonj ^
14:27:50 <fultonj> ack
14:27:51 <fmount> +1 thanks marios|ruck
14:27:56 <fmount> it's a good idea
14:28:02 <fultonj> feel free to review deployed_ceph then
14:28:02 <marios|ruck> is that in tripleo-ansible? i mean the deployed ceph is it a role?
14:28:21 <bshephar> fultonj: What does the updates and upgrade situation look like for that? Is it an independent update process for Ceph?
14:28:43 <fultonj> a patch to python tripleo client calls a new playbook which calls the existing roles
14:28:46 <marios|ruck> good question bshephar, or going from 'old' ceph deployment to the new one
14:28:46 <fultonj> marios|ruck: ^
14:28:55 <fmount> bshephar: from pacific yes, it probably will be a separated process
14:29:02 <fmount> since it's managed by cephadm
14:29:04 <fmount> and it's async
14:29:14 <fultonj> deployed_ceph is orthogonal to upgrades
14:29:14 <fmount> compared to the traditional ansible run
14:29:28 <bshephar> marios|ruck: I was thinking just about new deployments. But yes, i guess coming from existing overcloud with ceph to this model is a good point.
14:29:36 <fultonj> upgrades are the same regardless
14:29:42 <fmount> right, this is valid for both deployed ceph or regular overcloud deployments w/ ceph
14:29:44 <fultonj> and the upgrade process moves you to cephadm management
14:30:05 <bshephar> Nice, this is great.
14:30:31 <marios|ruck> any other discussion here? moving on in a sec
14:30:55 <fultonj> nothing more to add from ceph side, thanks
14:31:06 <marios|ruck> fultonj: fmount: i have a different ceph related question but will leave to open discussion :)
14:31:15 <fultonj> ok, sure
14:31:33 <marios|ruck> k thanks Tengu fmount fultonj for bringing your topics today moving on i don't see anything else in the etherpad
14:31:39 <Tengu> np :)
14:31:49 <marios|ruck> #topic Bugs and blueprints
14:31:49 <marios|ruck> #link https://bugs.launchpad.net/tripleo/
14:31:49 <marios|ruck> #link https://storyboard.openstack.org/#!/project/openstack/tripleo-ansible
14:31:49 <marios|ruck> #link  https://launchpad.net/tripleo/+milestone/xena-2
14:31:50 <marios|ruck> #link https://launchpad.net/tripleo/xena
14:31:58 <marios|ruck> we have had a couple of gate blockers this last 2 weeks
14:31:59 <opendevreview> Kevin Carter proposed openstack/tripleo-common master: [FIPS] Remove paramiko from tripleo-common  https://review.opendev.org/c/openstack/tripleo-common/+/803354
14:32:35 <marios|ruck> https://bugs.launchpad.net/tripleo/+bug/1937333 https://bugs.launchpad.net/tripleo/+bug/1938079
14:32:41 <marios|ruck> i think there was one more but don't have it handy
14:32:51 <marios|ruck> last couple of days we are a bit more stable
14:32:53 <marios|ruck> promotions are good
14:33:00 <marios|ruck> any bug related business from anyone?
14:33:26 <marios|ruck> moving on ...
14:33:32 <marios|ruck> #topic Project releases or stable backports
14:33:34 <marios|ruck> #info tripleo wallaby repos https://releases.openstack.org/teams/tripleo.html#wallaby
14:33:40 <marios|ruck> covered releases in the one off items
14:33:48 <marios|ruck> any other release requests ?
14:34:04 <marios|ruck> ping me if you need something, otherwise i'll start another round through the branches in order in a week or so
14:34:25 <marios|ruck> #topic specs
14:34:25 <marios|ruck> #info https://review.opendev.org/q/project:openstack/tripleo-specs
14:34:25 <marios|ruck> #info https://opendev.org/openstack/tripleo-specs/src/branch/master/specs/xena
14:34:36 <marios|ruck> couple of specs to call out
14:35:04 <marios|ruck> https://review.opendev.org/c/openstack/tripleo-specs/+/801630 TripleO.Next - Task-Core + Directord
14:35:21 <marios|ruck> https://review.opendev.org/c/openstack/tripleo-specs/+/797676 TripleO.Next - Container Pods Spec
14:35:42 <marios|ruck> we've mentioned the healthchecks spec and the 'moving repos to independent release' earlier
14:36:01 <marios|ruck> any comments or questions?
14:36:21 <marios|ruck> #topic open discussion
14:36:23 <marios|ruck> Anything else that folks want to bring up to the meeting?
14:36:36 <marios|ruck> fultonj: fmount: o/ what can we do about https://bugs.launchpad.net/tripleo/+bug/1934879/comments/15
14:36:42 <marios|ruck> fmount: fultonj: can we move to octopus for Train?
14:36:56 <fmount> marios|ruck: I don't think so
14:37:02 <fmount> no octopus for train
14:37:16 <marios|ruck> :/
14:37:18 <marios|ruck> k
14:37:24 <fmount> and the commit that fixes this issue was backported to nautilus
14:37:29 <fmount> in July
14:37:37 <fmount> but we don't have any new release
14:37:40 <fmount> atm
14:37:52 <marios|ruck> ah i see so its not they didn't accept the patch it was no new build made
14:38:03 <fmount> new tag (upstream) means we can do a new build in cbs and update our containers w/ the ceph bits
14:38:14 <fmount> yeah
14:38:19 <fmount> there's no build
14:38:38 <marios|ruck> k thanks just was on my mind since i updated the bug earlier
14:38:40 <fmount> if I see a new build I can proactively test and revert the skiplist change
14:38:52 <opendevreview> chandan kumar proposed openstack/tripleo-quickstart-extras master: Move build-test-packages as a part of undercloud setup  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/790926
14:38:52 <fmount> but until that moment, we can just skip it
14:39:04 <fmount> marios|ruck: sure np, thanks for asking
14:39:21 <marios|ruck> any other tripleo business please  ?
14:39:52 <marios|ruck> #info next meeting tuesday 17th August
14:40:12 <marios|ruck> thanks all o/
14:40:20 <marios|ruck> #endmeeting tripleo