14:00:01 <marios|ruck> #startmeeting tripleo 14:00:01 <opendevmeet> Meeting started Tue Aug 3 14:00:01 2021 UTC and is due to finish in 60 minutes. The chair is marios|ruck. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:01 <opendevmeet> The meeting name has been set to 'tripleo' 14:00:15 <marios|ruck> #topic agenda 14:00:15 <marios|ruck> * Review last minutes & action items 14:00:15 <marios|ruck> * One off agenda items 14:00:15 <marios|ruck> * Bugs & Blueprints 14:00:15 <marios|ruck> * Projects releases or stable backports 14:00:18 <marios|ruck> * Specs 14:00:20 <marios|ruck> * open discussion 14:00:23 <marios|ruck> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:25 <marios|ruck> Hello folks, who is around today? o/ good morning good afternoon good evening 14:00:43 <bshephar> \o hey marios|ruck 14:00:43 <fultonj> o/ 14:00:46 <owalsh> o/ 14:00:52 <slagle> hello 14:01:00 <tkajinam> o/ 14:01:18 <spotz> o/ 14:01:27 <marios|ruck> k lets get started others can catch up as usual o/ hello all 14:01:32 <weshay|ruck> 0/ 14:01:35 <marios|ruck> #topic review last meeting logs & action items 14:01:37 <eliadcohen> o/ 14:01:45 <marios|ruck> #link https://meetings.opendev.org/meetings/tripleo/2021/tripleo.2021-07-20-14.00.html 14:01:54 <marios|ruck> anything to be mentioned from the last meeting? /me checks 14:02:00 <marios|ruck> was a bit quiet last one 14:02:06 <fmount> o/ 14:02:27 <marios|ruck> k moving on in a sec if nothing here 14:02:43 <rlandy|ruck> o/ 14:02:51 <marios|ruck> #topic one off agenda items 14:02:52 <marios|ruck> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:03:07 <marios|ruck> taking them in order as usual - feel free to add your stuff ^^ 14:03:16 <marios|ruck> #info * [TripleO] Yoga PTG timeslots vote http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023750.html * 1300-1700 UTC * Mon 18th October + Tuesday+ Wednesday and Thursday 21st. 14:03:41 <marios|ruck> booked the 13-1700 timeslot for the meetup 14:03:45 <marios|ruck> same one we used other times 14:03:54 <marios|ruck> there was no alternative preferred from the doodle poll 14:04:00 <marios|ruck> any comments ? 14:04:37 <marios|ruck> gonne leave the independent one last skipping will revisit cos it may have discussion 14:04:43 <marios|ruck> #info * Releases: victoria https://review.opendev.org/c/openstack/releases/+/801353 ussuri https://review.opendev.org/c/openstack/releases/+/802043 14:04:50 <marios|ruck> #info stable/stein now EOL https://review.opendev.org/c/openstack/releases/+/798925 14:05:04 <marios|ruck> some releases made and stein is also (with queens) now EOL no more patches here please they will be dropped 14:05:07 <marios|ruck> cannot merge 14:05:14 <marios|ruck> any comments on these ones? 14:05:28 <weshay|ruck> nope 14:05:33 <marios|ruck> ;) thanks weshay|ruck 14:05:35 <weshay|ruck> :) 14:05:38 <marios|ruck> #info * [tripleo] Changing TripleO's release model http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023809.html spec https://review.opendev.org/c/openstack/tripleo-specs/+/801512 14:05:40 <odyssey4me> o/ 14:05:42 <marios|ruck> k this one 14:05:50 <marios|ruck> so we have brought up this subject many times over the last ~2 months ish 14:05:53 <marios|ruck> latest is we now have a spec 14:06:04 <marios|ruck> so we can get specific and try to converge on a decision 14:06:08 <marios|ruck> wether to proceed or not 14:06:11 * Tengu adds it to his pile of reviews 14:06:14 <marios|ruck> thanks to folks that have commented already on that 14:06:25 <Tengu> thanks for pushing that spec up, marios|ruck ! 14:06:31 <marios|ruck> any comments on this item on the proposal anyone has no idea what we're talking about? 14:06:44 <marios|ruck> Tengu: :) 14:06:52 <cloudnull> o/ 14:07:09 <marios|ruck> my main concern is as outlined in the spec, we're effectively going to say that 'tripleo no longer supports all openstack releases' 14:07:15 <marios|ruck> i don't know if that is an ok thing to say 14:07:22 <opendevreview> Jiri Podivin proposed openstack/validations-common master: DNM https://review.opendev.org/c/openstack/validations-common/+/803316 14:07:31 <marios|ruck> if there are no comments here then consider commenting on the spec. 14:07:43 <marios|ruck> at the moment there is no big -1 or -2 on that so it is leaning more towards happening than not 14:07:47 <cloudnull> +1 I think that 's ok to call out. 14:07:48 <marios|ruck> if you disagree SPEAK UP :D 14:07:50 <marios|ruck> weshay|ruck: ^^^ 14:07:56 <Tengu> dropping support for old things is more than probably OK... at some point, ppl should upgrade -.-. supporting pike is a non-sense for instance. 14:08:08 <Tengu> DROP'EM'ALL! 14:08:09 <marios|ruck> Tengu: well, not just about old things 14:08:13 <cloudnull> support == long term support on branches. 14:08:17 <marios|ruck> Tengu: we may choose not to create xena 14:08:28 <Tengu> w00t 14:08:31 <marios|ruck> Tengu: so as soon as xena is created as a new thing in all the projects, we'll say 'no we don't do that' 14:08:42 <marios|ruck> Tengu: ther is some suggested mitigation in the spec, about using tags 14:08:47 <marios|ruck> but it isn't ideal 14:09:03 <Tengu> I'll check it out 14:09:11 <weshay|ruck> ya.. I think it's a choice between what is critical vs. important.. and focusing on what's critical will be better for the community 14:09:11 <cloudnull> re: we may choose not to create xena +1 I think that's OK 14:09:14 <weshay|ruck> imho 14:09:43 <marios|ruck> k any more comments here or we will move on in a sec 14:09:51 <marios|ruck> please checkout spec https://review.opendev.org/c/openstack/tripleo-specs/+/801512 14:09:55 <marios|ruck> comment ther ^ 14:10:17 <marios|ruck> Tengu: next item is yours please go ahead? 14:10:24 <Tengu> yep, thanks 14:10:48 <Tengu> So, following a spec dropping every single container healthchecks due to the issues we hit, and the podman way of doing them... 14:11:09 <Tengu> I've pushed a new spec, about "how to implement healthchecks without podman", with actual, real healthcheck endpoints for the APIS 14:11:15 <Tengu> #link https://review.opendev.org/c/openstack/tripleo-specs/+/803152 14:11:36 <Tengu> so it's still on WIP, but I'd happily get some feedback, especially since this is a cross-team effort 14:12:02 <Tengu> namely: teams developping an API or webservice would need to provide a /status, /healthcheck or related if they want monitoring 14:12:15 <Tengu> end of message :) 14:12:25 <marios|ruck> Tengu: thanks for proposing that checking if i have already seen it cant remember 14:12:33 <marios|ruck> ah is new new ;) 14:12:35 <Tengu> ah, and it's a collaboration with the CloudOps folks. 14:12:36 <marios|ruck> adding to list 14:12:44 <Tengu> :) 14:13:00 <marios|ruck> any comments/questions ? 14:13:07 <bshephar> Tengu: How about for things that aren't API endpoints though? nova_compute being an example 14:13:28 <Tengu> bshephar: for that kind of "non-api service", they are actually consumed by an API 14:13:36 <Tengu> so those API end-point should ensure they are able to communicate properly 14:13:57 <Tengu> that's the case for DB, message queueing (rabbitmq for instance) and so on 14:14:23 <Tengu> as said, that spec is still a WIP, and I'd love to get comments/ideas/pointers. 14:14:39 <Tengu> bshephar: for instance, your comment would be really nice on the spec to start a discussion :) 14:14:45 <bshephar> I'll spend some more time reading the spec and add additional comments there. :) 14:14:47 <Tengu> (and engage Compute ppl) 14:15:05 <Tengu> as said: cross-team effort - DF can't handle all the things ;) 14:15:18 <slagle> yea honestly, i think we need to describe what is needed and let the nova team figure out how to implement it 14:15:26 <Tengu> exactly 14:15:34 <opendevreview> Merged openstack/tripleo-heat-templates master: Change default for NovaNfsVersion to NFSv4.2 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/801428 14:15:50 <Tengu> get a canvas, and let ppl act within it 14:16:06 <bshephar> Yeah, I mean, if the Nova API is up, you can verify nova_compute is working using the nova API. Maybe just cover the basics and the rest will figure itself out 14:16:11 <opendevreview> Jiri Podivin proposed openstack/python-tripleoclient master: Replacing MD5 use to comply with FIPS https://review.opendev.org/c/openstack/python-tripleoclient/+/801460 14:16:22 <Tengu> :) 14:16:44 <Tengu> we have here an opportunity to make an actual monitoring. 14:17:01 <Tengu> until now, the healthchecks are... well. we NEVER could trust them enough to act on the services based on the failures. 14:17:04 <slagle> well i still think we need a way to ask each individual container if it's doing what it's "supposed to do" 14:17:29 <slagle> we need to be able to check nova-compute, without asking the api 14:17:31 <Tengu> slagle: I'd rather avoid using the podman native healthchecks, and anything "podman exec" related, frankly. 14:17:42 <Tengu> systemd might provide info though 14:17:45 <marios|ruck> Tengu: did you end up removing the 'old' healthchecks? 14:17:49 <odyssey4me> would it not be possible to just have the /healthceck access point work for those that have a proper one, and not there for those that don't? 14:17:49 <Tengu> as in "systemd status ...." 14:17:53 <marios|ruck> Tengu: or not fully or did we change direction on that? 14:17:55 <opendevreview> Sorin Sbârnea proposed openstack/tripleo-repos master: Add ansible-test sanity checks job (nv) https://review.opendev.org/c/openstack/tripleo-repos/+/803353 14:18:07 <marios|ruck> Tengu: hmm guess not i see them in logs from today jobs 14:18:09 <Tengu> marios|ruck: it's on hold right now, in order to get that spec done. We can't really stay without any replacement 14:18:15 <opendevreview> Francesco Pantano proposed openstack/tripleo-heat-templates master: DNM - Test Pending Ceph Pacific daemons in CI https://review.opendev.org/c/openstack/tripleo-heat-templates/+/778915 14:18:28 <marios|ruck> Tengu: i see so removal depends on this 14:18:41 <Tengu> marios|ruck: had to sync with CloudOps, they missed the first spec. So now I'm really closely collaborating with them 14:18:55 <marios|ruck> Tengu: cos they rely on those i guess ack 14:19:02 <Tengu> apparently, some ppl are actually relying on the healthchecks (that's carzy ;)) 14:19:06 <marios|ruck> haha 14:19:24 <marios|ruck> k any other comments will move on in a sec please review the spec https://review.opendev.org/c/openstack/tripleo-specs/+/803152 14:19:27 <bshephar> Yeah, +1 on avoiding podman specifc implementation there 14:20:26 <marios|ruck> next one item from tripleo-ci 14:20:36 <marios|ruck> #info https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 wire up tripleo_get_hash ansible module in repo-setup role (paused/behind major refactor at https://review.opendev.org/c/openstack/tripleo-repos/+/801029 and https://review.opendev.org/c/openstack/tripleo-repos/+/801621 14:21:00 <marios|ruck> this was discussed at ptg we had a session on tripleo-repos .. we now have two independent sub modules there for get hash and yum repos 14:21:08 <marios|ruck> some pointers there if you are interested in following that work 14:21:18 <marios|ruck> any comments or questions about that? 14:22:03 <eliadcohen> no comments marios|ruck 14:22:06 <marios|ruck> #info reminder https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-08-03-Community-Call every tuesday community call 1330 UTC 14:22:10 <marios|ruck> thanks eliadcohen :D 14:22:16 <marios|ruck> just a reminder about the ci community call 14:22:22 <marios|ruck> every tuesday 1330 UTC 14:22:35 <marios|ruck> ping us on #tripleo or #oooq if you want to join we can add you to the google meet 14:22:40 <zbr> cloudnull: let me know when you have few minutes to chat about sanity checks. 14:22:49 <marios|ruck> zbr: we are in the tripleo meet ;) 14:22:53 <marios|ruck> moving on 14:23:00 <marios|ruck> storage folks o/ please go ahead fultonj 14:23:02 <cloudnull> happy to me at the top of the hour 14:23:16 <fultonj> fmount: care to comment about promotion? 14:23:21 <cloudnull> to **meet -cc zbr 14:23:24 <fmount> sure thanks fultonj 14:23:39 <fmount> we are working on the promotion of -pacific v16.2.5 14:23:56 <fmount> but we had an issue w/ ceph-nfs because of the Ceph container build 14:24:16 <fmount> and we're looking for a new rebuild of the ceph daemon image before moving forward w/ it 14:24:28 <fmount> the new bits allow us to re-enable the ceph dashboard 14:24:32 <fmount> in the cephadm context 14:24:35 <opendevreview> Martin Schuppert proposed openstack/tripleo-heat-templates stable/wallaby: Change default for NovaNfsVersion to NFSv4.2 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/803285 14:24:39 <fmount> and finalize the migration to cephadm 14:24:48 <fmount> #link https://review.opendev.org/c/openstack/tripleo-common/+/797226 14:24:58 <fmount> the review to make the promotion when it's ready ^^^ 14:25:15 <fmount> fultonj: would you cover the deployed ceph part? 14:25:23 <fultonj> the deployed_ceph feature basically working 14:25:25 <fultonj> #link https://review.opendev.org/q/topic:%22deployed_ceph%22+(status:open%20OR%20status:merged) 14:25:42 <fultonj> it has 2 python unit tests ATM and I'll be adding a few more 14:25:46 <fultonj> i'll add molecule test too 14:26:04 <fultonj> adding it to scenario001 will be trickier 14:26:11 <fultonj> i'd try a POC of that in a separate review 14:26:24 <fultonj> comments on the testing for that? 14:26:35 <fultonj> it's a new cli option to deploy ceph before overcloud 14:26:43 <fmount> yeah not sure, we should make some work in quickstart I guess 14:26:46 <fmount> to add ceph first 14:26:51 <fmount> before deploying the overcloud 14:27:38 <marios|ruck> fmount: maybe you can join one of the ci community calls when you're ready to discuss wiring into scenario1 if you have questions/doubts 14:27:43 <marios|ruck> and fultonj ^ 14:27:50 <fultonj> ack 14:27:51 <fmount> +1 thanks marios|ruck 14:27:56 <fmount> it's a good idea 14:28:02 <fultonj> feel free to review deployed_ceph then 14:28:02 <marios|ruck> is that in tripleo-ansible? i mean the deployed ceph is it a role? 14:28:21 <bshephar> fultonj: What does the updates and upgrade situation look like for that? Is it an independent update process for Ceph? 14:28:43 <fultonj> a patch to python tripleo client calls a new playbook which calls the existing roles 14:28:46 <marios|ruck> good question bshephar, or going from 'old' ceph deployment to the new one 14:28:46 <fultonj> marios|ruck: ^ 14:28:55 <fmount> bshephar: from pacific yes, it probably will be a separated process 14:29:02 <fmount> since it's managed by cephadm 14:29:04 <fmount> and it's async 14:29:14 <fultonj> deployed_ceph is orthogonal to upgrades 14:29:14 <fmount> compared to the traditional ansible run 14:29:28 <bshephar> marios|ruck: I was thinking just about new deployments. But yes, i guess coming from existing overcloud with ceph to this model is a good point. 14:29:36 <fultonj> upgrades are the same regardless 14:29:42 <fmount> right, this is valid for both deployed ceph or regular overcloud deployments w/ ceph 14:29:44 <fultonj> and the upgrade process moves you to cephadm management 14:30:05 <bshephar> Nice, this is great. 14:30:31 <marios|ruck> any other discussion here? moving on in a sec 14:30:55 <fultonj> nothing more to add from ceph side, thanks 14:31:06 <marios|ruck> fultonj: fmount: i have a different ceph related question but will leave to open discussion :) 14:31:15 <fultonj> ok, sure 14:31:33 <marios|ruck> k thanks Tengu fmount fultonj for bringing your topics today moving on i don't see anything else in the etherpad 14:31:39 <Tengu> np :) 14:31:49 <marios|ruck> #topic Bugs and blueprints 14:31:49 <marios|ruck> #link https://bugs.launchpad.net/tripleo/ 14:31:49 <marios|ruck> #link https://storyboard.openstack.org/#!/project/openstack/tripleo-ansible 14:31:49 <marios|ruck> #link https://launchpad.net/tripleo/+milestone/xena-2 14:31:50 <marios|ruck> #link https://launchpad.net/tripleo/xena 14:31:58 <marios|ruck> we have had a couple of gate blockers this last 2 weeks 14:31:59 <opendevreview> Kevin Carter proposed openstack/tripleo-common master: [FIPS] Remove paramiko from tripleo-common https://review.opendev.org/c/openstack/tripleo-common/+/803354 14:32:35 <marios|ruck> https://bugs.launchpad.net/tripleo/+bug/1937333 https://bugs.launchpad.net/tripleo/+bug/1938079 14:32:41 <marios|ruck> i think there was one more but don't have it handy 14:32:51 <marios|ruck> last couple of days we are a bit more stable 14:32:53 <marios|ruck> promotions are good 14:33:00 <marios|ruck> any bug related business from anyone? 14:33:26 <marios|ruck> moving on ... 14:33:32 <marios|ruck> #topic Project releases or stable backports 14:33:34 <marios|ruck> #info tripleo wallaby repos https://releases.openstack.org/teams/tripleo.html#wallaby 14:33:40 <marios|ruck> covered releases in the one off items 14:33:48 <marios|ruck> any other release requests ? 14:34:04 <marios|ruck> ping me if you need something, otherwise i'll start another round through the branches in order in a week or so 14:34:25 <marios|ruck> #topic specs 14:34:25 <marios|ruck> #info https://review.opendev.org/q/project:openstack/tripleo-specs 14:34:25 <marios|ruck> #info https://opendev.org/openstack/tripleo-specs/src/branch/master/specs/xena 14:34:36 <marios|ruck> couple of specs to call out 14:35:04 <marios|ruck> https://review.opendev.org/c/openstack/tripleo-specs/+/801630 TripleO.Next - Task-Core + Directord 14:35:21 <marios|ruck> https://review.opendev.org/c/openstack/tripleo-specs/+/797676 TripleO.Next - Container Pods Spec 14:35:42 <marios|ruck> we've mentioned the healthchecks spec and the 'moving repos to independent release' earlier 14:36:01 <marios|ruck> any comments or questions? 14:36:21 <marios|ruck> #topic open discussion 14:36:23 <marios|ruck> Anything else that folks want to bring up to the meeting? 14:36:36 <marios|ruck> fultonj: fmount: o/ what can we do about https://bugs.launchpad.net/tripleo/+bug/1934879/comments/15 14:36:42 <marios|ruck> fmount: fultonj: can we move to octopus for Train? 14:36:56 <fmount> marios|ruck: I don't think so 14:37:02 <fmount> no octopus for train 14:37:16 <marios|ruck> :/ 14:37:18 <marios|ruck> k 14:37:24 <fmount> and the commit that fixes this issue was backported to nautilus 14:37:29 <fmount> in July 14:37:37 <fmount> but we don't have any new release 14:37:40 <fmount> atm 14:37:52 <marios|ruck> ah i see so its not they didn't accept the patch it was no new build made 14:38:03 <fmount> new tag (upstream) means we can do a new build in cbs and update our containers w/ the ceph bits 14:38:14 <fmount> yeah 14:38:19 <fmount> there's no build 14:38:38 <marios|ruck> k thanks just was on my mind since i updated the bug earlier 14:38:40 <fmount> if I see a new build I can proactively test and revert the skiplist change 14:38:52 <opendevreview> chandan kumar proposed openstack/tripleo-quickstart-extras master: Move build-test-packages as a part of undercloud setup https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/790926 14:38:52 <fmount> but until that moment, we can just skip it 14:39:04 <fmount> marios|ruck: sure np, thanks for asking 14:39:21 <marios|ruck> any other tripleo business please ? 14:39:52 <marios|ruck> #info next meeting tuesday 17th August 14:40:12 <marios|ruck> thanks all o/ 14:40:20 <marios|ruck> #endmeeting tripleo