14:00:00 #startmeeting tripleo 14:00:00 Meeting started Tue May 11 14:00:00 2021 UTC and is due to finish in 60 minutes. The chair is marios. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:03 The meeting name has been set to 'tripleo' 14:00:13 #topic agenda 14:00:13 * Review last minutes & action items 14:00:13 * One off agenda items 14:00:13 * Bugs & Blueprints 14:00:13 * Projects releases or stable backports 14:00:15 * Specs 14:00:18 * open discussion 14:00:20 Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:29 hello o/ tripleo good morning good afternoon good evening all who is around today? 14:00:32 o/ 14:00:34 o/ 14:00:40 o/ 14:00:44 o/ 14:01:03 «o/ 14:01:17 <_dvd> \o 14:01:20 o/ 14:01:27 i got some feedback that we move a little quickly in these meetings so i am going to slow it down a liiitle bit today. if you have an opinion about that please let me know :) 14:01:32 o/ 14:01:33 o/ 14:02:00 #topic review last meeting logs & action items 14:02:11 #link http://eavesdrop.openstack.org/meetings/tripleo/2021/tripleo.2021-04-27-14.00.html 14:02:17 #info last meeting 27th april 14:02:26 o/ 14:02:36 anyone have something they want to hilight from that? ^^^ please speak up if so. /me checks logs 14:03:32 there was an AI for me about the xena milestones/launchpad but i did that will comment in a minute about that 14:03:38 k moving on then 14:04:00 #topic one off agenda items 14:04:02 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:04:13 let's do these in order as usual 14:04:26 <_dvd> #link https://review.opendev.org/c/openstack/tripleo-ansible/+/783969 14:04:28 chandan kumar proposed openstack/tripleo-quickstart master: [DNM] remove tripleo operator https://review.opendev.org/c/openstack/tripleo-quickstart/+/790701 14:04:30 dviroel: o/ hey do you want to go ahead looks like you're first 14:04:35 thanks :) 14:04:44 <_dvd> so this is a blocker for 16.2 14:04:56 <_dvd> I did a lot of tests, mostly upgrades and dcn 14:05:18 <_dvd> that would be great to have this fully QE'd before we're too close for the deadline 14:05:27 ack... fyi we avoid using product numbers here they mean nothing if you aren't using red hat product so it is inappropriate (sorry i have to say that as i am ptl just reminder easy mistake to make in this project) 14:05:43 _dvd: ^^ 14:05:55 folks please help review this patch https://review.opendev.org/c/openstack/tripleo-ansible/+/783969 add it to your queue 14:06:01 :) 14:06:04 <_dvd> my apologies, I'm not familiar 14:06:06 apologies dviroel i confused the nick 14:06:11 np marios 14:06:19 _dvd: no problem as i said i have to say that as i am ptl ;) 14:06:30 <_dvd> sure :) 14:06:53 _dvd: jeez i just saw how huge that is 14:06:53 _dvd: you'll have to run around the channel twice 14:06:59 _dvd: is there no way you can break it down a bit? 14:07:11 <_dvd> so I'm very responsive with this patch, it's highest priority for my team, so if you have any feedback, or suggestions, you can ping me 14:07:19 <_dvd> lol Tengu 14:07:22 mwhahaha: cloudnull: please add that to your queue? https://review.opendev.org/c/openstack/tripleo-ansible/+/783969 14:07:27 slagle: ^^ 14:07:27 no 14:07:30 mwhahaha: k thanks 14:07:31 :D 14:07:32 i don't like that thing at all 14:07:35 * mwhahaha is serious 14:07:37 ack that is a review 14:07:39 * marios too 14:07:43 i don't think tripleo should be handling that 14:08:00 i really don't like the tsx patch and have been avoiding it so i don't -2 it 14:08:31 <_dvd> marios, yup, basically there's a change in some kernel number and some other kernel number from some RHEL version that could affect live migration of VMs during upgrade process, and this patch is basically to make it transparent to the operator 14:08:35 _dvd: seems like the kind of thing that would be best discussed at ptg (i.e. why didn't we do that ..) 14:08:37 <_dvd> and I agree wholeheartly with mwhahaha 14:08:47 _dvd: if you are unable to address the concerns this cycle 14:08:52 chandan kumar proposed openstack/tripleo-quickstart master: [DNM] remove tripleo operator https://review.opendev.org/c/openstack/tripleo-quickstart/+/790701 14:08:57 weshay|ruck: sshnaidm ^^ 14:09:01 it's defintately not something that i'm ok backporting to train 14:09:07 which is why i say this needs a different solution 14:09:28 <_dvd> mwhahaha, ideally we'd get this fix at the kernel level and I believe our PM tried that with no success 14:09:57 mwhahaha: can you please go write how much you dislike the patch on the patch? 14:10:03 sure i -2d it 14:10:06 hehehe 14:10:06 <_dvd> I tried to make a bunch of molecule validations to make this as solid as possible 14:10:16 oh good, now i don't have to review it 14:10:25 _dvd: can we continue the conversation in gerrit i think we should move in a sec because of time 14:10:39 _dvd: i mean i got mwhahaha to look at your patch? 14:10:43 * marios didn't know he was gonna -2 it 14:10:46 * marios shrugs 14:10:48 :/ 14:10:49 :D 14:10:52 <_dvd> mwhahaha, do you have any other solution ? 14:11:02 i did review the first round, and was generally ok with it now 14:11:07 _dvd: but more seriously, i think gerrit is a good place to discuss? 14:11:09 <_dvd> I don't mind dropping it and rewriting 14:11:17 _dvd: so others can see and also understand the changes made 14:11:22 solve it in nova? 14:11:25 * mwhahaha shrugs 14:11:37 <_dvd> well that would be great to have an alternative solution before working on something blindly :) 14:11:38 hanlding kernel flags is not something tripleo should care about 14:11:54 it's a workaround for libvirt/nova problems 14:12:02 which is the appropriate place to address this issue 14:12:04 not tripleo 14:12:10 <_dvd> ack 14:12:16 it broke live migrate. that's not a tripleo thing 14:12:25 why can't libvirt/nova know about tsx? 14:12:39 or fix it? 14:12:42 * mwhahaha shrugs 14:13:10 <_dvd> mwhahaha, there's some work being done in nova but that wouldn't fix the upgradefrom 16.1 to 16.2 issue 14:13:26 <_dvd> mosly with already running VMs 14:13:31 i would document this 14:13:35 and not try and fix in code 14:13:43 <_dvd> some of the operators are expecting 0 downtime 14:13:45 and maybe a validation 14:14:02 <_dvd> there's no nova fix that can be applied without rebooting VMs 14:14:11 anyway let's move on. i get that expectation but we need to push back on rhel better when they do this (they always do this) 14:14:42 the tsx value is changing b/c of security right? so we shouldn't automate around that. 14:15:10 document and let the user set it back to less secure if they want 0 downtime live migration 14:15:10 <_dvd> so this is more of a workaround where people are expected to disable TSX when they can 14:15:23 a validation might be nice indeed. 14:15:44 but automation..... really, no. especially if it involves hardcoded versions 14:15:48 (like the kernel) 14:16:31 Tengu: but you can understand the use case... it isn't about fresh deployments, more about handling the upgrade case without the user havingg to make manual intervention 14:16:48 yea sorry but this needs manual intervention 14:16:59 this is not something we should try and automate around 14:17:22 we've done that too much and it only leads to pain 14:17:23 sure. still - not sure it's for tripleo to automate. a validation can point the issue, and be added to the pre-update/upgrade as well as post-update/upgrade. Documentation can also be added. 14:17:28 _dvd: do you have enough feedback for now to go back to your team with? 14:17:33 <_dvd> I don't totally agree with the manual intevention, the feedback we have from operators is that there's a lot of stuff to think already for upgrades and deployments, etc. They are probably gonna spend days and weeks just because of this TSX thing 14:17:36 _dvd: or to continue on gerrit 14:17:43 it will be a real pain to maintain in the near future. 14:17:50 <_dvd> but yeah, I'll continue on gerrit 14:17:57 _dvd: or possibly join one of the deployment team sync (ping mwhahaha or Tengu or slagle about that?) 14:18:04 _dvd: a validation can make the thing "easier" to spot. 14:18:06 if you want to automate it, do it elsewhere 14:18:09 aka not tripleo 14:18:41 <_dvd> all right 14:18:42 k let's move on cos this is the first of a few topics in the etherpad today 14:18:56 thanks _dvd and all for that discussion carry on in https://review.opendev.org/c/openstack/tripleo-ansible/+/783969 14:19:03 slagle: around? please go ahead? 14:19:08 sure 14:19:18 tripleo-common is now removed from global-requirements and upper-constraints 14:19:21 #link https://review.opendev.org/c/openstack/requirements/+/790091 14:19:23 #link https://review.opendev.org/c/openstack/requirements/+/789706 14:19:24 nice 14:19:41 our unit tests and functional tests will now test the same code. 14:19:48 a novel concept 14:19:49 seems like a good thing 14:20:08 ++ 14:20:10 slagle: also per our discussion few days ago it may mean we are no longer _obliged_ to cut stable/branch for tripleo-common before verything else 14:20:27 yes i think it makes that easier 14:20:40 next...ephemeral-heat update 14:20:52 i have a ci patch to switch containers-multinode 14:20:55 #link https://review.opendev.org/c/openstack/tripleo-quickstart/+/777108 14:21:11 it was passing prior to quay breakage, but it should be passing again soon 14:21:41 and just wanted to mention there will be further backports for ephemeral-heat and network-data-v2 stuff into wallaby 14:22:04 slagle: you think we should wait for those? ^^^ or make a release once we stabilise with the current backports 14:22:11 slagle: then another once the heat/v2 stuff happens 14:22:27 slagle: i haven't seen those patches for a couple weeks don't know what the status is are they already merged in master you mean? 14:22:32 we need to address the ovn mac address vs standalone before backporting to wallaby 14:22:32 releases are cheap, so i dont think we need to wait 14:22:46 i thought that was fixed? 14:22:49 no 14:22:55 #link https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790166 14:23:02 it's broken on reinstall now 14:23:08 it needs revisiting 14:23:39 ok 14:23:45 see also comments on https://review.opendev.org/c/openstack/tripleo-ansible/+/790204 14:24:47 alright, so that needs to be looked at agin 14:25:45 that's all i have really 14:25:55 thanks slagle 14:26:09 storage folks fmount fultonj gfidente o/ around? please go ahead 14:26:26 quay.ceph.io had an outage which affected CI 14:26:29 it came back 14:26:35 Thanks to Chandan for looking into workaround 14:26:41 #https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 14:26:55 perhaps we keep looking into ^ so we're more resiliant ? 14:27:35 we can disucss further in the review 14:27:45 bug 1928078 14:27:45 bug 1928078 in tripleo "cephadm issue hitting ceph-related CI jobs" [Critical,Confirmed] https://launchpad.net/bugs/1928078 - Assigned to Francesco Pantano (fmount) 14:27:53 was caused by a premature promotion of cephadm 14:28:11 fultonj: marios I left a comment in the lp bug ^ 14:28:20 we'll follow up to ensure it's tested before it's promoted 14:28:27 ++ exactly ^^ 14:28:28 fmount: looking 14:28:37 it's the pattern we had w/ ceph-ansible but with the new package instead 14:28:41 thanks fmount 14:29:21 we're still working on moving fully to cephadm during X 14:29:42 we're working with ceph orch people on ganesha blockers 14:29:56 that's all 14:30:05 thanks fultonj 14:30:24 thanks for the update fultonj fmount as always. anyone have any questions or comments before we move on? 14:31:00 spotz: o/ hello you have the first item in community 14:31:05 spotz: do you want to introduce that? 14:31:22 Hey so this was actually asked of me internally vs community. 14:31:25 typing:) 14:31:40 #info "multi-region in OpenStack Tripleo?" is anyone working on that? discussed a while back in https://review.opendev.org/c/openstack/tripleo-specs/+/566448/ and a bit more recently there https://review.opendev.org/c/openstack/tripleo-specs/+/600555 14:32:07 spotz: as i said there seems to be discussion in both the specs with technical reasons for pushing back against it 14:32:11 OpenStack can bee iinstalled multi-regionn using a single keystone. But we through TrripleO doesn't have that functionality so I was asking if there weere any plans 14:32:25 spotz: so i am not aware of any other efforts, and perhaps someone here knows or plans to work there 14:32:34 slagle: perhaps? ^^ "multi-region in OpenStack Tripleo?" 14:32:56 hmmm sounds more a multi-region with single keystone, than multi-region 14:32:59 O so need a newe keyboard:) 14:33:14 not currently on the radar afaik 14:33:31 don't know of anything 14:33:43 maybe Security knows - they own keystone.. ? 14:33:50 xek: -^^ ? 14:33:56 spotz: i think it definitively answers your question at least from tripleo side 14:34:22 Yeah:) And that's fine. 14:34:24 iirc the best thing to do is link keystone to some ldap/idm backend, for this kind of things... ? 14:34:36 at least you'll get shared accounts. 14:34:40 the times it has come up in the past, most don't understand what multi-region actually is and is not. when they learn it's just about sharing keystone, there is no interest 14:34:55 Tengu, that's multi domain, not multi region iiuc 14:35:22 tkajinam: /me raises his multipass 14:35:34 multi region would requires some effort in some aspects 14:35:52 at least I have spent some amount of time to add missing region_name parameters to maaaany places 14:36:09 which is required for multi region, though I don't think I have fixed everything 14:36:23 you could do it with dcn to a certain extent but no it's not currently something we are working on 14:37:07 spotz: you ok to move on for now? 14:37:38 marios Yep I'm good I'll pass on the info 14:37:51 spotz: thank you for the item and thanks all for chiming in on that 14:38:03 Daniel Bengtsson proposed openstack/tripleo-validations stable/victoria: Add validation to check DockerInsecureRegistryAddress matches the UC hostname https://review.opendev.org/c/openstack/tripleo-validations/+/790709 14:38:11 i'll try to get through the next few community items real quick please interrupt me if you want to say something about any of them 14:38:21 for the xena1 triage i'll do that in a bit 14:38:26 #info http://lists.openstack.org/pipermail/openstack-discuss/2021-April/022130.html * Proposing Cédric Jeanneret (Tengu) for tripleo-core James Slagle 14:38:31 thanks slagle and congrats Tengu o/ 14:38:39 #info http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022209.html * [ptg] Summaries blogpost 14:38:56 this is a nice compilation of the known summaries across the different projects ^^^ 14:39:04 http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022278.html * [TripleO] stable/wallaby branch for tripleo repos \o/ ping if problems :) 14:39:13 #info http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022278.html * [TripleO] stable/wallaby branch for tripleo repos \o/ ping if problems :) 14:39:21 so we created stable/wallaby ... and its a bit of a mess 14:39:30 because of triple-common being a bit stale 14:39:40 there are historical reasons for why we branch it first 14:39:56 some of which is related to the patch that slagle proposed https://review.opendev.org/c/openstack/requirements/+/790091 14:40:21 but the main thing i want to say is that it was my fault and i accept responsibliity, mainly for not declaring and enforcing a feature freeze, once we had created the initial tripleo-cmmon 14:40:27 i will try and document this for the next person 14:40:38 we are fixing with https://review.opendev.org/q/topic:unlock-wallaby 14:40:58 so it should be almost done last 2/3 patches now and we will be able to stabilise tripleo-common/python-tripoleoclient and re-tag them 14:41:08 and finally 14:41:09 #info http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022323.html * [TripleO] Closed out wallaby milestone in Launchpad and moved bugs to xena-1 Marios Andreou 14:41:23 i moved bugs to xena-1 - pleaes take some time to triage your xena1 bugs 14:41:39 * marios pause for any comments or questions on any of those (sorry for rush but time) 14:42:16 k thanks 14:42:18 lets move on 14:42:22 Daniel Bengtsson proposed openstack/tripleo-validations stable/victoria: Add validation to check DockerInsecureRegistryAddress matches the UC hostname https://review.opendev.org/c/openstack/tripleo-validations/+/790709 14:42:23 jbadiapa: o/ around? 14:42:27 jbadiapa: please go ahead 14:42:29 yeap, 14:43:00 I would like to point out that the documentation is outdated 14:43:07 jbadiapa: yes +1 :D 14:43:31 jbadiapa, any particular or everything? 14:43:51 ysandeep: well in general some of our upstream docs are stale 14:43:56 ysandeep: our tripleo-ci stuff isn't too bad 14:44:04 ysandeep: but there are very old branch references there for example 14:44:04 I'm going to work on updating the documentation for being able to use tripleo-quickstart to deploy environments 14:44:07 ysandeep: like rocky etc 14:44:34 there is still kolla pointed for building containers and so on 14:44:38 jbadiapa: that sounds good 14:44:44 jbadiapa: any contributions welcome 14:44:51 I know that people are really busy 14:44:54 :) 14:44:56 jbadiapa: wondering if we at least want to share a topic branch for xena documentation improvmenets? 14:45:01 (sorry, was busy hitting bugs) 14:45:04 like xena-docs-fixes or something 14:45:04 should we be encouraging use of quickstart or standalone? 14:45:28 i thougth quickstart became a CI-only tool 14:45:42 fultonj I guess that depends on what you want to test 14:45:45 fultonj: you can still in theory use it for deployments if you want to 14:46:21 * ysandeep still deploys my own lab with quickstart 14:46:42 k jbadiapa please bring any updates here for reviews on the next meeting ? 14:46:46 ok 14:46:54 sure, 14:46:59 jbadiapa: something further you wanted to discuss on this one? 14:47:00 the xena-docs-fixes topic seems a good idea 14:47:09 I thnk that's all from my side 14:47:23 sure fultonj i have heard a few folks asking about making docs contributions since the ptg 14:47:32 maybe common topic branch can help track it 14:47:55 #info please try to use xena-docs-fixes topic branch if you work on docs improvements this cycle 14:47:55 e.g. want to try standalone 14:47:57 https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/standalone.html 14:48:06 here's a million options going back to queens 14:48:13 +1 for common topic branch + we should identify which docs are stale 14:48:13 now tnat you're in the middle of the doc, step 1... 14:48:13 moving on cos time sorry 14:48:14 :) 14:48:25 thanks jbadiapa 14:48:26 fultonj thx 14:48:31 marios, thx 14:48:32 _dvd: o/ hey do you want to go ahead please 14:48:38 Daniel Bengtsson proposed openstack/tripleo-validations stable/victoria: Add validation to check DockerInsecureRegistryAddress matches the UC hostname https://review.opendev.org/c/openstack/tripleo-validations/+/790709 14:48:40 _dvd: you have review requests there 14:48:46 is there something further to add 14:48:52 <_dvd> sure, but is it the right place? 14:49:11 <_dvd> it's 2 simple fixes 14:49:12 _dvd: yes absolutely fine to ask for reviews if you want to add some context about it fine 14:49:13 <_dvd> #link https://review.opendev.org/c/openstack/validations-common/+/788119/ 14:49:18 otherwise 'please check these' 14:49:23 <_dvd> #link https://review.opendev.org/c/openstack/validations-common/+/788105/ 14:49:46 any comments questions about those? 14:50:15 k thanks _dvd 14:50:18 <_dvd> basically, without this, validations can fail without providing the right error 14:50:37 <_dvd> and safeconfigparser is not the right parser because oslo_config accepts duplicates options 14:50:54 Daniel Bengtsson proposed openstack/tripleo-validations stable/victoria: Add validation to check DockerInsecureRegistryAddress matches the UC hostname https://review.opendev.org/c/openstack/tripleo-validations/+/790709 14:50:56 <_dvd> I don't know what else to add :) 14:51:18 _dvd: ack :) thanks for the context it will help to sell that to any reviewers reading this in logs later ;) 14:51:31 jlarriba: o/ hey please go ahead? backup and restore items 14:51:36 sure 14:51:53 so, basically i am asking for some eyes for review 14:51:54 #link https://review.opendev.org/c/openstack/tripleo-ansible/+/790214 14:52:18 jlarriba: as discussed in the review you want this back to wallaby too ? no further ? 14:52:53 We want to change the way the backup of the overcloud is done (without having to stop everything at a time) and a change that implemented that went through, but it is better to first keep at least one version for deprecation of the old feature 14:53:02 this goes just to wallaby 14:53:20 but it will also enable to introduce bugfixes that cover from wallaby until train 14:54:03 k any comments or questions about this item 14:54:04 i brought it here because of a bit on a hurry due to necessary bugfixes that need to go to train due to upcoming internal releases 14:54:21 jlarriba: but what does train have to do with it if this is only to wallaby 14:55:33 k going to move on from here in a sec we can discuss in the review 14:55:37 we only have 5 mins left 14:55:43 yeah, ok 14:55:51 thank you jlarriba 14:56:01 moving on then thanks all for those items and updates and discussions 14:56:03 that was all 14:56:11 #topic Bugs and blueprints 14:56:12 #link https://bugs.launchpad.net/tripleo/ 14:56:12 #link https://storyboard.openstack.org/#!/project/openstack/tripleo-ansible 14:56:12 #link https://launchpad.net/tripleo/+milestone/xena-1 14:56:12 #link https://launchpad.net/tripleo/xena 14:56:14 #link https://launchpad.net/tripleo/wallaby 14:56:23 i wanted to spend 5 mins doing som etriage but we don't have time 14:56:40 do please usual reminder please check https://launchpad.net/tripleo/+milestone/xena-1 and update the status of any bugs you are asssigned to or aware of etc 14:56:43 thank you 14:56:58 i am gonna fly through there please shout if you want to say something ! 14:56:59 :D 14:57:06 #topic Project releases or stable backports 14:57:08 #info tripleo wallaby repos https://releases.openstack.org/teams/tripleo.html#wallaby 14:57:15 so already mentioned the stable/wallaby release with links earlier 14:57:41 once we stabilise with merges on https://review.opendev.org/q/topic:unlock-wallaby we can cut some new versions of all the things for wallaby 14:57:51 #topic specs 14:57:51 #info https://review.opendev.org/q/project:openstack/tripleo-specs 14:57:52 #info https://opendev.org/openstack/tripleo-specs/src/branch/master/specs/wallaby 14:58:07 there are a few specs posted thanks Tengu sbaker and any others 14:58:14 mwhahaha: why are we executing external_deploy_steps for roles with 0 node count? Isn't that a waste of cycles? 14:58:16 please keep them coming ... these are specs from items discussed at ptg 14:58:21 please help to review these 14:58:25 and finally 14:58:29 #topic open discussion 14:58:30 Anything else that folks want to bring up to the meeting? 14:58:30 wew 14:58:46 i like that bit where i said "I'm gonna take it a bit slower this time" 14:58:48 it was great 14:58:49 hjensas: external deploy tasks run on undercloud always. it doesn't get skipped because we don't do role checks 14:59:03 hjensas: external deploy tasks aren't role aware 14:59:17 well ok then, thanks all 14:59:25 next meeting in 2 weeks 25 May 14:59:36 have a good rest of day o/ 14:59:41 thanks marios :) 14:59:47 thx marios 14:59:47 #endmeeting tripleo