14:00:38 <EmilienM> #startmeeting tripleo 14:00:39 <openstackgerrit> John Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split-controlplane-standalone.yml playbook https://review.openstack.org/617368 14:00:39 <openstack> Meeting started Tue Dec 4 14:00:38 2018 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:39 <ssbarnea|rover> marios_|ruck: are of tempest-sendmail.tripleo.org:8080 error on gate check? http://logs.openstack.org/24/567224/141/check/tripleo-ci-centos-7-containers-multinode/8888af1/job-output.txt.gz 14:00:39 <jaosorior> yes 14:00:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:42 <openstack> The meeting name has been set to 'tripleo' 14:00:44 <EmilienM> #topic agenda 14:00:45 <Tengu> :) 14:00:48 <arxcruz> o/ 14:00:49 <EmilienM> * Review past action items 14:00:50 <jaosorior> o/ 14:00:50 <EmilienM> * One off agenda items 14:00:52 <EmilienM> * Squad status 14:00:54 <EmilienM> * Bugs & Blueprints 14:00:55 <florianf> o/ 14:00:55 <chandan_kumar> \o/ 14:00:56 <EmilienM> * Projects releases or stable backports 14:00:57 * jaosorior half here 14:00:58 <EmilienM> * Specs 14:01:00 <EmilienM> * open discussion 14:01:02 <EmilienM> Anyone can use the #link, #action and #info commands, not just the moderator. 14:01:04 <EmilienM> Welcome TripleOers! 14:01:05 <jfrancoa> o/ 14:01:05 <ksambor> o/ 14:01:07 <fultonj> o/ 14:01:07 <weshay_mtg> 0/ 14:01:11 <marios> o/ 14:01:11 <jtomasek_> o> 14:01:15 <panda> o/ 14:01:17 <jistr> \o/ 14:01:18 <rfolco> o/ 14:01:26 <chem> o/ 14:01:32 <EmilienM> #topic review past action items 14:01:33 <Tengu> EmilienM: Triple-Owlers" maybe? :) 14:01:36 <EmilienM> none, moving on 14:01:41 <EmilienM> #topic one off agenda items 14:01:46 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:01:46 <ccamacho> hi! 14:02:00 <EmilienM> arxcruz: go ahead 14:02:08 <beagles> o/ 14:02:09 <arxcruz> So, as some of you guys know, in the future, we want to replace validate-tempest role with os_tempest role 14:02:19 <arxcruz> Right os_tempest role is part of openstack-ansible 14:02:22 <jrist> o/ 14:02:33 <arxcruz> right now is a little bit simple, just run tempest and set some configs 14:02:39 <jrist> EmilienM: TripleOwls 14:02:55 <arxcruz> we are working to improve the functionalities there, and once we have everything that it requires to replace validate-tempest, make the move 14:03:05 <arxcruz> right now we add support to python-tempestconf and stackviz 14:03:06 <openstackgerrit> Merged openstack/python-tripleoclient master: Don't exit on RUNNING status on delete https://review.openstack.org/619583 14:03:07 <openstackgerrit> Merged openstack/tripleo-common master: Bump the Ceph container image to 3.2.0 https://review.openstack.org/619503 14:03:08 <openstackgerrit> Merged openstack/tripleo-heat-templates master: ctlplane pre-alloc IPs - deprecated/non-deprecated role name https://review.openstack.org/620391 14:03:10 <openstackgerrit> Merged openstack/tripleo-heat-templates master: Add a gateway IP to the Management net in network_data.yaml https://review.openstack.org/599886 14:03:11 <openstackgerrit> Merged openstack/tripleo-common stable/rocky: Bump the Ceph container image to 3.2.0 https://review.openstack.org/619505 14:03:12 <arxcruz> that are two tools that tripleo uses 14:03:16 <arxcruz> to execute tempest 14:03:31 <arxcruz> this sprint, we are working to improve python-tempestconf options 14:03:44 <openstackgerrit> Bogdan Dobrelya proposed openstack/paunch master: Implement podman rename via re-apply of containers https://review.openstack.org/621607 14:03:46 <holser_> o/ 14:03:47 <arxcruz> and we hope soon we have a POC job running with os_tempest 14:04:10 <arxcruz> help are more than welcome, the openstack-ansible guys are very welcome to collaboration as well 14:04:33 <arxcruz> that's it from my side, i plan to have a status weekly here in the meeting 14:04:35 <arxcruz> :) 14:04:39 <EmilienM> cool, thanks! 14:04:58 <EmilienM> #info arxcruz working on switching tripleo ci to use openstack-ansible-os_tempest status 14:05:06 <EmilienM> let us know if you need help! 14:05:12 <arxcruz> sure :) 14:05:18 <EmilienM> anything else before we move to the next agenda topic? 14:05:35 <EmilienM> #topic Squad status 14:05:37 <arxcruz> nope, unless chandan_kumar or weshay_mtg have something to say 14:05:43 <EmilienM> ci 14:05:44 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:05:46 <EmilienM> upgrade 14:05:48 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:05:50 <EmilienM> containers 14:05:52 <EmilienM> #link https://trello.com/b/S8TmOU0u/tripleo-podman 14:05:54 <EmilienM> edge 14:05:56 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:05:58 <EmilienM> integration 14:06:00 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:06:02 <EmilienM> ui/cli 14:06:04 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:06:06 <EmilienM> validations 14:06:09 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:06:10 <chandan_kumar> arxcruz: nope from myside all said already 14:06:10 <EmilienM> networking 14:06:12 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:06:14 <EmilienM> security 14:06:16 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-security-squad 14:06:18 <EmilienM> I hope I didn't miss a squad 14:06:20 <EmilienM> is there any specific info someone wants to communicate on the squads? 14:06:50 <EmilienM> alright, moving on 14:06:53 <EmilienM> #topic bugs & blueprints 14:06:58 <EmilienM> #link https://launchpad.net/tripleo/+milestone/stein-2 14:07:00 <EmilienM> #link https://storyboard.openstack.org/#!/project_group/76 14:07:05 <EmilienM> For Stein we currently have 29 blueprints and about 907 (+167) open Launchpad bugs and 104 stories in Storyboard. 14:07:29 <EmilienM> (not sure why we have that number of bugs since last week but anyway) 14:07:48 <EmilienM> do we have any outstanding bug to report this week? 14:08:24 <marios_|ruck> EmilienM: we file a timeout bug this morning for containers multinode not sure this is the place for that discussion though 14:08:35 <marios_|ruck> https://bugs.launchpad.net/tripleo/+bug/1806632 tripleo-ci-centos-7-containers-multinode job is timing out more often than it passes & blocking gate 14:08:36 <openstack> Launchpad bug 1806632 in tripleo "tripleo-ci-centos-7-containers-multinode job is timing out more often than it passes & blocking gate" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) 14:08:36 <EmilienM> link? 14:08:46 <EmilienM> #link https://bugs.launchpad.net/tripleo/+bug/1806632 timout bug 14:09:00 <marios_|ruck> EmilienM: really the discussion is do we want to make it non voting. otherwise we need to drastically decrease the deployment time 14:09:23 <EmilienM> tripleo-ci-centos-7-containers-multinode is one of our most critical jobs I think, I would avoid to make it non voting 14:09:26 <marios_|ruck> EmilienM: posted the patch to make it non voting but just for this discussion 14:09:34 <EmilienM> have we identified where does it timeout? 14:09:38 <EmilienM> container pulls I see? 14:09:47 <marios_|ruck> EmilienM: right, but we have to do something since it just causes th queue to back up 14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:17 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1806521 14:10:19 <openstack> Launchpad bug 1806521 in tripleo "deployment fails on Ensure system is NTP time synced" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) 14:10:19 <EmilienM> merging code without tripleo-ci-centos-7-containers-multinode tested, it's basically merging untested code 14:10:26 <ssbarnea|rover> sorry to interrupt but tempest-sendmail.tripleo.org does not respond on 8080 thus -> https://bugs.launchpad.net/tripleo/+bug/1806699 14:10:27 <openstack> Launchpad bug 1806699 in tripleo "HTTPConnectionPool(host='tempest-sendmail.tripleo.org', port=8080): Max retries exceeded with url" [Undecided,Triaged] 14:10:29 <marios_|ruck> EmilienM: yeah i mean its like 1+ hr undercloud deploy 1+hr for overcloud/controller some info on that bug 14:10:59 <EmilienM> right, this isn't good 14:11:09 <EmilienM> who is working / investigating this one? 14:11:26 <marios_|ruck> EmilienM: agree, as i said posted this https://review.openstack.org/622187 just for the discussion 14:11:55 <EmilienM> can you WIP -1 maybe? 14:11:57 <marios_|ruck> EmilienM: well me and soren started dealing with it today. but this isn't ci 'issue' its more generally about decreasing/optimise the deployment 14:12:00 <marios_|ruck> EmilienM: ack 14:12:01 <openstackgerrit> Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Use a footnote. https://review.openstack.org/622332 14:12:06 <EmilienM> I would hate having this job non voting before we find a proper root cause 14:12:17 <hjensas> When I do testing in rdocloud I deploy undercloud and build images in parallell via ansible async tasks. afaik we don't build images in CI? But could we pull containers and do ServiceChains in paralell? 14:12:19 <EmilienM> marios_|ruck: I will take a look today at these container pull runtimes 14:13:05 <EmilienM> hjensas: I'm not sure we can easily pull containers without ServiceChains done 14:13:18 <EmilienM> because we use ServiceChains to figure out what containers we need to pull 14:13:58 <hjensas> EmilienM: right, good point. 14:14:07 <EmilienM> anyway, thanks marios_|ruck for bringing it 14:14:14 <EmilienM> why it doesn't have alert and such? 14:14:19 <marios_|ruck> hjensas: but there might be similar optimisations we can explore 14:14:25 <EmilienM> I think it needs to be more visible 14:14:27 <marios_|ruck> EmilienM: sure adding 14:14:42 <EmilienM> anything else bug related? 14:14:50 <marios_|ruck> EmilienM: well cos it isn't consistent fail/timeout i mean it does pass sometime s but fails far too often but adding alert 14:15:02 <jrist> jtomasek_: updated squad status 14:15:28 <jtomasek_> jrist: thx 14:15:38 <EmilienM> alright, moving on 14:15:41 <florianf> yes 14:15:43 <florianf> https://bugs.launchpad.net/tripleo/+bug/1805825 14:15:44 <openstack> Launchpad bug 1805825 in tripleo "tripleo_undercloud_conf_file hiera variable is missing after the instack-undercloud retirement" [Critical,New] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) 14:16:13 <EmilienM> how can we help? 14:16:44 <florianf> This bug causes some validations to fail since rocky. There's a patch up, but we need some t-h-t expertise to move this forward 14:16:53 <florianf> this is the patch: https://review.openstack.org/#/c/614470/2 14:17:34 <EmilienM> florianf: is it for addressing shardy's comment? 14:18:27 <florianf> Not just this, also about how to dynamically add a bind mount to the mistral_executor container based on the undercloud conf path 14:19:11 <EmilienM> ah ok 14:19:19 <EmilienM> I see, you need "if" etc probably 14:19:52 <EmilienM> florianf: I'll take a look as soon as I can unless someone helps in the review 14:19:58 <florianf> The question is if we (validations squad) should try to dig into how to achieve this with tht or if someone with a little more expertise can push this through a little more quickly (or help) 14:20:21 <florianf> EmilienM: That would be very very awesome... 14:20:43 <EmilienM> florianf: it would help if you could write in your review what you want to achieve 14:20:47 <EmilienM> it's not super clear to me yet 14:20:53 <EmilienM> when it's done i'll take a look 14:20:58 <florianf> EmilienM: cool, I'll try to summarize it in a comment 14:21:36 <EmilienM> ok 14:21:37 <florianf> There's another tht pathc related to validations that should be pretty uncontroversial and needs another +2 : https://review.openstack.org/#/c/620642/ 14:21:44 <EmilienM> anything else on bugs? 14:21:49 <florianf> (also in tripleo-heat-templates). 14:22:03 <EmilienM> florianf: the gate is in bad shape, not sure we can approve more things 14:22:09 <EmilienM> as you can see in this meeting 14:22:25 <florianf> EmilienM: yes, true. 14:22:58 <EmilienM> alright, moving on 14:22:59 <EmilienM> #topic projects releases or stable backports 14:23:06 <florianf> EmilienM: I'll nudge someone for a review when the situation improves... 14:23:31 <EmilienM> I don't have an update on that this week, we are not releasing stable branches quite often lately, should probably do it before end of year. 14:23:55 <EmilienM> #link https://releases.openstack.org/stein/index.html 14:24:07 <EmilienM> #link https://releases.openstack.org/stein/schedule.html 14:24:16 <openstackgerrit> Nicolas Hicher proposed openstack-infra/tripleo-ci master: testenv-worker: add keystone v3 support https://review.openstack.org/604470 14:24:22 <EmilienM> Stein 2 milestone is January 11 14:24:43 <EmilienM> so less than a months with holidays and such 14:24:55 <EmilienM> moving on 14:24:57 <EmilienM> #topic specs 14:25:01 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:25:10 <EmilienM> anything to discuss about specs? 14:25:15 <jistr> i have one to highlight 14:25:27 <EmilienM> shoot 14:25:28 <jistr> we'd like to include operating system upgrade into major upgrades 14:25:33 <jistr> #link https://review.openstack.org/622324 14:25:42 <jistr> there are several important constraints 14:25:53 <jistr> e.g. deployed server envs which cannot do automated reprovisioning 14:26:11 <jistr> and pacemaker which cannot form a cluster across OS releases, which prevents a rolling upgrade 14:26:23 <jistr> so all in all "you thought FFU was crazy, read this" 14:26:32 <EmilienM> lol 14:26:35 <jistr> but we can't stay on CentOS 7 forever 14:26:42 <EmilienM> can't wait to see FFU + this thing 14:26:50 <jistr> so we need to do this *somehow*, or we need to assume some stance towards the problem, whatever it is :D 14:27:11 <jistr> EmilienM: yea FFU + this will be... :) 14:27:17 <jistr> not sure how to finish that sentence 14:27:27 <jistr> but anyway, something that needs to be addressed 14:27:31 <EmilienM> jistr: thanks for making this specs visibile, I'm pretty sure it would be worth it to announce on the ML too 14:27:43 <jistr> i'll probably send out an e-mail because this affects everyone in charge of any composable service 14:27:56 <jistr> just wanted to give it a shout out here too 14:28:03 <jistr> -- end, thanks for the space -- 14:28:46 <chem> jistr: thanks for the shouting :) 14:28:52 <jistr> (the spec is long but i put effort into structuring it clearly so hopefully it will read ok) 14:29:46 <EmilienM> good, thanks 14:29:55 <EmilienM> anything else on specs? 14:31:05 <EmilienM> #topic open discussion 14:31:28 <EmilienM> is there any feedback / question / comment / anything someone wants to bring here? 14:31:28 <jrist> Would be nice to get some reviews/eyes on some of our automation stuff, if someone has any experience. 14:31:37 <EmilienM> jrist: links 14:31:58 <jrist> https://review.openstack.org/#/c/618563/ 14:32:12 <jrist> https://review.openstack.org/#/q/project:openstack/tempest-tripleo-ui 14:32:20 <jrist> big one https://review.openstack.org/#/c/605424/ 14:32:24 <jrist> selenium infra 14:32:33 <EmilienM> #link UI automation testing needs review https://review.openstack.org/#/c/618563/ https://review.openstack.org/#/q/project:openstack/tempest-tripleo-ui https://review.openstack.org/#/c/605424/ 14:32:36 <jrist> relates to the patch mwhahaha put up last night 14:32:46 <jrist> regarding removing tripleo-ui from gate 14:32:52 <jrist> we are on that path already 14:32:58 <EmilienM> good stuff, more testing! 14:33:01 <jrist> to remove tripleo-ui from gate and get it their own gate 14:33:06 <jrist> EmilienM: +++ 14:33:14 <jrist> eyes would really be appreciated 14:33:21 <jrist> the selenium stuff is all python 14:33:41 <EmilienM> nice, folks interested, please review ^ 14:33:44 <marios_|ruck> ci community call next at https://etherpad.openstack.org/p/tripleo-ci-squad-meeting https://bluejeans.com/4113567798 14:33:49 <EmilienM> anything else for today? 14:34:05 <EmilienM> alright. wish you all a great week and have fun 14:34:07 <EmilienM> #endmeeting