14:00:38 <EmilienM> #startmeeting tripleo
14:00:39 <openstackgerrit> John Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split-controlplane-standalone.yml playbook  https://review.openstack.org/617368
14:00:39 <openstack> Meeting started Tue Dec  4 14:00:38 2018 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:39 <ssbarnea|rover> marios_|ruck: are of tempest-sendmail.tripleo.org:8080 error on gate check? http://logs.openstack.org/24/567224/141/check/tripleo-ci-centos-7-containers-multinode/8888af1/job-output.txt.gz
14:00:39 <jaosorior> yes
14:00:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:42 <openstack> The meeting name has been set to 'tripleo'
14:00:44 <EmilienM> #topic agenda
14:00:45 <Tengu> :)
14:00:48 <arxcruz> o/
14:00:49 <EmilienM> * Review past action items
14:00:50 <jaosorior> o/
14:00:50 <EmilienM> * One off agenda items
14:00:52 <EmilienM> * Squad status
14:00:54 <EmilienM> * Bugs & Blueprints
14:00:55 <florianf> o/
14:00:55 <chandan_kumar> \o/
14:00:56 <EmilienM> * Projects releases or stable backports
14:00:57 * jaosorior half here
14:00:58 <EmilienM> * Specs
14:01:00 <EmilienM> * open discussion
14:01:02 <EmilienM> Anyone can use the #link, #action and #info commands, not just the moderator.
14:01:04 <EmilienM> Welcome TripleOers!
14:01:05 <jfrancoa> o/
14:01:05 <ksambor> o/
14:01:07 <fultonj> o/
14:01:07 <weshay_mtg> 0/
14:01:11 <marios> o/
14:01:11 <jtomasek_> o>
14:01:15 <panda> o/
14:01:17 <jistr> \o/
14:01:18 <rfolco> o/
14:01:26 <chem> o/
14:01:32 <EmilienM> #topic review past action items
14:01:33 <Tengu> EmilienM: Triple-Owlers" maybe? :)
14:01:36 <EmilienM> none, moving on
14:01:41 <EmilienM> #topic one off agenda items
14:01:46 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:01:46 <ccamacho> hi!
14:02:00 <EmilienM> arxcruz: go ahead
14:02:08 <beagles> o/
14:02:09 <arxcruz> So, as some of you guys know, in the future, we want to replace validate-tempest role with os_tempest role
14:02:19 <arxcruz> Right os_tempest role is part of openstack-ansible
14:02:22 <jrist> o/
14:02:33 <arxcruz> right now is a little bit simple, just run tempest and set some configs
14:02:39 <jrist> EmilienM: TripleOwls
14:02:55 <arxcruz> we are working to improve the functionalities there, and once we have everything that it requires to replace validate-tempest, make the move
14:03:05 <arxcruz> right now we add support to python-tempestconf and stackviz
14:03:06 <openstackgerrit> Merged openstack/python-tripleoclient master: Don't exit on RUNNING status on delete  https://review.openstack.org/619583
14:03:07 <openstackgerrit> Merged openstack/tripleo-common master: Bump the Ceph container image to 3.2.0  https://review.openstack.org/619503
14:03:08 <openstackgerrit> Merged openstack/tripleo-heat-templates master: ctlplane pre-alloc IPs - deprecated/non-deprecated role name  https://review.openstack.org/620391
14:03:10 <openstackgerrit> Merged openstack/tripleo-heat-templates master: Add a gateway IP to the Management net in network_data.yaml  https://review.openstack.org/599886
14:03:11 <openstackgerrit> Merged openstack/tripleo-common stable/rocky: Bump the Ceph container image to 3.2.0  https://review.openstack.org/619505
14:03:12 <arxcruz> that are two tools that tripleo uses
14:03:16 <arxcruz> to execute tempest
14:03:31 <arxcruz> this sprint, we are working to improve python-tempestconf options
14:03:44 <openstackgerrit> Bogdan Dobrelya proposed openstack/paunch master: Implement podman rename via re-apply of containers  https://review.openstack.org/621607
14:03:46 <holser_> o/
14:03:47 <arxcruz> and we hope soon we have a POC job running with os_tempest
14:04:10 <arxcruz> help are more than welcome, the openstack-ansible guys are very welcome to collaboration as well
14:04:33 <arxcruz> that's it from my side, i plan to have a status weekly here in the meeting
14:04:35 <arxcruz> :)
14:04:39 <EmilienM> cool, thanks!
14:04:58 <EmilienM> #info arxcruz working on switching tripleo ci to use openstack-ansible-os_tempest status
14:05:06 <EmilienM> let us know if you need help!
14:05:12 <arxcruz> sure :)
14:05:18 <EmilienM> anything else before we move to the next agenda topic?
14:05:35 <EmilienM> #topic Squad status
14:05:37 <arxcruz> nope, unless chandan_kumar or weshay_mtg have something to say
14:05:43 <EmilienM> ci
14:05:44 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting
14:05:46 <EmilienM> upgrade
14:05:48 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:05:50 <EmilienM> containers
14:05:52 <EmilienM> #link https://trello.com/b/S8TmOU0u/tripleo-podman
14:05:54 <EmilienM> edge
14:05:56 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status
14:05:58 <EmilienM> integration
14:06:00 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:06:02 <EmilienM> ui/cli
14:06:04 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
14:06:06 <EmilienM> validations
14:06:09 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:06:10 <chandan_kumar> arxcruz: nope from myside all said already
14:06:10 <EmilienM> networking
14:06:12 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:06:14 <EmilienM> security
14:06:16 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-security-squad
14:06:18 <EmilienM> I hope I didn't miss a squad
14:06:20 <EmilienM> is there any specific info someone wants to communicate on the squads?
14:06:50 <EmilienM> alright, moving on
14:06:53 <EmilienM> #topic bugs & blueprints
14:06:58 <EmilienM> #link https://launchpad.net/tripleo/+milestone/stein-2
14:07:00 <EmilienM> #link https://storyboard.openstack.org/#!/project_group/76
14:07:05 <EmilienM> For Stein we currently have 29 blueprints and about 907 (+167) open Launchpad bugs and 104 stories in Storyboard.
14:07:29 <EmilienM> (not sure why we have that number of bugs since last week but anyway)
14:07:48 <EmilienM> do we have any outstanding bug to report this week?
14:08:24 <marios_|ruck> EmilienM: we file a timeout bug this morning for containers multinode not sure this is the place for that discussion though
14:08:35 <marios_|ruck> https://bugs.launchpad.net/tripleo/+bug/1806632 tripleo-ci-centos-7-containers-multinode job is timing out more often than it passes & blocking gate
14:08:36 <openstack> Launchpad bug 1806632 in tripleo "tripleo-ci-centos-7-containers-multinode job is timing out more often than it passes & blocking gate" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)
14:08:36 <EmilienM> link?
14:08:46 <EmilienM> #link https://bugs.launchpad.net/tripleo/+bug/1806632 timout bug
14:09:00 <marios_|ruck> EmilienM: really the discussion is do we want to make it non voting. otherwise we need to drastically decrease the deployment time
14:09:23 <EmilienM> tripleo-ci-centos-7-containers-multinode is one of our most critical jobs I think, I would avoid to make it non voting
14:09:26 <marios_|ruck> EmilienM: posted the patch to make it non voting but just for this discussion
14:09:34 <EmilienM> have we identified where does it timeout?
14:09:38 <EmilienM> container pulls I see?
14:09:47 <marios_|ruck> EmilienM: right, but we have to do something since it just causes th queue to back up
14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:17 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1806521
14:10:19 <openstack> Launchpad bug 1806521 in tripleo "deployment fails on Ensure system is NTP time synced" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
14:10:19 <EmilienM> merging code without tripleo-ci-centos-7-containers-multinode tested, it's basically merging untested code
14:10:26 <ssbarnea|rover> sorry to interrupt but tempest-sendmail.tripleo.org does not respond on 8080 thus -> https://bugs.launchpad.net/tripleo/+bug/1806699
14:10:27 <openstack> Launchpad bug 1806699 in tripleo "HTTPConnectionPool(host='tempest-sendmail.tripleo.org', port=8080): Max retries exceeded with url" [Undecided,Triaged]
14:10:29 <marios_|ruck> EmilienM: yeah i mean its like 1+ hr undercloud deploy 1+hr for overcloud/controller some info on that bug
14:10:59 <EmilienM> right, this isn't good
14:11:09 <EmilienM> who is working / investigating this one?
14:11:26 <marios_|ruck> EmilienM: agree, as i said posted this https://review.openstack.org/622187 just for the discussion
14:11:55 <EmilienM> can you WIP -1 maybe?
14:11:57 <marios_|ruck> EmilienM: well me and soren started dealing with it today. but this isn't ci 'issue' its more generally about decreasing/optimise the deployment
14:12:00 <marios_|ruck> EmilienM: ack
14:12:01 <openstackgerrit> Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Use a footnote.  https://review.openstack.org/622332
14:12:06 <EmilienM> I would hate having this job non voting before we find a proper root cause
14:12:17 <hjensas> When I do testing in rdocloud I deploy undercloud and build images in parallell via ansible async tasks. afaik we don't build images in CI? But could we pull containers and do ServiceChains in paralell?
14:12:19 <EmilienM> marios_|ruck: I will take a look today at these container pull runtimes
14:13:05 <EmilienM> hjensas: I'm not sure we can easily pull containers without ServiceChains done
14:13:18 <EmilienM> because we use ServiceChains to figure out what containers we need to pull
14:13:58 <hjensas> EmilienM: right, good point.
14:14:07 <EmilienM> anyway, thanks marios_|ruck for bringing it
14:14:14 <EmilienM> why it doesn't have alert and such?
14:14:19 <marios_|ruck> hjensas: but there might be similar optimisations we can explore
14:14:25 <EmilienM> I think it needs to be more visible
14:14:27 <marios_|ruck> EmilienM: sure adding
14:14:42 <EmilienM> anything else bug related?
14:14:50 <marios_|ruck> EmilienM: well cos it isn't consistent fail/timeout i mean it does pass sometime s but fails far too often but adding alert
14:15:02 <jrist> jtomasek_: updated squad status
14:15:28 <jtomasek_> jrist: thx
14:15:38 <EmilienM> alright, moving on
14:15:41 <florianf> yes
14:15:43 <florianf> https://bugs.launchpad.net/tripleo/+bug/1805825
14:15:44 <openstack> Launchpad bug 1805825 in tripleo "tripleo_undercloud_conf_file hiera variable is missing after the instack-undercloud retirement" [Critical,New] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot)
14:16:13 <EmilienM> how can we help?
14:16:44 <florianf> This bug causes some validations to fail since rocky. There's a patch up, but we need some t-h-t expertise to move this forward
14:16:53 <florianf> this is the patch: https://review.openstack.org/#/c/614470/2
14:17:34 <EmilienM> florianf: is it for addressing shardy's comment?
14:18:27 <florianf> Not just this, also about how to dynamically add a bind mount to the mistral_executor container based on the undercloud conf path
14:19:11 <EmilienM> ah ok
14:19:19 <EmilienM> I see, you need "if" etc probably
14:19:52 <EmilienM> florianf: I'll take a look as soon as I can unless someone helps in the review
14:19:58 <florianf> The question is if we (validations squad) should try to dig into how to achieve this with tht or if someone with a little more expertise can push this through a little more quickly (or help)
14:20:21 <florianf> EmilienM: That would be very very awesome...
14:20:43 <EmilienM> florianf: it would help if you could write in your review what you want to achieve
14:20:47 <EmilienM> it's not super clear to me yet
14:20:53 <EmilienM> when it's done i'll take a look
14:20:58 <florianf> EmilienM: cool, I'll try to summarize it in a comment
14:21:36 <EmilienM> ok
14:21:37 <florianf> There's another tht pathc related to validations that should be pretty uncontroversial and needs another +2 : https://review.openstack.org/#/c/620642/
14:21:44 <EmilienM> anything else on bugs?
14:21:49 <florianf> (also in tripleo-heat-templates).
14:22:03 <EmilienM> florianf: the gate is in bad shape, not sure we can approve more things
14:22:09 <EmilienM> as you can see in this meeting
14:22:25 <florianf> EmilienM: yes, true.
14:22:58 <EmilienM> alright, moving on
14:22:59 <EmilienM> #topic projects releases or stable backports
14:23:06 <florianf> EmilienM: I'll nudge someone for a review when the situation improves...
14:23:31 <EmilienM> I don't have an update on that this week, we are not releasing stable branches quite often lately, should probably do it before end of year.
14:23:55 <EmilienM> #link https://releases.openstack.org/stein/index.html
14:24:07 <EmilienM> #link https://releases.openstack.org/stein/schedule.html
14:24:16 <openstackgerrit> Nicolas Hicher proposed openstack-infra/tripleo-ci master: testenv-worker: add keystone v3 support  https://review.openstack.org/604470
14:24:22 <EmilienM> Stein 2 milestone is January 11
14:24:43 <EmilienM> so less than a months with holidays and such
14:24:55 <EmilienM> moving on
14:24:57 <EmilienM> #topic specs
14:25:01 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:25:10 <EmilienM> anything to discuss about specs?
14:25:15 <jistr> i have one to highlight
14:25:27 <EmilienM> shoot
14:25:28 <jistr> we'd like to include operating system upgrade into major upgrades
14:25:33 <jistr> #link https://review.openstack.org/622324
14:25:42 <jistr> there are several important constraints
14:25:53 <jistr> e.g. deployed server envs which cannot do automated reprovisioning
14:26:11 <jistr> and pacemaker which cannot form a cluster across OS releases, which prevents a rolling upgrade
14:26:23 <jistr> so all in all "you thought FFU was crazy, read this"
14:26:32 <EmilienM> lol
14:26:35 <jistr> but we can't stay on CentOS 7 forever
14:26:42 <EmilienM> can't wait to see FFU + this thing
14:26:50 <jistr> so we need to do this *somehow*, or we need to assume some stance towards the problem, whatever it is :D
14:27:11 <jistr> EmilienM: yea FFU + this will be... :)
14:27:17 <jistr> not sure how to finish that sentence
14:27:27 <jistr> but anyway, something that needs to be addressed
14:27:31 <EmilienM> jistr: thanks for making this specs visibile, I'm pretty sure it would be worth it to announce on the ML too
14:27:43 <jistr> i'll probably send out an e-mail because this affects everyone in charge of any composable service
14:27:56 <jistr> just wanted to give it a shout out here too
14:28:03 <jistr> -- end, thanks for the space --
14:28:46 <chem> jistr: thanks for the shouting :)
14:28:52 <jistr> (the spec is long but i put effort into structuring it clearly so hopefully it will read ok)
14:29:46 <EmilienM> good, thanks
14:29:55 <EmilienM> anything else on specs?
14:31:05 <EmilienM> #topic open discussion
14:31:28 <EmilienM> is there any feedback / question / comment / anything someone wants to bring here?
14:31:28 <jrist> Would be nice to get some reviews/eyes on some of our automation stuff, if someone has any experience.
14:31:37 <EmilienM> jrist: links
14:31:58 <jrist> https://review.openstack.org/#/c/618563/
14:32:12 <jrist> https://review.openstack.org/#/q/project:openstack/tempest-tripleo-ui
14:32:20 <jrist> big one https://review.openstack.org/#/c/605424/
14:32:24 <jrist> selenium infra
14:32:33 <EmilienM> #link UI automation testing needs review https://review.openstack.org/#/c/618563/ https://review.openstack.org/#/q/project:openstack/tempest-tripleo-ui https://review.openstack.org/#/c/605424/
14:32:36 <jrist> relates to the patch mwhahaha put up last night
14:32:46 <jrist> regarding removing tripleo-ui from gate
14:32:52 <jrist> we are on that path already
14:32:58 <EmilienM> good stuff, more testing!
14:33:01 <jrist> to remove tripleo-ui from gate and get it their own gate
14:33:06 <jrist> EmilienM: +++
14:33:14 <jrist> eyes would really be appreciated
14:33:21 <jrist> the selenium stuff is all python
14:33:41 <EmilienM> nice, folks interested, please review ^
14:33:44 <marios_|ruck> ci community call next at https://etherpad.openstack.org/p/tripleo-ci-squad-meeting https://bluejeans.com/4113567798
14:33:49 <EmilienM> anything else for today?
14:34:05 <EmilienM> alright. wish you all a great week and have fun
14:34:07 <EmilienM> #endmeeting