14:08:09 <jaosorior> #startmeeting tripleo 14:08:10 <openstack> Meeting started Tue Nov 6 14:08:09 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:08:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:08:11 <jaosorior> #topic agenda 14:08:13 <openstack> The meeting name has been set to 'tripleo' 14:08:13 <jaosorior> * Review past action items 14:08:15 <jaosorior> * One off agenda items 14:08:17 <jaosorior> * Squad status 14:08:19 <jaosorior> * Bugs & Blueprints 14:08:21 <jaosorior> * Projects releases or stable backports 14:08:23 <jaosorior> * Specs 14:08:25 <jaosorior> * open discussion 14:08:27 <jaosorior> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:08:43 <jrist> o/ 14:08:51 <weshay> \0 14:08:54 <fultonj> o/ 14:08:57 <d0ugal> o/ 14:09:06 <rlandy> o/ 14:09:14 <ksambor> o/ 14:09:27 <arxcruz> o/ 14:09:29 <shardy> o/ 14:09:36 <hjensas> o/ 14:09:56 <mwhahaha> hi2u 14:09:57 <marios> o/ 14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1801769 14:10:18 <openstack> Launchpad bug 1801769 in tripleo "FailedValidation: Hostname is not fully qualified, fails undercloud install" [Critical,Triaged] 14:11:29 <jaosorior> #topic review past action items 14:11:41 <jaosorior> There was one: 14:11:43 <jaosorior> jaosorior to poke the squads and check status. Based on this we might need to update the list of active squads. 14:12:00 <jaosorior> I didn poke all the squads, so, I'll leave this action active until I do. 14:12:05 <jaosorior> at least we did get some updates on some of them 14:12:49 <jaosorior> #topic one off agenda items 14:12:50 <jtomasek> o/ 14:13:01 <jaosorior> First of them: (sshnaidm) New workflow for having master containers updated with tripleo packages from delorean-current (consistent) repo 14:13:13 <quiquell> o/ 14:13:17 <sshnaidm|rover> o/ 14:13:27 <dpeacock> oh crap this is at 09:00 now 14:13:28 <honza> o/ 14:13:29 <dpeacock> o/ 14:13:32 <matbu> o/ 14:13:34 <chem> o/ 14:13:44 <jrist> jaosorior: jtomasek or I will update squad for UI. still active, but busy with releasing 14:13:51 <jaosorior> jrist: thanks! 14:14:33 <trozet> weshay: is there anyway you can restore that queens undercloud.qcow2? our opnfv release is Friday 14:14:49 <sshnaidm|rover> jaosorior, so we have delorean-current repos for tripleo-* packages that we use in our CI jobs, it's the newest packages. And I suppose we need to have the same in containers to avoid all these long updates 14:14:58 <gfidente> o/ 14:15:10 <trozet> weshay: its fine to not have it for rocky and later, but since undercloud isnt containerized for our queens deployments we kind of rely on that image 14:15:11 <owalsh> o/ 14:15:37 <rfolco|ruck> o/ 14:15:53 <sshnaidm|rover> jaosorior, so after a time period or after every tripleo-* repo package is built we need to update all our containers with it and to publish on docker. 14:16:48 <mwhahaha> sshnaidm|rover: we might be able to leverage the container CI job I'm working on to do a periodic update 14:17:06 <sshnaidm|rover> mwhahaha, periodic update to all containers? 14:17:54 <mwhahaha> #link https://review.openstack.org/#/c/614392/ 14:18:01 <weshay> trozet, see pm #tripleo meeting atm man 14:18:03 <sshnaidm|rover> mwhahaha, as we saw in jobs the most time is wasted on containers updates from delorean-current repo, it's about 40 min - 1 hour, which causes timeouts 14:18:06 <mwhahaha> sshnaidm|rover: yea 14:18:43 <sshnaidm|rover> mwhahaha, so if it's periodic update of all containers - it's actually what I propose :) 14:19:00 <mwhahaha> sshnaidm|rover: it currently isnt but could be adopted to support that 14:19:22 <mwhahaha> sshnaidm|rover: btw container builds are broken due to grafana 14:19:28 <mwhahaha> It looks like 14:19:33 * mwhahaha afk for a bit 14:19:37 <sshnaidm|rover> mwhahaha, not sure I understand 14:19:49 <sshnaidm|rover> mwhahaha, ok, we can take it offline and discuss later 14:21:08 <sshnaidm|rover> that's it for this point from me, if anyone has comments.. 14:21:21 <jaosorior> sshnaidm|rover: is there any work you hve done to enable this? or were you looking for input first? 14:21:58 <sshnaidm|rover> jaosorior, not done yet, in planning stage 14:22:08 <sshnaidm|rover> and looking for input, yes 14:22:13 <hjensas> how much time do we spend on updating containers when packages are up-to-date in the containers? 14:22:26 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: (Partial) Revert "Use container_cli for post_upgrade_tasks & external_upgrade_tasks" https://review.openstack.org/615891 14:22:34 <hkominos> fultonj: ping 14:22:34 <sshnaidm|rover> hjensas, it will take time only to install a built package from patch 14:22:59 <sshnaidm|rover> hjensas, I think overall it sholdn't take more than 10 mins 14:23:27 <fultonj> hkominos: there's a tripleo meeting happening now; can you ping me after? 14:23:49 <hjensas> sshnaidm|rover: ok, thanks. 14:23:52 <sshnaidm|rover> hjensas, we can see it right after a promotion when all packages are up to date 14:24:15 <hkominos> fultonj: sure :) take care 14:24:28 <jaosorior> sshnaidm|rover: I think it makes sense, though I guess first you wanna sync with mwhahaha to get more input, right? 14:24:39 <sshnaidm|rover> jaosorior, yep 14:24:54 <sshnaidm|rover> jaosorior, need to sync if he's working on this too 14:25:13 <jaosorior> got it 14:25:27 <jaosorior> so, the next topic is: (sshnaidm) OVB check jobs turning off in Wed 14:25:46 <sshnaidm|rover> ok, so we actually decided to turn them off even today 14:25:52 <sshnaidm|rover> #link https://review.rdoproject.org/r/#/c/17272/ 14:26:05 <sshnaidm|rover> we turn off check jobs for patches 14:26:07 <openstackgerrit> Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM: Squash for fedora28 job https://review.openstack.org/615532 14:26:15 <sshnaidm|rover> periodic jobs will still run 14:26:22 <dpeacock> :q! 14:26:35 <sshnaidm|rover> and if you want to run ovb jobs on your patch, you can run them by comment "check-rdo" 14:26:38 <openstackgerrit> Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28 https://review.openstack.org/615479 14:26:45 <sshnaidm|rover> but nothing will run automatically 14:26:54 <jaosorior> sshnaidm|rover: what's the background on that? 14:27:25 <weshay> jaosorior, the background reason why we need to shut off 3rd party check? 14:27:27 <sshnaidm|rover> jaosorior, RDO cloud is not stable for last months 14:27:43 <jaosorior> right 14:28:15 <weshay> jaosorior, the pass rate is roughly 15%, lots of broken glass not providing value. The root cause seems to be networking and the load on rdo-cloud atm 14:28:21 <jaosorior> thought the instability would be addressed, and not that we would be moving away from testing our patches there. 14:28:39 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman https://review.openstack.org/606975 14:28:47 <weshay> so we're hoping to at least get full promotions running, when 50 red / broken check jobs are not currently running 14:29:01 <weshay> jaosorior, yes.. I also thought this would be addressed by now 14:29:26 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman https://review.openstack.org/606975 14:29:27 <jaosorior> when/if these issues in RDO cloud get addressed, is the plan to turn the checks back on? 14:29:40 <weshay> jaosorior, yes, of course.. 14:29:50 <weshay> and it could be we use another cloud for some time.. not sure 14:30:15 <weshay> jaosorior, I am trying coordinate a cross team response to the issue here 14:30:16 <jaosorior> alright 14:30:30 <jaosorior> weshay: any way I could help there? 14:30:34 <weshay> https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https 14:30:36 <weshay> ://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tr 14:30:36 <weshay> ee.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.ta 14:30:40 <weshay> iga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.i 14:30:48 <weshay> o/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/pro 14:30:52 <weshay> ject/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/ 14:30:57 <weshay> tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripl 14:31:01 <openstackgerrit> Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28 https://review.openstack.org/615479 14:31:06 <weshay> eo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci 14:31:11 <weshay> -board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-boar 14:31:13 <jaosorior> weshay: none of the links worked for me :/ 14:31:18 <weshay> d/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298vvvvvvvvvvv 14:31:23 <weshay> oh wow 14:31:25 <weshay> lolz.. my laptop just had a brain fart 14:31:33 <weshay> https://tree.taiga.io/project/tripleo-ci-board/epic/298 14:31:45 <weshay> jaosorior, ur on the epic 14:31:57 <jaosorior> ohoo, THAT epic :D 14:31:57 <weshay> also escalated https://trello.com/c/2k9DDzpX/778-cixlp1797526tripleociproa-bring-ovb-jobs-back-to-passing-at-75 14:32:14 <weshay> jaosorior, the communication on this has been very odd.. folks were trying to move that issue to preventative action 14:32:46 <sshnaidm|rover> jaosorior, just how looks rdo cloud ci: http://dashboard-ci.tripleo.org/dashboard/snapshot/spgOhTI6f9QG3CyyRu38rzfpIAg48LZE?orgId=1&from=now-90d&to=now 14:32:52 <jaosorior> weshay: lets catch up on this after this meeting 14:32:58 <weshay> jaosorior, mwhahaha and I should coordinate prior to a prod-chain infra meeting 14:33:00 <weshay> jaosorior, k 14:33:11 <jaosorior> thanks for the info 14:33:15 <jaosorior> sshnaidm|rover: ouch 14:33:33 <sshnaidm|rover> yep 14:34:11 <jaosorior> So, the next topic is mine 14:34:16 <jaosorior> * No meeting next week 14:34:32 <jaosorior> So, a bunch of folks will be in Berlin in the Summit next week, so will I 14:34:53 <jaosorior> so, we won't have a meeting next week 14:35:16 <jaosorior> so, if you're going, hope to see you folks there :D 14:35:48 <jaosorior> that's all for the one-off agenda items 14:36:25 <jaosorior> #topic Squad status 14:36:27 <jaosorior> ci 14:36:29 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:36:31 <jaosorior> upgrade 14:36:33 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:36:35 <jaosorior> containers 14:36:37 <jaosorior> #link https://trello.com/b/S8TmOU0u/tripleo-podman 14:36:39 <jaosorior> integration 14:36:41 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:36:43 <jaosorior> ui/cli 14:36:45 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:36:47 <jaosorior> validations 14:36:49 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:36:51 <jaosorior> networking 14:36:53 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:36:55 <jaosorior> security 14:36:57 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-security-squad 14:36:59 <jaosorior> edge 14:37:01 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:37:14 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman https://review.openstack.org/606975 14:37:32 <jaosorior> Any squad wanting to bring up their status, or a topic for the general public? 14:38:14 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow the environment to enable Podman on the overcloud https://review.openstack.org/612522 14:38:19 <openstackgerrit> Merged openstack-infra/tripleo-ci master: turn on tcpdump debug by default for ovb jobs https://review.openstack.org/610072 14:38:20 <openstackgerrit> Merged openstack-infra/tripleo-ci master: Collect atop statistics https://review.openstack.org/615515 14:39:05 <EmilienM> o/ 14:39:16 <jaosorior> \o 14:39:54 <jaosorior> #topic bugs & blueprints 14:39:56 <jaosorior> #link https://launchpad.net/tripleo/+milestone/stein-1 14:39:58 <jaosorior> For Stein we currently have 28 (for stein-1) and 4 (for stein-2) blueprints open in Launchpad. 14:40:01 <jaosorior> Bugs: 757 (-4) stein-2, 1 stein-3. 102 (-1) open Storyboard bugs. 14:40:02 <openstack> bug 757 in Launchpad itself "Assignee should be CC'd when assigneed to a private bug" [Medium,Invalid] https://launchpad.net/bugs/757 14:40:02 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76 14:40:36 <EmilienM> lol the bot 14:41:33 <jaosorior> #topic projects releases or stable backports 14:42:47 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Set setype for log and persistant data directories https://review.openstack.org/611801 14:42:49 <jaosorior> nothing here on my side 14:42:58 <jaosorior> #topic specs 14:43:00 <jaosorior> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:43:09 <jaosorior> Any specs that folks would like to bring up here? 14:44:12 <hjensas> #link https://review.openstack.org/451584 14:44:39 <jaosorior> I'll check it out, thanks hjensas ! 14:44:51 <hjensas> Most of the code to implement is up for review ... so would be good to get that done. :) 14:45:21 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Reimplement HA docker tagging https://review.openstack.org/614827 14:45:22 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Support HA container tagging with podman https://review.openstack.org/614830 14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman https://review.openstack.org/606975 14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Set setype for log and persistant data directories https://review.openstack.org/611801 14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow the environment to enable Podman on the overcloud https://review.openstack.org/612522 14:46:32 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Reimplement HA docker tagging https://review.openstack.org/614827 14:46:41 <jaosorior> #topic open discussion 14:46:43 <jaosorior> Anything else that folks want to bring up to the meeting? 14:47:15 <honza> #link https://review.openstack.org/#/c/607323/ 14:47:20 <openstackgerrit> Yurii Prokulevych proposed openstack/tripleo-upgrade master: Drop ssh-user from 'upgrade run' template. https://review.openstack.org/615897 14:47:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud https://review.openstack.org/612526 14:47:23 <honza> Need some eyes/help on that patch 14:48:42 <jaosorior> ack, added to my queue 14:48:59 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud https://review.openstack.org/612526 14:49:16 <honza> <3 14:50:29 <jaosorior> alright! thanks for joining folks! 14:50:31 <jaosorior> #endmeeting