14:00:15 <jaosorior> #startmeeting tripleo 14:00:16 <openstack> Meeting started Tue Sep 4 14:00:15 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:17 <jaosorior> #topic agenda 14:00:19 <jaosorior> * Review past action items 14:00:20 <openstack> The meeting name has been set to 'tripleo' 14:00:21 <jaosorior> * One off agenda items 14:00:23 <jaosorior> * Squad status 14:00:25 <jaosorior> * Bugs & Blueprints 14:00:27 <jaosorior> * Projects releases or stable backports 14:00:29 <jaosorior> * Specs 14:00:31 <jaosorior> * open discussion 14:00:33 <jaosorior> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:35 <jaosorior> Hey folks! who's around? 14:00:38 <mwhahaha> ahoy 14:00:48 <jaosorior> \o 14:00:53 <fultonj> o/ 14:00:54 <jbadiapa> o/ 14:00:54 <ksambor> o/ 14:00:58 <jrist> o/ 14:01:01 <dpeacock> o/ 14:01:02 <marios> o/ 14:01:13 <owalsh> o/ 14:01:20 <Tengu> \o/ 14:01:36 <slagle> hi 14:01:47 <florianf> o/ 14:01:52 <holser_> o/ 14:02:00 <chem> o/ 14:02:00 <shardy> o/ 14:02:13 <beagles> o/ 14:03:14 <jaosorior> #topic review past action items 14:03:23 <jaosorior> there was one: jaosorior to update the wiki and tripleo docs to refer to the squads and their respective etherpads 14:03:41 <jaosorior> So... I updated the wiki to mention the squads 14:03:51 <jaosorior> #link https://wiki.openstack.org/wiki/Meetings/TripleO 14:04:07 <jaosorior> And added the squads to your docs too: 14:04:09 <jaosorior> #link http://tripleo.org/contributor/index.html#squads 14:04:15 <EmilienM> o/ 14:04:49 <matbu> o/ 14:04:53 <jaosorior> And that's all on that action. 14:05:04 <jaosorior> #topic one off agenda items 14:05:09 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:05:15 <ccamacho> hey folks! o/ 14:05:34 <rascasoft> o/ 14:05:35 <jaosorior> So, due to a lot of folks being in the PTG, there will be no meeting next week 14:06:01 <jaosorior> I'll send a reminder of this to the mailing list 14:06:15 <jaosorior> #topic PTG agenda 14:06:23 <jaosorior> There is a proposed schedule for the PTG: 14:06:29 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ptg-stein 14:06:42 <jaosorior> but there is still time for folks to request changes and topics. So, please do. 14:07:09 <EmilienM> thanks jaosorior for bringing up an agenda ! 14:07:51 <Tengu> jaosorior: one note: if we're to do a presentation about the validation framework, I'll be remote from EMEA 14:08:15 <shardy> thanks jaosorior this is looking good 14:08:44 <jaosorior> Tengu: we can make sure to enable a bluejeans session so you can participate 14:09:05 <Tengu> jaosorior: securing that would be great - if possible at a nice emea-compatible time :) 14:09:08 <jaosorior> I believe such a bridge was available last year. EmilienM was it just your bridge that was put up for folks to join? 14:09:21 <EmilienM> yeah 14:09:27 <EmilienM> I'll take care of that no worries 14:09:35 <jaosorior> thanks EmilienM 14:09:53 <jbadiapa> If anyone can take a conference call micro it would be great 14:09:59 <Tengu> ah, it's the UX/UI part. just found the schedule. 14:10:14 <jaosorior> Tengu: I believe your topic is schedules for thursday in the evening, so it would be morning for oyu. 14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:19 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1786764 14:10:20 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1787910 14:10:21 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1789898 14:10:21 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1790199 14:10:21 <openstack> Launchpad bug 1786764 in tripleo "Wrong versions of tripleo-common in container images updated in CI" [Critical,Triaged] - Assigned to Sorin Sbarnea (ssbarnea) 14:10:22 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1790489 14:10:22 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1790490 14:10:23 <openstack> Launchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) 14:10:24 <openstack> Launchpad bug 1789898 in tripleo "check failing due to missing python-tripleoclient" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) 14:10:25 <openstack> Launchpad bug 1790199 in tripleo "fs035: openstack overcloud deploy returns error code without any signs or error" [Critical,Triaged] 14:10:27 <openstack> Launchpad bug 1790489 in tripleo "ERROR heat.engine.resource ResourceFailure: Error: resources.ControllerDeployment_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6" [Critical,New] 14:10:28 <openstack> Launchpad bug 1790490 in tripleo "failure with collect-logs : Verify Sphinx build" [Critical,Triaged] 14:10:29 <Tengu> jaosorior: more or less yep. 14:10:54 <jaosorior> #topic Logging 14:10:59 <jaosorior> jbadiapa this one is yours 14:11:24 <jbadiapa> jaosorior, thanks 14:11:37 <jbadiapa> tripleo logging status is not in a very good status 14:11:46 <florianf> mwhahaha: thanks for the quick review! 14:12:01 <jbadiapa> there are several services that are not properly configured 14:12:33 <jbadiapa> so, I would like to enable fluentd on all the scenarios if that is possible 14:12:53 <mwhahaha> jbadiapa: So I'm not sure we should enable it on all scenarios 14:13:06 <mwhahaha> i dont' think we want to get into a place where fluentd suddenly breaks everythign 14:13:15 <Tengu> jbadiapa: is fluentd going to last? I think I heard that one might be dropped in favor of something else. 14:13:23 <mwhahaha> additionally we don't have a native way to configure fluentd endpoint 14:13:42 <mwhahaha> i think this is a larger issue without having a complete logging solution that is managable via tripleo 14:13:59 <jbadiapa> there are couple of things here 14:14:00 <mwhahaha> that includes both agent and server 14:14:18 <Tengu> we should maybe use container standard logging with journald+rsyslog... 14:14:42 <jbadiapa> first, no matter where we going we need to have a proper logging solution, currently the solution is fluentd 14:15:35 <jbadiapa> but it is not ok at all, I meant there are several services that are not currently configured 14:15:51 <jbadiapa> and others they are not configured at all 14:16:19 <jbadiapa> BTW, enable fluentd would be temporally 14:16:24 <mwhahaha> jbadiapa: do we have any documentation around what is required to properly integrate a service with fluentd? 14:16:57 <jbadiapa> mwhahaha, I dont think so 14:16:58 <mwhahaha> if it' snot permanent, it's just going to break again and even just enabling it doesn't mean we have testing 14:17:12 <mwhahaha> jbadiapa: so we need to resolve that first if we expect folks to integrate with fluentd 14:19:08 <jbadiapa> ok 14:19:44 <openstackgerrit> Merged openstack/tripleo-heat-templates master: Add reflection of RpcPort to health checks https://review.openstack.org/583629 14:19:45 <openstackgerrit> Merged openstack/python-tripleoclient stable/rocky: Use tripleo-admin user for minor updates https://review.openstack.org/599364 14:20:36 <mwhahaha> jbadiapa: so we need fluentd integration docs, need a way to configure/deploy fluentd somewhere (undercloud?) in order to have the scenarios log to a central place, and then we'd need to update the CI framework to support pulling logs from this central place rather than each node 14:20:36 <jbadiapa> I agree that we need to doc how everything should be configured and what we expect form the different services 14:21:05 <mwhahaha> IMHO, a centralized logging on the undercloud could improve the overall CI logging but it's not a small undertaking 14:21:24 <jaosorior> jbadiapa: what's the plan for Stein? is it still to use fluentd or are we switching to rsyslog? 14:21:27 <shardy> how would that work if we go to all-in-one for the scenario jobs? 14:21:58 <mwhahaha> jbadiapa: can you fluentd to localhost? 14:23:06 <jbadiapa> jaosorior, the idea is adding rsyslog for stein and deprecate fluentd on the next release 14:23:22 <shardy> I guess we can work that out, I just wondered given the PTG topic of reworking the scenario jobs to use only one node 14:24:13 <jaosorior> shardy: I guess it shouldn't be too problematic if the server that the services are sending logs to is in the same host. 14:24:31 <jbadiapa> shardy, It would do but I havent tested TBH 14:24:52 <shardy> Ok cool, something to keep in mind I guess 14:25:33 <jbadiapa> mwhahaha, sorry but I'm not sure what you meant 14:26:03 <mwhahaha> jbadiapa: if you have a single node, can you configure the services to send to a local fluentd instance? 14:26:19 * mwhahaha isn't that familar with the general configuration expected with fluentd 14:26:52 <openstackgerrit> Sorin Sbarnea proposed openstack/tripleo-quickstart master: Avoid silent parsing failure of provisioned.csv https://review.openstack.org/595795 14:27:39 <jbadiapa> mwhahaha, you don't need to send the logs to nowhere, currently the gate is not sending the log anywhere 14:28:01 <mwhahaha> it wouldn't be nowhere, it would be localhost 14:28:07 <mwhahaha> anyway i guess that's a quesiton for another time 14:29:57 <jaosorior> jbadiapa: anything else you wanna bring up on this topic? 14:30:27 <jbadiapa> well, I guess that enable fluentd on all scenarios it not accepted, but at least it's necessary on the scenario we are currently working 14:31:22 <jaosorior> what scenario is that? 14:32:21 <jbadiapa> It depends on the service 14:33:24 <jbadiapa> and then disable it, otherwise is impossible to be checked on the gate 14:33:49 <jbadiapa> we can move on, I'll try to explain it on the mailing list 14:33:54 <jaosorior> understood, gotta test it out somehow. Is using depends-on enough? 14:33:56 <jaosorior> lets do that 14:34:08 <jaosorior> #topic Rocky RC2 14:34:11 <jaosorior> mwhahaha: ^^ 14:34:12 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/rocky-rc2 14:34:23 <mwhahaha> so we haven't yet released RC2, we had some CI issues last week and such 14:34:36 <mwhahaha> Hopefully this week we can land all the bits, i k now we had a few critical open bugs 14:35:04 <mwhahaha> I just went through and moved a bunch of bugs that were < Critical but targeted to RC2. Please make sure to target any new stuff to stein-1 unless it's specific to the RC 14:35:31 <mwhahaha> I'm going to go through the bugs and kickout anything not specifically rocky targeted later today 14:35:51 <EmilienM> https://review.openstack.org/#/c/598167/ is imho a blocker 14:35:53 <mwhahaha> Please let us know if there are any major Rocky related items otherwise we'll cut RC2 and move on 14:35:56 <EmilienM> I couldn't deploy rocky without it 14:36:10 <Tengu> +1 14:36:11 <EmilienM> yatin -1 it but we need to approve it asap 14:36:18 <EmilienM> owalsh: ^ you ok? 14:36:20 <mwhahaha> Can we get some nova/ironic folks to eyeball it? 14:36:24 <Tengu> can confirm, same issue on my side 14:36:28 <mwhahaha> It seems like a placement regression or something 14:36:31 <mwhahaha> if it's only deprecated 14:36:47 <Tengu> mwhahaha: I've added a comment in order to explain what I had on rocky 14:36:49 <mwhahaha> I checked and the referenced patch is not in stable/rocky 14:37:06 <Tengu> yeah, it's on master 14:38:00 <owalsh> EmilienM: https://review.openstack.org/598167 is required in rocky? 14:38:08 <openstackgerrit> Martin André proposed openstack/tripleo-heat-templates master: WIP Refactor openshift services for composable roles https://review.openstack.org/599618 14:38:33 <Tengu> owalsh: apparently yes. I can't deploy my lab without it because the filters don't properly detect RAM and Disk amount 14:38:37 <EmilienM> owalsh: well for me it was required 14:38:47 <EmilienM> I couldn't deploy (this morning) my overcloud 14:39:05 <Tengu> there's some trickery in there - Ram is detected as 0MB, same for Disk.... 14:39:41 <Tengu> and as it requires at least 4GB for the ram (and some more disk than the 0MB detected), it prevents deploy :(. 14:40:20 <mwhahaha> so sounds like we need to track down that issue 14:40:26 <openstackgerrit> Derek Higgins proposed openstack-infra/tripleo-ci master: [WIP] Testing ansible-ml2 job https://review.openstack.org/582294 14:40:27 <mwhahaha> yes i agree if it's real that's a blocker for rc2 14:40:38 <mwhahaha> anyway that's it on the RC2 front, let's make sure rocky works :D 14:40:51 <jaosorior> thanks mwhahaha 14:41:08 <jaosorior> #topic FYI: Blueprint to use newer ceph versions during Stein (fultonj) 14:41:16 <jaosorior> fultonj: ^^ 14:41:35 <fultonj> jaosorior: yes, so we have a blueprint 14:41:42 <fultonj> to test newer versions of ceph during stein 14:41:49 <owalsh> EmilienM: didn't think it was necessary, the nova patch is on master only. Either way +1 from me. 14:41:50 <fultonj> wanted to let people know the plan via the blueprint 14:42:04 <fultonj> jaosorior: and wanted you to ack 14:42:21 <fultonj> probably doesn't need a full discussion at PTG 14:42:26 <fultonj> so i bring it up here 14:42:29 <jaosorior> #link https://blueprints.launchpad.net/tripleo/+spec/ceph-mimic-nautilus 14:42:36 <jaosorior> thanks fultonj 14:42:49 <jaosorior> sounds good 14:42:53 <fultonj> thanks 14:43:22 <jaosorior> #topic Squad status 14:43:24 <jaosorior> ci 14:43:26 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:43:28 <jaosorior> upgrade 14:43:30 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:43:32 <jaosorior> containers 14:43:34 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-containers-squad-status 14:43:36 <jaosorior> integration 14:43:38 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:43:40 <jaosorior> ui/cli 14:43:42 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:43:44 <jaosorior> validations 14:43:46 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:43:48 <jaosorior> networking 14:43:50 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:43:52 <jaosorior> workflows 14:43:54 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status 14:43:56 <jaosorior> security 14:43:58 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-security-squad 14:44:00 <jaosorior> edge 14:44:02 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:44:27 <jaosorior> Any updates from any squad? 14:45:00 <Tengu> there's still the validation framework spec to validate :) 14:45:22 <Tengu> there's an open discussion regarding some wording/features, would be great to get that sorted out. 14:45:36 <Tengu> (so linked to the validation team I guess mostly) 14:46:12 <jaosorior> #topic bugs & blueprints 14:46:14 <jaosorior> #link https://launchpad.net/tripleo/+milestone/rocky-rc2 14:46:16 <jaosorior> #link https://launchpad.net/tripleo/+milestone/stein-1 14:46:18 <jaosorior> For Rocky we currently have 1 blueprints and about 36(+23) open Launchpad bugs. 14:46:20 <jaosorior> For Stein we currently have 27 blueprints open in Launchpad. 14:46:22 <jaosorior> Bugs: 36 rocky-rc1, 740 (+26) stein-1. 103 (-3) open Storyboard bugs. 14:46:24 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76 14:46:26 <jaosorior> Tengu: ack, gotta re-read your validations spec. 14:46:34 <Tengu> jaosorior: thanks :) 14:47:07 <openstackgerrit> Sorin Sbarnea proposed openstack/tripleo-quickstart master: Remove duplicate dictionary keys https://review.openstack.org/597547 14:48:26 <jaosorior> #topic projects releases or stable backports 14:48:40 <jaosorior> I guess this was already covered by mwhahaha's topic. 14:49:05 <jaosorior> #topic specs 14:49:07 <jaosorior> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:49:18 <jaosorior> any other specs that need reviews besides Tengu's ? 14:49:29 <jaosorior> either way, please review the open ones in the link provided above ^^ 14:50:42 <jaosorior> #topic open discussion 14:50:48 <jaosorior> Anything else folks want to bring up to the meeting? 14:52:38 <openstackgerrit> Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Avoid fatal task execution when falling back to QEMU https://review.openstack.org/597523 14:54:14 <jaosorior> alright folks! 14:54:16 <jaosorior> thanks for joining 14:54:22 <openstackgerrit> Merged openstack/python-tripleoclient master: Add fixtures instead mocks https://review.openstack.org/581342 14:54:30 <jaosorior> see you at the PTG, else, in two weeks 14:54:33 <jaosorior> #endmeeting