14:01:09 <jaosorior> #startmeeting TripleO 14:01:09 <openstack> Meeting started Tue Jan 22 14:01:09 2019 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:13 <openstack> The meeting name has been set to 'tripleo' 14:01:31 <jaosorior> #topic agenda 14:01:33 <jaosorior> * Review past action items 14:01:35 <jaosorior> * One off agenda items 14:01:37 <jaosorior> * Squad status 14:01:39 <jaosorior> * Bugs & Blueprints 14:01:41 <jaosorior> * Projects releases or stable backports 14:01:43 <jaosorior> * Specs 14:01:44 <EmilienM> o/ 14:01:45 <jaosorior> * open discussion 14:01:47 <jaosorior> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:01:49 <jaosorior> Hey folks! who's around? 14:01:55 <bandini> jaosorior: I'll try and check that lp bug later 14:01:58 <beagles> o/ 14:02:07 <fultonj> o/ 14:02:09 <jaosorior> bandini: thanks 14:02:15 <chandankumar> \o/ 14:02:16 <bogdando> hi 14:02:18 <marios> o/ 14:02:19 <matbu> o/ 14:02:28 <chem> o/ 14:02:59 <ksambor> o/ 14:03:32 <d0ugal> o/ 14:04:04 <Tengu> «o/ 14:04:08 <jbadiapa> o/ 14:04:27 <openstackgerrit> Matthias Runge proposed openstack/tripleo-heat-templates stable/rocky: Enable ovs-stats by default when using ovs https://review.openstack.org/632471 14:04:40 <jaosorior> #topic review past action items 14:04:41 <jaosorior> None. 14:04:42 <holser_> o/ 14:04:51 <jaosorior> #topic one off agenda items 14:04:53 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:05:02 <jaosorior> First topic: (gcerami) make fedora 28 with centos 7 containers job voting 14:05:36 <jaosorior> panda: ^^ 14:06:01 <toure> o/ 14:07:42 <jaosorior> panda or anybody else from the CI team that can comment on that? ^^ 14:07:47 <panda> yeah sorry 14:07:49 <panda> got sidetracked 14:08:14 <panda> so, the job has proven to have valuable informations on the interaction between tripleo and python 3 14:08:17 <jrist> o/ 14:08:29 <panda> we had some legit failure on the periodic runs 14:08:37 <panda> and we are fixing actual bugs, 14:08:59 <ccamacho> jelou! 14:09:07 <panda> so I would like to understand if it's ok for us to start blocking patchesd base on the result of that jo 14:09:18 <panda> b 14:10:01 <fultonj> panda: seems reasonable to me (fwiw) 14:10:04 <panda> we are talking about the fedora 28 job taht uses centos 7 containers 14:10:32 <panda> bugs like this for example https://bugs.launchpad.net/tripleo/+bug/1812837 14:10:33 <openstack> Launchpad bug 1812837 in tripleo "periodic fedora 28 job failing with "/bin/sh: line 1: exit: null: numeric argument required" in Run async deployment StandalonePostDeployment step" [Undecided,Triaged] 14:11:10 <ykarel> panda, but for catching these we need to run fedora jobs in distgit changes also ^^ 14:11:47 <panda> ykarel: are we ready to do it ? 14:12:20 <ykarel> panda, we started with testing centos standalone, didn't checked it can cover fedora too yet 14:13:19 <jaosorior> panda: should we enable it for the distgit changes before enabling it for every patch here? 14:13:42 <ykarel> overall for catching other issues that happened recently voting is must in upstream, and for particual issues like 1812837 we need in distgit 14:13:44 <panda> we also hit this one https://bugs.launchpad.net/tripleo/+bug/1812632 14:13:46 <openstack> Launchpad bug 1812632 in tripleo "Overcloud nova compute docker command failing on nova_cellv2_discover_hosts" [Critical,Fix released] - Assigned to Martin Schuppert (mschuppert) 14:13:55 <ykarel> yes mentioning ^^ only 14:15:08 <panda> ykarel: what you think, can this involves distgit too ? ^ 14:15:09 <ykarel> breakages in distgit changes are rare as compared to project changes 14:15:50 <ykarel> panda, 181263 is caused by tht change 14:16:01 <ykarel> fedora was non-voting, but no one noticed 14:16:07 <panda> ykarel: ok, so we need the job voting there 14:16:11 <ykarel> even that patch caused fs035 also 14:16:37 <ykarel> panda, yes, also it's good if reviewers not ignore non-voting jobs 14:16:43 <ykarel> results 14:16:48 <jaosorior> tox 14:16:58 <panda> jaosorior: so yeah, not only distgit 14:17:15 <panda> jaosorior: ERROR: toxini file 'tox.ini' not found 14:17:17 <jaosorior> panda: if we have started catching bugs with it. And we have the capacity, then yeah 14:17:22 <jaosorior> panda: lol 14:17:34 <jaosorior> panda: do we have the capacity to add that job? 14:17:49 <panda> jaosorior: it's standalone, we'll make the capacity 14:17:57 <ssbarnea|bkp2> jaosorior: i think that if we make f28 voting we will much better, as we already have to take care of these jobs. it would save time 14:18:25 <jaosorior> understood 14:18:45 <jaosorior> panda: the job is passing right now, right? 14:19:17 <panda> jaosorior: no, but it hits legit bugs. 14:19:46 <jaosorior> panda: the suggestion is still to add it as non-voting initially, right? and then move it to voting? 14:20:40 <panda> jaosorior: the job is already there non-voting, it has been for at least a month. It was quite resilient even if the hashes between fedora and centos repos where not aligned 14:21:10 <panda> jaosorior: no it's in periodic too, and we are basing promotion on that result 14:21:16 <panda> jaosorior: on both fedora and centos side 14:22:44 <jaosorior> alright, if the job has been stable, it's working, and it's legitimally catching bugs; lets block based on it. 14:23:26 <panda> thanks. We are available in the community meeting to discuss eventual details 14:23:39 <jaosorior> ack 14:23:44 <jaosorior> thanks for bringing it up panda 14:24:02 <jaosorior> Next topic: (chkumar) We are now able to run os_tempest in standalone job (patches in review) 14:24:31 <jaosorior> #link http://logs.openstack.org/00/627500/65/check/tripleo-ci-centos-7-standalone-os-tempest/198ae77/logs/undercloud/var/log/tempest/stestr_results.html.gz 14:24:45 <jaosorior> chandankumar: ^^ 14:24:48 <chandankumar> Hey It's me 14:25:12 <chandankumar> As we are working with OSA team to consume os_tempest role in Tripleo CI 14:25:33 <chandankumar> jaosorior: we are able to now run tempest using os_tempest in tripleo CI, 14:26:00 <chandankumar> here is the new job https://review.openstack.org/#/c/627500/ which does the same will be get merged soon 14:26:26 <chandankumar> and here is more updates what happened last week http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001946.html 14:26:33 <Tengu> (19 14:26:36 <chandankumar> we will try to finish integration by upcoming week 14:27:13 <jaosorior> chandankumar: nice work! I'll stay put if you need reviews. 14:27:24 <chandankumar> jaosorior: sure thanks :-) 14:27:47 <jaosorior> The next topic is from me: (jaosorior) Reminder: ssbarnea and I are going through Launchpad bugs each week 14:28:04 <jaosorior> So, if anybody is interested. It's one hour before this meeting 14:28:39 <jaosorior> That's all. 14:28:44 <panda> ssbarnea|bkp2 because it's rover I guess 14:31:44 <jaosorior> #topic Squad status 14:31:46 <jaosorior> ci 14:31:48 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:31:50 <jaosorior> upgrade 14:31:52 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:31:54 <jaosorior> containers 14:31:56 <jaosorior> #link https://trello.com/b/S8TmOU0u/tripleo-podman 14:31:58 <jaosorior> integration 14:32:00 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:32:02 <jaosorior> ui/cli 14:32:04 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:32:06 <jaosorior> validations 14:32:08 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:32:10 <jaosorior> networking 14:32:12 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:32:14 <jaosorior> security 14:32:16 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-security-squad 14:32:18 <jaosorior> edge 14:32:20 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:32:22 <jaosorior> Any squad wanting to bring up their status, or a topic for the general public? 14:32:34 <Tengu> hmm maybe for validation? 14:32:52 <jaosorior> Tengu: wanna give the update? 14:33:01 <Tengu> The validation framework is back on the bench, and we'll get weekly meeting (hopefully). 14:33:31 <Tengu> among the tasks, it was decided to move the validations as plain ansible roles in order to have a generic way to run and maintain them 14:33:47 <Tengu> more information about the on-going work is located on this public trello board 14:33:51 <Tengu> #link https://trello.com/b/x3h5FrnX/validation-framework 14:34:29 <jaosorior> thanks Tengu 14:35:24 <jaosorior> #topic bugs & blueprints 14:35:25 <jaosorior> #link https://launchpad.net/tripleo/+milestone/stein-3 14:35:27 <jaosorior> For Stein we currently have 21 (for stein-3) blueprints open in Launchpad. 14:35:29 <jaosorior> Bugs: 593 stein-3. 105 (0) open Storyboard bugs. 14:35:31 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76 14:36:26 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76 14:36:27 <jaosorior> #topic projects releases or stable backports 14:37:12 <jaosorior> #topic specs 14:37:14 <jaosorior> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:37:22 <jaosorior> Any specs someone would like to discuss here? 14:37:27 <openstackgerrit> Merged openstack/tripleo-heat-templates master: mistral-executor: bind mount the docker socket only when needed https://review.openstack.org/631775 14:39:12 <jaosorior> #topic open discussion 14:39:14 <jaosorior> Anything else that folks want to bring up to the meeting? 14:39:34 <Tengu> firewall management in tripleo maybe? 14:39:42 <fultonj> looks like https://review.openstack.org/#/c/622324 got a lot of feedback. i assume jistr integrated it into it's most recent update. 14:39:45 <Tengu> (sorry to bring that now ;)) 14:40:02 <marios> ci community call immediately after tripleo weekly https://bluejeans.com/4113567798 14:40:34 <fultonj> jistr: do you think it's ready ? (i need to read the updated version) 14:40:36 <jistr> fultonj: yea i did. There are some small tweaks to make to the spec still, but i think majority of it is quite close to reality. 14:41:00 <jistr> fultonj: i'll WIP it, one more update to match the currently-being-implemented CLI 14:41:08 <jistr> and then it's good i think 14:41:32 <fultonj> so if anyone provided feedback intially, they should check that it's been added as they think it should. 14:41:36 <fultonj> i'll do that 14:41:38 <fultonj> for ceph 14:41:41 <fultonj> thanks jistr 14:42:56 <jaosorior> fultonj: I'll check it out as well. Thanks for brining it up 14:42:59 <jistr> yup thanks fultonj & and all who provided feedback 14:43:19 <openstackgerrit> Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use os_tempest for running tempest on standalone https://review.openstack.org/628415 14:43:32 <jaosorior> Tengu: firewall management sounds good. 14:43:37 <jaosorior> * like a good topic 14:43:53 <jistr> btw there's a good bunch of patches for that spec already posted by me and chem, if you're interested in reviewing those too https://review.openstack.org/#/q/topic:bp/upgrades-with-os+(status:open+OR+status:merged) 14:43:55 <Tengu> hehe 14:44:24 <Tengu> jaosorior: so yeah, basically we have some issues with iptables management. some dangling rules, some default, unmanaged rules and the like. 14:44:35 <openstackgerrit> Chandan Kumar proposed openstack-infra/tripleo-ci master: Run tempest using os_tempest role in standalone job https://review.openstack.org/627500 14:44:45 <Tengu> I started working on the latter, especially the rules added by the iptables-services package itself: https://review.openstack.org/#/c/632117/ 14:45:18 <Tengu> I'd love to get some feedback on the approach, and if anyone has some idea about how to remove those nasty default rules from an already-deployed infra.... :) 14:45:47 <Tengu> fact is, one of them actually open the ssh port for the world - this is actually wanted now, as per https://review.openstack.org/#/q/Ie548f7216610e15af24c96f65a58cc8de603235c - but it is now optional. 14:46:06 <jaosorior> Tengu well, it is wanted for the undercloud only. Not the overcloud. 14:46:16 <Tengu> also, the default rules pushed by iptables-services prevent any logging to happen, as it has a REJECT before the LOG... 14:46:17 <jaosorior> the overcloud should only allow access in the ctlplane network. 14:46:32 <Tengu> jaosorior: yeah, in addition - ssh is wide open on the overcloud nodes, like the controllers..... 14:46:52 <jaosorior> Tengu: not anymore, AFAIK 14:47:22 <Tengu> jaosorior: well, once https://review.openstack.org/#/c/632117/ is merged, we'll be clean for new install. 14:47:36 <Tengu> but older ones will keep the 4-5 rules, unless we find a way to drop them. 14:47:38 <jaosorior> either way, the point remains, we have a mess with iptables rules management 14:47:55 <Tengu> the big issue is, puppet doesn't see those rules as they don't have any comment. 14:48:11 <Tengu> puppetlabs-firewall manages rules using the ressouce name as a comment directly. 14:48:44 <Tengu> so the only way I see is via ansible, in an update/upgrade task, and remove those rules with a state: absent 14:49:39 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-heat-templates master: Include the DB password in a Mistral environment for creating backups and restores https://review.openstack.org/632438 14:49:50 <Tengu> Also, we have to keep in mind that removing rule from puppet will NOT remove it from the system 14:50:11 <Tengu> we must set the "ensure => absent" in order to actually remove the rule. 14:50:36 <Tengu> we don't have a way to purge unknown rules, since other software are injecting stuff inside iptables, like neutron. 14:50:51 <jaosorior> Tengu: well, purging unkown rules would be problematic for neutron, wouldn't it? 14:51:03 <Tengu> yeah, that's my point. 14:51:06 <Tengu> UNLESS.... 14:51:18 <Tengu> we create dedicated chains, and neutron pushes its rules in them only. 14:51:27 <jaosorior> Tengu: might wanna talk to beagles about that 14:51:30 <Tengu> and we manage the "-j neutron-chain-name" within puppet 14:51:38 <Tengu> yeap 14:52:20 <Tengu> so yeah. pretty dense topic, I'll stop here for now, but I'm pretty sure I'll be back on it shortly ;). 14:52:43 <jaosorior> alright, at this point it makes sense to talk to some folks that know neutron and ask what can we do to play well with it 14:52:49 <Tengu> just wanted to draw attention on that. 14:53:00 <jaosorior> sounds to me like it's something we do want to fix 14:53:08 <Tengu> yep. 14:53:15 <Tengu> well, it would be good, at least :). 14:53:21 <jaosorior> having ports randomly open cause we didn't purge them is not a good idea 14:53:29 <Tengu> for the sake of security, clean env and some other consideration 14:53:38 <jaosorior> Tengu: thanks for bringing it up 14:53:44 <Tengu> yw :) 14:53:54 <jaosorior> Tengu: lets start tracking this in the security squad 14:53:58 <jaosorior> I'll bring it up there too. 14:54:17 <jaosorior> Alright! any other topics people wanna bring up? 14:54:22 <fultonj> Reminder: deadline for openstack summit CFP is tomorrow 11:59PM PT 14:54:37 <jaosorior> oh! true! 14:54:57 <beagles> fwiw- I think neutron managed chains are already on things like neutron-openvswi-PREROUTING 14:55:04 <beagles> I'll check to see if this is always the case 14:55:13 <beagles> jaosorior, Tengu 14:55:17 <beagles> ^^^ 14:55:23 <jaosorior> beagles: thanks! let us know if that's the case. 14:55:45 <Tengu> beagles: thanks! I'm in another mtg, but I'll be happy to discuss with you :) 14:55:47 <beagles> there may be other rules neutron "sticks in there" as a workaround to another issue - i.e. being able to talk to the openvswitch daemon 14:55:55 <beagles> Tengu, ack 14:56:11 <jaosorior> #endmeeting