15:02:02 <amoralej> #startmeeting RDO meeting - 2019-04-10
15:02:03 <openstack> Meeting started Wed Apr 10 15:02:02 2019 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:07 <openstack> The meeting name has been set to 'rdo_meeting___2019_04_10'
15:02:09 <fultonj> o/
15:02:23 <fmount> o/
15:02:23 <amoralej> #chair fultonj Duck
15:02:23 <openstack> Current chairs: Duck amoralej fultonj
15:02:30 <amoralej> #chair fmount
15:02:32 <openstack> Current chairs: Duck amoralej fmount fultonj
15:02:53 * Duck o/ o| o/
15:03:21 <amoralej> :)
15:03:23 <PagliaccisCloud> ٩( ᐛ )و
15:03:33 <amoralej> #chair PagliaccisCloud
15:03:34 <openstack> Current chairs: Duck PagliaccisCloud amoralej fmount fultonj
15:03:41 <baha> o/
15:04:22 <jpena> o/
15:04:22 <amoralej> #chair baha
15:04:22 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fmount fultonj
15:04:28 <amoralej> #chair jpena
15:04:29 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fmount fultonj jpena
15:04:34 <mjturek> o/
15:04:40 <amoralej> #chair mjturek
15:04:41 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fmount fultonj jpena mjturek
15:05:04 <amoralej> let's start
15:05:19 <amoralej> #topic Ansible 2.8
15:05:37 <amoralej> mjturek, we still didn't poromote 2.7! :)
15:05:45 <amoralej> fultonj, ^
15:05:46 <chandankumar> \o/
15:05:47 <fultonj> amoralej: it's blocked on ceph
15:05:47 <amoralej> not mjturek
15:06:00 <fultonj> https://review.rdoproject.org/r/#/c/18721
15:06:08 <amoralej> #info After RDO releases with 2.7, are we going to move to Ansible 2.8 in Stein?
15:06:11 <fultonj> ansible 2.7 --> ceph-ansible 4 --> nautilus
15:06:20 <amoralej> so, the question is if we are going to update to 2.8 in stein, right?
15:06:21 <fultonj> we're close to nautilus as per the next topic which we can unpack more there
15:06:32 <amoralej> yeah, let's discuss about 2.8
15:06:39 <fultonj> yes, wondering if 2.8 will go into stein after it's released
15:06:55 <rdogerrit> Sagi Shnaidman created rdo-infra/ansible-role-tripleo-ci-reproducer master: DNM: centos fixes  https://review.rdoproject.org/r/20120
15:06:56 <amoralej> so, in the past, we update stein in stable releases
15:07:13 <fultonj> so, in the past, we update stein in stable releases?
15:07:14 <rdogerrit> Sorin Sbarnea (zbr) proposed rdo-jobs master: Adds tripleo-build-containers-fedora-28-stein  https://review.rdoproject.org/r/19754
15:07:15 <amoralej> i think it can be done
15:07:19 <amoralej> grr
15:07:21 <fultonj> stein has no past yet right?
15:07:22 <amoralej> i meant, ansible
15:07:25 <fultonj> ah
15:07:27 <fultonj> :)
15:07:31 <amoralej> we updated ansible in stable releases
15:07:36 <amoralej> can be done
15:07:37 <amoralej> but
15:07:49 <amoralej> it needs everyone using ansible to support it
15:07:51 <ykarel> o/
15:08:07 <amoralej> tripleo, ceph-ansible, networking-ansible
15:08:18 <amoralej> #chair ykarel
15:08:19 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fmount fultonj jpena mjturek ykarel
15:08:33 <zbr> o/
15:08:39 <amoralej> #chair zbr
15:08:39 <openstack> Current chairs: Duck PagliaccisCloud amoralej baha fmount fultonj jpena mjturek ykarel zbr
15:08:52 <fultonj> amoralej: ceph-ansible will support 2.8 when it comes
15:09:17 <rdogerrit> Merged rdo-infra/ci-config master: Fix typo from [1] at the centosci job  https://review.rdoproject.org/r/20119
15:09:28 <fultonj> (and they probably want to stop testing w/ 2.7)
15:10:02 <amoralej> fultonj, well, it'd be great if they can keep testing both at least until it's updated in RDO
15:10:03 <fmount> amoralej: so we can reopen tests soon ^ for ansible 2.8
15:10:09 <fultonj> presumably tripleo will support 2.8 (i think that's the downstream plan)
15:10:12 <amoralej> fmount, yes
15:10:15 <fultonj> amoralej: totally agree
15:10:22 <fultonj> i don't like them only supporting one ansible version
15:10:25 <Erasmus> today right?
15:10:33 <zbr> ... hopefully. I know LOTS of places where "roles:" are still under user.
15:11:02 <amoralej> fultonj, when it's released and built in CBS you can propose a review to update ansible in both stein and train
15:11:09 <amoralej> to get it tested
15:11:10 <fultonj> amoralej: yes, i'd work wit them so they support both and don't break rdo
15:11:18 <rh-jelabarre> how much difference is there between 2.7 & 2.8?
15:11:30 <fultonj> them --> ceph-ansible team
15:11:51 <fultonj> rh-jelabarre: 2.8 isn't released yet
15:12:08 <amoralej> rh-jelabarre, no idea at all, but usually ansible updates are not "transparent"
15:12:09 <zbr> there is one essential difference when it comes to use or roles, mainly "roles:" were replaced by "import/include_roles.". Is not a trivial change, but is doable.
15:12:19 <fultonj> https://docs.ansible.com/ansible/devel/roadmap/ROADMAP_2_8.html
15:12:30 <zbr> code updated can still work with 2.7, as the feature already exists in 2.7
15:12:56 <fultonj> rh-jelabarre: better to assume it won't work and test before moving to it
15:13:03 <fultonj> then fix issues then move to it
15:13:24 <fultonj> amoralej: ok, so there's no object to RDO switcing to ansible 2.8 when it's out
15:13:29 <amoralej> what is important to understand by projects is that they need to be integraged in wider projects as OpenStack/Tripleo
15:13:46 <amoralej> making very restrictive assumptions/requirements in terms of requirements
15:14:02 <amoralej> makes it very hard to work together
15:14:05 <fultonj> amoralej: ++
15:14:28 <fultonj> amoralej: longer term it would be nice if projects shipped as a contianer
15:14:41 <amoralej> yeah, containers may help
15:14:47 <fultonj> e.g. if ceph-ansible was a container i could exec then they can use the version they test with
15:15:26 <amoralej> but even in that case, someone needs to take care of managing containers and getting them fixed, updated and fix security issues as they arise
15:15:27 <fultonj> amoralej: ok, let's plan to follow up on this after 2.8 comes out
15:15:43 <amoralej> anyway, for your question
15:15:45 <fultonj> amoralej: it's true
15:15:58 <amoralej> yeah, we will not block updating to 2.8
15:16:09 <fmount> fultonj: so plan will be to create a review to update ansible when it will be ready
15:16:10 <fultonj> if ceph-ansible does move to offering a container, then they'd have that responsiblity
15:16:34 <amoralej> but keep in mind that may depend on others proyects to get it updated
15:16:49 <fmount> amoralej: sure ^
15:16:56 <fmount> amoralej: totally agree
15:17:22 <fultonj> #action after ansible 2.8 GAs we can move stein to ansible 2.8 provided we coordinate and don't break the overall system
15:17:36 <amoralej> the workflow will be the same, propose patch to rdoinfo and start testing and discussing there
15:18:15 <fultonj> thanks amoralej
15:18:17 <amoralej> let's move on to next topic
15:18:27 <amoralej> #topic Nautilus and Stein
15:18:37 <fultonj> fmount: you want to take this one?
15:18:38 <amoralej> first of all
15:18:43 <fmount> fultonj: yeah
15:19:01 <amoralej> #info OpenStack stein has been released upstream right today
15:19:25 <fmount> we're trying to removing all hacks we had in the scenarios to make them green
15:19:31 <fmount> *remove
15:19:57 <fmount> so, we've seen with that https://review.openstack.org/#/c/651231/2/deployment/ceph-ansible/ceph-base.yaml
15:20:16 <fmount> (requested by ceph-ansible people)
15:20:36 <fmount> that tweaking with bridges the election started and everything worked
15:20:39 <fultonj> dsavineau: ^
15:21:11 <amoralej> but, i understood the issue was isolated to the syntax of "mon hosts"?
15:21:41 <fmount> so now we're able to make the job green applying hack both on networking side and on ceph-ansible side
15:22:03 <amoralej> fmount, what hack do you need in networking side
15:22:14 <amoralej> i though all you need is to fix ceph-ansible
15:22:18 <fmount> amoralej: https://review.openstack.org/#/c/651231/2/deployment/ceph-ansible/ceph-base.yaml
15:22:35 <fmount> our last test included this ^
15:22:51 <fmount> and we removed the ceph-ansible hack
15:23:11 <amoralej> that's weird
15:23:15 <fultonj> amoralej: it seems we can hack the network or ceph-ansible
15:23:34 <fmount> fultonj: amoralej exactly &
15:23:57 <amoralej> so, we need *both* or *any* of the hacks?
15:23:59 <fultonj> we should open a bug for both
15:24:05 <fultonj> amoralej: either
15:24:07 <fultonj> we don't need both
15:24:16 <fultonj> i propose we hack do a ceph.conf override in the CI job only as a workaround which references both bugs
15:24:21 <fultonj> to take care of scenario 001
15:24:22 <fmount> amoralej: yes, either
15:24:30 <fmount> amoralej: fultonj https://review.openstack.org/#/c/651239
15:24:35 <fultonj> then we remove the ceph.conf override wehn both issues are taken care of
15:24:37 <rdogerrit> Yatin Karel proposed openstack/neutron-lbaas-distgit stein-rdo: openstack-neutron-lbaas-14.0.0-1  https://review.rdoproject.org/r/20079
15:24:52 <fmount> this is what we can use to move on the scenario001
15:25:27 <fmount> and we'll have two bugs to keep investigate the issues
15:25:45 <fmount> (we'll indicate the bugs as comment in the review)
15:25:55 <amoralej> ok
15:26:23 <fmount> amoralej: we have first https://bugzilla.redhat.com/show_bug.cgi?id=1697977
15:26:23 <openstack> bugzilla.redhat.com bug 1697977 in Ceph-Ansible "V1 + V2 combination doesn't work properly in a Nautilus fresh deployment" [Unspecified,New] - Assigned to gabrioux
15:26:34 <fmount> amoralej: we need to open a bug for network
15:26:43 <fmount> as per https://review.openstack.org/#/c/651231
15:27:12 <ykarel> fmount, fultonj are there scenarios other than standalone? i would say good to test those as well to found other gaps if any
15:27:19 <ykarel> ovb, multinode etc
15:27:39 <ykarel> as there network would be different
15:28:04 <fmount> imho we just have this bug in the standalone, but we of course need to test everything
15:28:36 <fultonj> fmount ykarel we should test it w/ multinode too
15:29:05 <fultonj> ykarel: didn't we have a way to test multinode w/ nautilus and pending stein?
15:29:30 <ykarel> fultonj, we can add some job like we did for scenario004
15:29:53 <fmount> ykarel: yes, it would be great ^
15:29:56 <fultonj> sometthing like https://review.rdoproject.org/r/#/c/19141/ ?
15:30:01 <fultonj> but for master not pike?
15:30:03 <ykarel> fultonj, like https://review.rdoproject.org/r/#/c/19711/4/zuul.d/rdoinfo-jobs.yaml
15:30:34 <fultonj> fmount: want to try to put that ^ together w/ ykarel ?
15:30:41 <fmount> fultonj: sure
15:31:00 <fultonj> so are we ok with the plan for scenario 001?
15:31:05 <fultonj> amoralej ykarel fmount ^
15:31:07 <amoralej> wfm
15:31:17 <fmount> ok for me
15:31:21 <fultonj> basically make 001 green via the cephconfig mon override
15:31:26 <fultonj> and include a test for multinode
15:31:35 <fultonj> if so, do we want to talk about rgw (004)?
15:31:37 <fmount> we can add cephconfigoverride with bug comments
15:31:51 <fmount> talking about scenario004
15:32:05 <fmount> we have https://github.com/ceph/ceph/pull/27470
15:32:10 <fmount> from ceph side
15:32:27 <fmount> after some checks it will be merged
15:32:43 <fmount> and ceph-ansible people rebuild containers upstream
15:33:11 <amoralej> fmount, i guess we need ceph to do a new release 14.2.1 or similar, right?
15:33:20 <fmount> I think so
15:33:53 <rdogerrit> Yatin Karel proposed openstack/neutron-vpnaas-distgit stein-rdo: openstack-neutron-vpnaas-14.0.0-1  https://review.rdoproject.org/r/20109
15:33:54 <fultonj> amoralej: the ceph contianer people said they could turn the container around quickly
15:33:59 <fmount> from https://review.openstack.org/#/c/651337 <= I think is good for me as per https://docs.openstack.org/swift/latest/api/large_objects.html
15:34:06 <fultonj> but we'd have to see when there'd be a new ceph release with the rgw fix
15:34:09 <amoralej> fultonj, but we also need the packages to be updated
15:34:23 <amoralej> as we are hitting that in puppet-openstack-integration which does not use containers
15:34:31 <fultonj> amoralej: true
15:34:45 <fultonj> so CBS would need new builds
15:34:55 <amoralej> yes
15:35:04 <fultonj> CBS needs to make new RPMs and contianer needs to make new contianer and both need to wait for new ceph release
15:35:20 <amoralej> yep
15:35:41 <amoralej> there is a patch proposed in glance that would also make scenario004 to pass
15:35:59 <fmount> https://review.openstack.org/#/c/651337 <= this
15:36:00 <fultonj> amoralej: are you thinking the glance patch could turn 004 green
15:36:02 <amoralej> if whttps://review.openstack.org/#/c/651337/
15:36:04 <amoralej> yeah
15:36:10 <amoralej> fultonj, yes, it would
15:36:17 <fmount> yeah fultonj  that patch make the job green
15:36:25 <fmount> (tested locally)
15:36:43 <amoralej> so, if we get that merged and cherry-picked
15:36:55 <amoralej> that'd also be enough i'd say
15:37:07 <rdogerrit> Yatin Karel proposed openstack/neutron-fwaas-distgit stein-rdo: openstack-neutron-fwaas-14.0.0-1  https://review.rdoproject.org/r/20076
15:37:12 <fmount> yeah
15:37:34 <rdogerrit> rdo-trunk created openstack/aodh-distgit rpm-master: openstack-aodh: failed to build bef0f9b  https://review.rdoproject.org/r/20121
15:37:34 <ykarel> amoralej, we would need glance_store release as well
15:37:47 <amoralej> yes
15:37:54 <amoralej> we could ship the patch in the distgit
15:38:07 <ykarel> yup that can be an alternative
15:38:08 <amoralej> but it'd be better to get a new release
15:38:08 <rdogerrit> rdo-trunk created openstack/aodh-distgit rpm-master: openstack-aodh: failed to build bef0f9bf  https://review.rdoproject.org/r/20122
15:38:11 <amoralej> as it's in u-c
15:38:22 <ykarel> yes
15:39:20 <amoralej> let's try to move both works in parallel
15:39:24 <amoralej> and see who wins :)
15:39:30 <fultonj> :)
15:39:33 <fmount> :D
15:40:02 <amoralej> fultonj, is ceph reluctant to create new bug-fixes tags?
15:40:12 <amoralej> or it will be easy to get one?
15:40:40 <fultonj> ktdreyer: ^ any idea?
15:41:02 <fultonj> ktdreyer: regarding a fix for https://tracker.ceph.com/issues/39160
15:42:24 <fultonj> amoralej: i don't know i think that's a open question i'll try to find out
15:42:32 <amoralej> ok, no problem
15:42:56 <amoralej> it'd be great if we could get jobs passing in next week or so
15:43:17 <amoralej> we will not release RDO CloudSIG Stein until it's ready for nautilus
15:43:22 <amoralej> it will default to nautilus
15:43:41 <fultonj> amoralej: thanks for working to get nautilus in there
15:43:49 <fmount> amoralej: yes, it would be great, thanks
15:43:54 <rdogerrit> Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: Refactor image promoting to be distro aware  https://review.rdoproject.org/r/19966
15:44:07 <amoralej> ok
15:44:07 <fmount> amoralej: our effort is on making jobs pass
15:44:14 <amoralej> yeah, we are progressing
15:44:20 <amoralej> and catching real issues
15:44:23 <amoralej> what is great
15:44:34 <amoralej> well, not having issues is even better :)
15:44:48 <amoralej> but as there is no bugfree software
15:44:54 <amoralej> let's catch them!
15:44:58 <fultonj> +1
15:45:06 <fmount> ++
15:45:42 <amoralej> i think we can move on to next topic
15:45:56 <fmount> yes, we can move on
15:45:59 <amoralej> ok
15:46:02 <rh-jelabarre> wasn't the term "gotta catch 'em all"?
15:46:12 <amoralej> yeah :D
15:46:38 <amoralej> next topic, i'm not sure if it's already fixed
15:47:05 <ykarel> fultonj, also good to switch glancebackend in scenario004 so rgw is really tested
15:47:12 <ykarel> https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario004-standalone.yaml#L71
15:47:19 <ykarel> rbd --> swift
15:47:42 <rdogerrit> Merged openstack/sahara-plugin-spark-distgit stein-rdo: python-sahara-plugin-spark-1.0.0-1  https://review.rdoproject.org/r/20106
15:47:48 <amoralej> yes, covering both rbd and swift is good
15:47:49 <fultonj> wow
15:47:54 <amoralej> zbr, "gerrit does not allow users to list group membership" is that yours?
15:48:05 <zbr> amoralej: yep, but is sorted.
15:48:15 <amoralej> nice, we can skip, then
15:48:22 <amoralej> i wasn't sure
15:48:23 <amoralej> good
15:48:41 <amoralej> then, let's move to open floor
15:48:45 <amoralej> #topic open floor
15:48:46 <zbr> one one minor observation, is seems that gerrit gives a weird merging error on changes made on config branch.
15:49:17 <amoralej> in config repo?
15:49:24 <zbr> have a look at https://review.rdoproject.org/r/#/c/20063/
15:50:10 <amoralej> "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset."
15:50:23 <amoralej> jpena, ^ are you aware of issues with that repo?
15:50:29 <amoralej> that's rdo-infra/ci-config
15:50:31 <amoralej> not config
15:50:32 <amoralej> mmm
15:50:38 <amoralej> oh
15:50:42 <amoralej> it's the branch
15:50:48 <zbr> (is not urgent as I no longer need it but could be another issue)
15:51:04 <amoralej> i'm not sure if using gerrit to manage refs/meta/config is appropiated
15:51:05 <jpena> the refs/meta/config branch is special, I think this should be changed via the resource engine
15:51:07 <amoralej> first time i see it
15:51:51 <zbr> on codeeng I used the gui editing of ACLs a lot, and never seen it, must be zuul specific.
15:54:01 <amoralej> zbr, in softwarefactory the repos configuration is managed using yaml files in the config repo
15:54:08 <amoralej> resources
15:54:43 <rdogerrit> Merged openstack/neutron-lbaas-distgit stein-rdo: openstack-neutron-lbaas-14.0.0-1  https://review.rdoproject.org/r/20079
15:55:04 <amoralej> zbr, you can ask if you still need to fix acls after the meeting
15:55:18 <zbr> no need anymore. thanks.
15:55:39 <baha> Am I safe to bring up a quick open floor topic?
15:55:40 <amoralej> other topics for open floor?
15:55:45 <amoralej> sure, go ahead baha
15:55:47 <baha> \o/
15:56:16 <baha> mjturek and I have a patch up to transition the ppc64le containers build job to the new set of tripleo playbooks, and use buildah. The patch is at https://review.rdoproject.org/r/#/c/20031/ , and any feedback or reviews would be appreciated!
15:57:22 <zbr> can i ask something about multi arch and container registry?
15:57:54 <rdogerrit> Merged openstack/neutron-vpnaas-distgit stein-rdo: openstack-neutron-vpnaas-14.0.0-1  https://review.rdoproject.org/r/20109
15:58:05 <amoralej> zbr, sure
15:58:14 <amoralej> i think baha and mjturek are working on that
15:58:17 <zbr> afaik, docker hub supports multi arch containers and they have the same name. meaning that when you pull a container it will pick the right arch without you having to mention it (if it exists obviously)
15:59:02 <rdogerrit> Sagi Shnaidman proposed rdo-infra/ansible-role-tripleo-ci-reproducer master: DNM: try standalone  https://review.rdoproject.org/r/19917
15:59:07 <zbr> this makes multi-arch deployments very easy to use. if we use different namespaces for each arch, it does add a serious amount of extra work to deal with that.
15:59:41 <zbr> think about a case where you could have mixed arch in the same deployment.
16:00:09 <amoralej> i think there has been some discussint about that
16:00:10 <baha> zbr: There's some ongoing discussion on how to handle that on the list, I think the current idea is to have the RDO registry use separate namespaces, and to have dockerhub collapse them into one namespace using manifest lists (which the RDO registry does not seem to support)
16:00:22 <baha> I
16:00:34 <baha> I'm not an expert on it though so don't quote me =)
16:00:52 <mjturek> my understanding is the same as baha's but it's been hard to wrap our heads around.
16:01:00 <zbr> baha: ok, so the separation is temporary due to our limitations. hopefully we will overcome them one day.
16:01:20 <mjturek> zbr: the hope is to  overcome them when publishing to dockerhub I believe
16:01:26 <zbr> that clearly answers my question, i was only worried about longer term easy of use.
16:02:13 <mjturek> zbr: thank you! feel free to reach out to us and Vorrtex with questions/concerns
16:04:20 <rdogerrit> Merged openstack/neutron-fwaas-distgit stein-rdo: openstack-neutron-fwaas-14.0.0-1  https://review.rdoproject.org/r/20076
16:04:41 <amoralej> ok
16:05:15 <amoralej> we are out of time
16:05:20 <amoralej> any other topic?
16:06:22 <amoralej> any volunteer to chair next week?
16:07:08 <ykarel> i can take
16:07:20 <amoralej> #action ykarel to chair next week
16:07:23 <amoralej> thanks ykarel
16:07:30 <amoralej> i'm closing the meeting
16:07:33 <amoralej> thanks all for joining
16:07:45 <mjturek> thanks for running amoralej
16:07:49 <amoralej> #endmeeting