15:00:31 <jpena> #startmeeting RDO meeting - 2019-05-08
15:00:33 <openstack> Meeting started Wed May  8 15:00:31 2019 UTC and is due to finish in 60 minutes.  The chair is jpena. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:36 <jpena> #topic roll call
15:00:37 <openstack> The meeting name has been set to 'rdo_meeting___2019_05_08'
15:00:51 <jpena> remember to add any last-minute items to the agenda at https://etherpad.openstack.org/p/RDO-Meeting
15:01:18 <amoralej> o/
15:01:21 <ykarel> o/
15:01:23 <fultonj> o/
15:01:24 <fmount> o/
15:01:24 <jpena> #chair amoralej ykarel
15:01:25 <openstack> Current chairs: amoralej jpena ykarel
15:01:29 <jpena> #chair fultonj fmount
15:01:30 <openstack> Current chairs: amoralej fmount fultonj jpena ykarel
15:02:48 <baha> o/
15:03:14 <jpena> #chair baha
15:03:15 <openstack> Current chairs: amoralej baha fmount fultonj jpena ykarel
15:05:19 <fultonj> mwhahaha: i think you're first on the rdo meeting agenda
15:05:23 <jpena> let's start with the agenda
15:05:29 <mwhahaha> i have stuff?
15:05:29 <jpena> #topic request to update to puppet 6 in RDO:- https://trello.com/c/isJkCzYZ
15:05:35 <jpena> mwhahaha: ^^
15:05:37 <jpena> #chair mwhahaha
15:05:38 <openstack> Current chairs: amoralej baha fmount fultonj jpena mwhahaha ykarel
15:05:45 <fultonj> https://etherpad.openstack.org/p/RDO-Meeting
15:05:48 <fultonj> puppet6 ?
15:05:55 <mwhahaha> so yea we'd like to move to 6, we'll need to wait until centos8 though
15:06:02 <mwhahaha> because it requires a newer ruby
15:06:06 <mwhahaha> there's work to get it into fedora
15:06:08 <mwhahaha> that's it
15:06:31 <ykarel> mwhahaha, we can start with puppet6 in Fedora
15:06:43 <jpena> #info puppet 6 will need to wait until centos 8 is available, can do some pre-work in Fedora
15:06:50 <ykarel> mwhahaha, i tried scratch build of puppet 6 today, and deployed with poi scenario001
15:06:54 <amoralej> who mantains puppet in fedora?
15:06:54 <ykarel> and it went good
15:07:06 <amoralej> cool ykarel
15:07:29 <amoralej> ykarel, could you send a PR?
15:07:29 <ykarel> i see 7 members https://src.fedoraproject.org/rpms/puppet
15:07:34 <ykarel> amoralej, yes sending
15:07:45 <ykarel> just got test results from local run
15:08:30 <ykarel> mwhahaha, so to what version need to move, 6.4.1
15:08:38 <amoralej> what version of ruby it requires?
15:08:39 <ykarel> i see this in an ubuntu job, so tried this
15:08:43 <ykarel> amoralej, 2.3 atleast
15:08:46 <mwhahaha> yea i think that's current
15:08:47 <amoralej> ykarel, you tested it in centos7?
15:08:49 <amoralej> or fedora?
15:08:51 <amoralej> with p-o-i
15:08:53 <amoralej> i mean
15:08:55 <ykarel> amoralej, in Fedora with poi
15:10:34 <amoralej> mwhahaha, do you expect puppet modules to just work with puppet-6?
15:10:38 <amoralej> or changes are expected?
15:10:43 <mwhahaha> i think they should work
15:10:46 <ykarel> amoralej, mwhahaha i needed one change in poi to get it running
15:10:51 <mwhahaha> we'll need to address anything as needed
15:10:58 <ykarel> amoralej, mwhahaha added a new module: cron_core
15:11:08 <mwhahaha> yea there were a few core modules taht got split out
15:11:18 <mwhahaha> selinux/augeas/cron
15:11:38 <mwhahaha> i fixed up the puppet tripleo stuff to work with the latest (as far as i know)
15:12:00 <amoralej> so we will be able to update it in fedora but not in centos7
15:12:07 <amoralej> we'll have to wait for centos8
15:12:13 <amoralej> ruby in centos7 is 2.0
15:12:16 <amoralej> right?
15:12:58 <ykarel> amoralej, yes
15:13:04 <ykarel> it 2.0.0 in CentOS7
15:13:11 <amoralej> yep
15:13:14 <ykarel> so need to wait for bump in centos
15:13:18 <amoralej> yes
15:13:20 <ykarel> pushed PR:- https://src.fedoraproject.org/rpms/puppet/pull-request/6
15:13:27 <amoralej> but we can start by updating in fedora
15:13:32 <amoralej> we have non-voting fedora jobs
15:13:37 <amoralej> at least p-o-i
15:13:43 <amoralej> mwhahaha, ^ wdyt?
15:14:23 <amoralej> 1st step, update it in fedora
15:14:32 <amoralej> and use fedora jobs to test with puppet6
15:14:42 <amoralej> then update it in centos8 whe it's ready
15:15:29 <ykarel> +1
15:16:51 <jpena> next topic?
15:16:56 <amoralej> #action ykarel to propose pr to update puppet in fedora
15:17:19 <amoralej> #action amoralej to update puppet in fedora stabilized repo once is built in fedora
15:18:33 <jpena> #topic ceph/ansible post-update follow up
15:18:37 <fultonj> next topic...
15:18:42 <fultonj> We seem to have moved to Ceph Nautilus and Ansible 2.7 https://review.rdoproject.org/r/#/c/18721 (merged)
15:18:51 <amoralej> fultonj, i hope so :)
15:18:56 <fultonj> thanks very much to amoralej and ykarel and fmount for their help on this!
15:19:02 <fultonj> Is this case closed or is there anything else we need to talk about here?
15:19:03 <fmount> yes, and thank you amoralej fultonj ykarel  for the effort
15:19:17 <fultonj> i also just wanted to close the loop here in case there's any hanging issues
15:19:25 <amoralej> fultonj, i think we are done, just waiting for inal releases of tripleo packages
15:19:27 <ykarel> For train we can consider move to ansible 2.8 once it's out
15:19:41 <ykarel> ^^ also need support in ceph-ansible
15:19:50 <amoralej> ykarel, are you aware of the plans for new tripleo releases?
15:19:55 <amoralej> in stein, i mean
15:20:02 <fmount> yes, good idea, we can start a new review as we've done for 2.7 when it will be out
15:20:04 <fultonj> ceph-ansible 4.0.0.rc5 and newer support ansible 2.8
15:20:14 <ykarel> amoralej, nope
15:20:18 <ykarel> mwhahaha, EmilienM ^^
15:20:53 <mwhahaha> not at the moment, we can push new releases if perfered
15:21:00 <mwhahaha> preferred rather
15:21:34 <amoralej> mwhahaha, there are a couple of fixes in tht and tripleo-common iirc needed for nautilus
15:21:45 <mwhahaha> k i can try and cut one later
15:21:49 * mwhahaha adds to the list o' work
15:22:12 <fultonj> i expect a review like https://review.rdoproject.org/r/#/c/18721 but with bump to ansible 2.8 (after it's released) will simply pass on the ceph section
15:22:27 <fultonj> but feel free to ping me if it doesn't
15:22:46 <amoralej> fultonj, you are optimistic :)
15:23:20 <fultonj> :)
15:23:33 <amoralej> the good part is that last time ci jobs were useful to find out real issues
15:23:46 <amoralej> so, if jobs pass, i think we are in good shape
15:24:01 <amoralej> we'll propose to bump it as soon as we have a build in cbs
15:24:55 <ykarel> fmount, fultonj so i think you can switch scenario004 to test rgw now
15:25:23 <amoralej> ykarel, i thought your patch to move scenario004 to radosgw was merged
15:25:26 <amoralej> wasn't it?
15:25:30 <fmount> ykarel: sure, I think we can do it
15:25:51 <ykarel> amoralej, i only pushed it as a part of testing
15:25:58 <ykarel> with other changes
15:26:03 <amoralej> ok
15:26:09 <ykarel> fmount, ack
15:26:09 <amoralej> i think it'd improve coverage
15:26:29 <ykarel> yes me too
15:26:43 <fultonj> fmount: can you look into how that works to confirm we can make the change ykarel is suggesting?
15:27:00 <fmount> fultonj: of course, I've added it on my todo list
15:27:03 <fultonj> thanks fmount
15:27:10 <fultonj> it would be a good update to the tripleo-docs too
15:29:49 <fultonj> so i guess that's it for ceph updates for now
15:29:52 <fultonj> thanks
15:30:09 <fmount> yes I think it's enough
15:30:18 <jpena> good, let's move on
15:30:19 <fmount> thanks fultonj
15:30:30 <jpena> #topic ppc64le container uploads
15:30:44 <baha> Right!
15:31:00 <baha> So, the ppc64le containers build job is now stable https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/ (thanks again to jpena and amoralej for the fast merges)
15:31:45 <amoralej> that's cool baha
15:31:55 <amoralej> but no logs?
15:31:55 <baha> =)
15:32:03 <amoralej> https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/711/ ?
15:32:29 <baha> Logs over at https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/
15:33:03 <baha> https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/711/logs/logs/build.log is the most recent build log, for example
15:33:23 <amoralej> it'd be good to have a logs link in https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/711/
15:33:26 <amoralej> but ok
15:33:29 <amoralej> sounds good
15:33:55 <baha> Sure, we can update it to make sure it spits the link out
15:34:36 <baha> But since we've transitioned over to the new tripleo playbooks and have logs uploading, now seemed like a good time to start talking about container uploads again. A bunch of us met up at the Open Infra summit, and after some discussion, tonyb has suggested doing this the way he and Vorrtex need requires the rdoproject.org registry to be updated to support manifest lists
15:35:01 <baha> weshay was there too, and voiced his support for it
15:35:46 <jpena> so this is a change in the plan, right? Initially, we did not plan to use manifest lists in the rdo registry, but on dockerhub
15:36:04 <jpena> I'm checking if the openshift registry supports it... I remember the current version doesn't, but need to check 3.11
15:36:04 <ykarel> baha, but looks like it's not building ppc containers, as in that job dlrn repo is used, and that have only x86_64 packages
15:36:07 <ykarel> amoralej, right?
15:36:18 <tonyb> jpena: that was a mistake, all registries involved need to support manifests
15:36:21 <rdogerrit> Bogdan Dobrelya proposed rdoinfo master: Bump puppet for Stein release  https://review.rdoproject.org/r/20572
15:36:22 <baha> ykarel: the dlrn repos have ppc packages, we've been building off them for quite some time
15:36:42 <amoralej> ykarel, there are ppc packages in deps repo
15:36:46 <amoralej> and dlrn is noarch
15:36:50 <tonyb> so I guess yes it is a change in the plan and a new work item
15:36:55 <baha> That's the correct statement
15:37:02 <ykarel> amoralej, okk good, /me remember now
15:37:35 <baha> We were running around trying to get the noarch packages updated for quite a while, you'd think I'd remember better
15:37:52 <ykarel> https://trunk.rdoproject.org/centos7-master/deps/latest/
15:37:59 <jpena> I'm starting to remember now. According to https://trello.com/c/4EcAIJrd/1303-5-quayregistry-add-support-for-manifest-list-rd, there's no support for manifest lists in the openshift registry, so... To do this, we would need to make a deeper change and switch registries
15:38:38 <jpena> which I'm not saying it's a bad idea
15:38:56 <jpena> however, switching to let's say the docker registry would mean changes in more jbos
15:38:57 <jpena> jobs
15:39:56 <amoralej> so, openshift registry does not support manifest list at all?
15:40:02 <jpena> nope
15:40:05 <amoralej> ups
15:40:19 <Vorrtex> tonyb I thought you had mentioned Quay did update with manifest list support last week?
15:40:19 <jpena> quay just announced support in their latest release (4.0 I think)
15:40:40 <amoralej> yeah, i saw announcement of quay
15:40:49 <jpena> quay is not open source yet, afaik, or is it?
15:41:38 <tonyb> jpena: quay.io 3.0 realeased last week has manifest supprt
15:41:53 <tonyb> let me double check
15:42:39 <jpena> tonyb: I think so. The issue is we cannot install it in the RDO Infra (requires a license)
15:42:56 <tonyb> jpena: What are you using ATM?
15:43:03 <jpena> tonyb: the openshift registry
15:43:14 <tonyb> *sigh*
15:43:20 * jpena shrugs
15:43:33 <tonyb> Everyone around the table thought differnt :(
15:44:15 <jpena> in any case, we could even use the docker registry, I've tested it and would be pretty easy to deploy. My point is that, if we choose that route, it will require several changes in our container building processes to match
15:44:40 <tonyb> jpena: okay
15:44:55 <amoralej> jpena, does docker registry supports ssl properly and basic stuff?
15:44:57 <amoralej> namespaces
15:44:58 <amoralej> etc...
15:45:47 <jpena> amoralej: SSL yes, I would think namespaces too, but I'm not sure about ACL's (like user X can only push to namespace Y)
15:45:57 <rdogerrit> Merged config master: Add pmu-tools to centos-opstools  https://review.rdoproject.org/r/20596
15:46:40 <tonyb> So what can we do to a) evaluate which registry to move to (and if we can get a license for quay) ; and b) prepare for process changes?
15:46:43 <amoralej> we'd need to evaluate a bit alternatives
15:47:04 <rdogerrit> Yatin Karel proposed rdo-infra/ansible-role-weirdo-puppet-openstack master: Install puppet-watcher from package in master  https://review.rdoproject.org/r/20601
15:47:40 <jpena> for b) I would involve weshay and the TripleO CI team
15:47:48 <Vorrtex> jpena docker has "grants" which apparently work like ACLs... According to this: https://docs.docker.com/ee/ucp/authorization/
15:48:24 <jpena> Vorrtex: that's Docker "Enterprise Edition" (read "pay for it")
15:48:47 <Vorrtex> jpena oh, sorry, my bad.
15:48:57 <jpena> but I can check. I did some tests when we started talking about it, but stopped once it was said we would not need manifest lists
15:49:01 * weshay reads
15:51:59 <tonyb> Is there a list of 'minimum acceptable features' somewhere?
15:52:27 <jpena> weshay: TL;DR: supporting manifest lists in the RDO Registry requires a different registry (openshift registry doesn't support it). Switching to a new registry would mean the process to push container images could change (besides the migration itself)
15:52:41 <jpena> so container jobs could require fixes
15:52:59 <amoralej> current authentication is specific to openshift registry, right?
15:53:59 <jpena> amoralej: I think so
15:56:17 <amoralej> i think the first step is to find out what registry may be adequate for us
15:56:22 <amoralej> and impact on ci jobs
15:56:57 <jpena> I can do some more research on the docker registry and prepare an etherpad
15:57:06 <jpena> if someone else can check other options, that'd be great
15:59:03 <jpena> I have started https://review.rdoproject.org/etherpad/p/new-rdo-registry to track
16:01:49 <jpena> we're going beyond our time, anything else to add?
16:03:23 <weshay> jpena tonyb baha, if the manifest is only needed after the ppc containers were validated, then the rdo registry does not need to change
16:04:04 <weshay> when we spoke about it.. creating the manifest would be part of the combined promotion.. then sent to dockerhub.. perhaps I'm misunderstanding
16:04:31 <jpena> that was my initial understanding, but it looks like the discussion at the Summit changed it
16:04:57 <weshay> jpena I think what I'm reading here does not represent what I heard at summit
16:05:34 <jpena> o_O
16:05:48 <jpena> so... as a first step, should we re-discuss it?
16:05:48 <weshay> if we *need* manifests before promotion.. then yes we'll need a diff registry
16:06:10 <tonyb> weshay, jpena: okay.  I'm just plain confused now
16:06:17 <jpena> me too :)
16:06:24 <baha> ditto =D
16:06:29 <weshay> I think it depends on how steve baker is going to implement manifest support in tc
16:06:46 <tonyb> weshay: I thought the problem had to do with not being able to combine at promotion images from multiple orgs/namespaces
16:07:23 <weshay> right.. let's cover that...
16:07:24 <tonyb> weshay: Well that IIUC will pull all layers from a manifest when additionalarchitectures are enabled
16:07:44 <weshay> so..  using the same namespace...
16:08:06 <weshay> to be able to support that.. we need manifests?
16:08:55 <tonyb> weshay: that was my understanding
16:09:00 <weshay> ya.. me too
16:09:01 <Vorrtex> manifest list images cannot span across separate namespaces.
16:09:33 <weshay> so while the containers are on different registries...  do they still need a manifest, no right
16:10:07 <weshay> the two containers come together only in one place.. docker.io
16:11:33 <weshay> do we need both x86 and ppc containers in the rdo registry  together?
16:11:37 <tonyb> weshay: from a pure container POV the answer woudl be no (manifests are not required), but from a toci POV, then yes we need manifests
16:11:51 <weshay> in rdo registry?
16:11:57 <tonyb> ... until we can do something like a all-in-one install on ppc64le
16:12:29 <tonyb> weshay: I would think so assuming that's the registry we look at during testing
16:12:31 <weshay> noting... toci does not pull containers for the rdo-registry, it's only a stage env
16:12:46 <weshay> s/for/from
16:13:00 <tonyb> ok
16:14:17 <weshay> jpena if steve changes how containers are prepped and initially creates a manifest for everything.... then we probably do need to change the rdo registry
16:14:23 <weshay> that's the bit I am not clear atm
16:15:02 <jpena> weshay: ok. In that case, I'll keep on investigating options, and we'll need to be aware of the migration process (both for the infra itself and the jobs)
16:15:44 <jpena> #action jpena to check potential alternatives for the RDO registry
16:15:59 <jpena> anything else to discuss in this topic?
16:17:04 <tonyb> I'm good
16:17:11 <jpena> ok then
16:17:13 <baha> Thank you jpena
16:17:17 <jpena> #topic chair for the next meeting
16:17:23 <jpena> any volunteer before we close?
16:18:06 <ykarel> i can take it
16:18:16 <jpena> thanks ykarel
16:18:38 <jpena> it's late. If we have anything for the open floor, let's discuss it after the meeting
16:18:41 <jpena> #endmeeting