15:00:37 #startmeeting RDO meeting - 2017-08-09 15:00:38 Meeting started Wed Aug 9 15:00:37 2017 UTC and is due to finish in 60 minutes. The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:39 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:41 The meeting name has been set to 'rdo_meeting___2017_08_09' 15:00:51 #topic roll call 15:01:12 yo o/ 15:01:27 \o\ 15:01:29 \o/ 15:01:32 /o/ 15:01:49 :) 15:01:55 o\ 15:02:04 #chair dmsimard PagliaccisCloud chandankumar 15:02:05 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard 15:02:22 Yatin Karel proposed openstack/os-brick-distgit rpm-master: switch from oslosphinx to openstackdocstheme https://review.rdoproject.org/r/8338 15:02:27 o/ 15:02:31 ᕕ( ᐕ )ᕗ ᕠ( ᐛ )و ᕕ( ᐕ )ᕗ 15:02:32 o/ 15:02:59 ok PagliaccisCloud has got me there 15:03:39 #chair number80 ykarel 15:03:39 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard number80 ykarel 15:04:38 trown, amoralej fyi we have a problem here.. http://logs.openstack.org/33/481233/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-pike/2094c73/console.html 15:04:48 http://logs.openstack.org/33/481233/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-pike/2094c73/console.html#_2017-08-09_13_02_37_199321 15:04:57 ok, let's start with the first topic 15:05:29 oh sorry 15:05:30 weshay, we are in a meeting, i'll take a look later 15:05:32 no prob 15:05:56 #topic Bundling oslosphinx with openstackdocstheme for projects not landing migration in time for Pike upper-constraints (but failing master-head) 15:06:01 dmsimard, that's yours 15:06:06 oh hai 15:06:24 o/ 15:06:25 I haven't really had time to check how many projects were still in this situation, if any 15:06:44 #link https://review.rdoproject.org/r/#/q/topic:openstackdocstheme 15:07:20 The gist of the issue is that some projects are not going to land the openstackdocstheme patch in the first release of Pike due to feature freeze and because they have not tagged something in time 15:07:25 #chair jschlueter 15:07:25 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter number80 ykarel 15:07:52 dmsimard, we had to apply the proposed solution of adding both oslo-sphinx and openstackdocsthem in a couple of packages 15:08:04 This causes the problem that master-head is FTBFS and master-uc is building -- but if you submit a patch to master-uc it fails to build because DLRN-rpmbuild (CI) builds with master-head, not master-uc. 15:08:45 Right, so I just wanted to have consensus that we /will/ be adding oslosphinx and docsthemes to a few projects in order to unblock this (there should be a clear TODO in the spec for removing it) 15:09:02 +1 from me 15:09:05 +1 15:09:24 +1 15:09:45 jschlueter: you ok with ^ ? 15:09:50 I know you hate sphinx :) 15:10:11 dmsimard: I hate sphinx runtime dep, BuildDep ok 15:10:21 ok, thanks 15:10:25 +1 15:11:06 #agreed projects blocked in transit between oslosphinx and openstackdocstheme will carry both packages as build requirements to unblock until there is a proper tagged release with only openstackdocstheme in it. 15:11:11 dmsimard: maybe stage a review for removing it with a workflow -1 and a well identified upstream commit or condition to watch for 15:11:47 jschlueter: it's something that I would like to be picked up automatically in the future 15:11:47 jschlueter, i added FIXME notes but yeah, it makes sense to create the review with W -1 15:11:57 jschlueter: as part of syncing requirements between reqs.txt and spec 15:12:10 jschlueter: i.e, tooling sees that oslosphinx is no longer there, drops it 15:12:23 but yes, there should be a TODO/FIXME 15:12:29 dmsimard: that's a big leap from where we are currently 15:12:30 Yatin Karel proposed openstack/os-brick-distgit rpm-master: switch from oslosphinx to openstackdocstheme https://review.rdoproject.org/r/8338 15:12:46 jschlueter: it's something I wrote about two years ago :) 15:12:59 jschlueter: https://trello.com/c/woaK75me/132-create-a-tool-to-match-requirements-between-reqstxt-and-specfiles 15:13:15 ok, if there isn't anything else, we can move on to next topic 15:13:43 ok 15:13:48 #topic Any particular plan around shipping Pike almost simultaneously with CentOS 7.4 ? 15:13:59 that's mine too 15:14:04 #info https://seven.centos.org/2017/08/centos-linux-7-1708-based-on-rhel-7-4-source-code/ 15:14:24 #info As reference, the issues found in 7.3 release - https://review.rdoproject.org/etherpad/p/release-centos-7.3 15:14:29 I just wanted to gently remind everyone that we're essentially shipping 7.4 and pike side by side 15:14:59 yeah, more fun :) 15:15:13 like, we're almost in RC and 7.4 is going to be rolling out in continous release stream 15:15:22 thisisfine.gif 15:15:47 I'll basically ask a crazy question just for the sake of discussion and ideas 15:16:16 If 7.4 is not out by Pike, are we delaying the release of Pike ? 15:16:25 Or the other way around 15:16:33 well, not that we can delay 7.4 15:17:16 Matthias Runge proposed config master: Add four more packages to centos-opstools https://review.rdoproject.org/r/8348 15:17:17 I guess what I'm trying to convey (badly) is that there could be things in 7.4 that makes it hard to support both 7.3 and 7.4 15:17:18 dmsimard: no delay for our release 15:17:26 so, definition of done is that the repos pass ci pipeline 15:17:27 we'll fix RHEL 7.4 issues when they arise 15:17:39 myoung: are you there ? 15:17:50 myoung and weshay will be testing trunk on RHEL 7.4 15:17:57 dmsimard: ping 15:18:11 myoung: have you been testing RDO on RHEL 7.4 since like, last week (or even before that?) 15:18:48 dmsimard: RDO on RHEL 7.3, earlier this week we talked about testing on 7.4. I can likely give that a try later on today or tomorrow 15:18:51 some packages in rhel are not provided in centos base but in sig repos, specially kvm, that tends to be a problem 15:19:11 and ceph 15:19:14 RDO Phase 2 tests the images created (centos) in RDO Phase 1, as well as images I build that are RDO (master, ocata, newton) + RHEL 7.3 15:19:26 Gabriele Cerami proposed config master: tripleo-upsteam: calling script from the cloned project dir https://review.rdoproject.org/r/8356 15:19:30 myoung: the faster we can test 7.4 to identify any potential issue the better 15:19:38 ++ 15:19:41 because we still don't have 7.4 in centos and we won't for a while 15:19:47 ++ 15:20:14 we will only have it on ci.centos, however we could enable centos CR repo in our DIB images so that we can see problems on jobs running from review.rdo 15:20:17 temorarily we need to support both 7.3 and 7.4, as jobs in rdo-ci will run with 7.4 from CR but jobs in rdo-cloud and in jobs in upstream gates 15:20:23 weshay, dmsimard: ack, I'll put them in non-voting however, as I don't want to turn everything red until we shake out the issues, presently we're on 7.3 to match RDO (centos 7.3) 15:20:24 dmsimard: it was decided already to have such pipeline internally 15:20:40 myoung: sure, they don't need to vote. 15:20:50 number80: what pipeline ? 15:20:53 oh... 15:20:54 o/ 15:20:57 +1 to enable CR in images for jobs running from ci.centos.org 15:20:59 rdo phase2 15:21:02 RDO testing on 7.4 15:21:04 #chair myoung weshay 15:21:05 Current chairs: PagliaccisCloud amoralej chandankumar dmsimard jschlueter myoung number80 weshay ykarel 15:21:06 number80: oh ok 15:21:24 amoralej: you would only enable CR on the weirdo ci.centos.org jobs ? 15:21:41 dmsimard, yes, we should use the same repos in all jobs in a pipeline 15:21:53 we shouldn't run oooq jobs with CR and weirdo without it 15:21:54 and 15:21:58 the sooner the better 15:22:07 it'd be nice if we could have both images 15:22:07 amoralej: ok, we can do that -- it'll also be an opportunity to update that image to the same we are using in review.rdo 15:22:20 amoralej: well, not exactly the same, but more in line 15:22:47 dmsimard, ok 15:23:13 #action dmsimard to update the weirdo dib image for ci.centos.org jobs to enable centos CR 15:23:57 #action myoung to start testing RDO on RHEL 7.4 15:24:00 myoung: ^ fair ? 15:24:03 just we need to be careful if we need to push something to fix jobs in 7.4 to not break 7.3 15:24:40 amoralej: the tricky part is that fixes can end up being anywhere.. in weirdo, in puppet, p-o-i, in packaging.. 15:24:47 we'll have to do on case by case basis 15:24:52 yes 15:24:57 dmsimard: ack, fair. Earlier this week I brought online 7.4 slaves so in theory things should be good to go. 15:25:13 last time we created a temporary repo with fixes 15:25:32 that was enabled only when using 7.4 15:25:38 7.3, i mean 15:25:40 I remember that 15:26:00 ok, anything else we need to think about ? 15:26:07 dmsimard, amoralej: yes, RDO on RHEL took some experimentation, and there were a number of odd packaging / dep issues to resolve. It's going to be iterative but I think it's more understood than when we started doing this last fall. 15:26:08 WWAS ? (What would Alan say) 15:26:54 :) 15:27:17 dmsimard: we support RHEL so we ought to find a solution 15:27:29 number80: oh, I remember something 15:27:32 likely a temporary repo to host fixed deps 15:27:48 number80: I saw in the CentOS 1708 blog post that the ppc build is likely not going to land simultaneously 15:27:54 is that an issue ? 15:27:55 my fear is that copr do not use EPEL buildroot which are based on RHEL 15:28:14 dmsimard: ppc64le is not primary arch so acceptable 15:28:30 we haven't finished bootstrap nor have a full CI pipeline 15:28:58 number80: ok, just thought I would point that out 15:29:09 ack 15:29:10 number80: are we shipping a stable ppc64le release ? 15:29:22 dmsimard: we're trying too 15:29:27 -o 15:29:37 ok 15:30:03 I think we can move on 15:30:07 ok 15:30:20 #topic DLRN builder instance migration to RDO Cloud 15:30:30 that's from jpena, but i'll introduce it 15:30:50 #info Plan: https://review.rdoproject.org/etherpad/p/DLRN-builder-migration 15:30:59 #info Proposal: do it during the week of Aug 14-18 15:31:17 so, we are moving DLRN builder instance to RDO cloud 15:31:56 one of the drivers is that we have the database in rdo-cloud and having dlrn close to it should help to alleviate db access issues 15:32:10 not just that but also improve build performance in general 15:32:16 yes 15:32:21 amoralej: we're doing it builder by builder, right ? not all at once ? 15:32:38 yes, builder by builder 15:32:47 repos will be always available at trunk.r.o 15:33:15 but we need to disable build process while syncing content between old and new server 15:33:34 yeah, I think the downtime should not be /too/ bad for each individual builder, depends on how long the rsync takes 15:33:42 We're still purging on a regular basis, yes ? 15:33:47 yes 15:33:59 we may even run a head-only if the lag is too big for master-uc 15:34:31 yeah ironically master-uc is the one that is going to take the longest and has the most builds 15:34:50 one of the impact is that we should change jobs in ci.centos to use public instance instead of internal 15:35:02 We're looking at ~185GB of data to transfer across all builders, it's not so bad. 15:35:25 amoralej: We can also consider leaving the fedora and newton builders in ci.centos to die 15:35:36 amoralej: considering we need to move to f26 and newton will EOL soon 15:36:04 mmmm, that sounds good dmsimard 15:36:17 amoralej: I think we only need to move trunk-primary.rdoproject.org DNS for the jobs to start using the new place 15:36:17 when is newton eol? 15:36:36 expected 2017-10-11 15:36:49 dmsimard, but we want them to keep using the builder or should be moved to the server? 15:36:54 to public, i mean 15:37:12 I don't understand the question 15:37:28 you mean, move trunk-primary to the new server, right? 15:37:48 yeah, it's ok to keep the concept of trunk-primary and trunk-backup 15:38:07 we have some jobs which use repositories from trunk-primary, mainly taking advantage of low latency as it was close 15:38:09 but I think jobs only use it to retrieve the hash and promote unless mistaken 15:38:16 ah, ok 15:38:17 they actually use trunk.rdo for repos 15:38:23 * dmsimard looks 15:38:27 i thought they use it for everything 15:38:51 I know weirdo doesn't 15:38:59 it just recovers the hash and uses trunk.rdo 15:39:04 no, only jobs in ci.centos, oooq 15:39:45 ok, then we are safe 15:39:55 https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/centos-master-current-tripleo.sh 15:40:01 https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/promote-repo-promote.sh 15:40:25 trown, weshay: OOOQ doesn't actually use the "trunk-primary.rdoproject.org" DNS right ? That's just to know where to promote ? 15:40:41 like, it uses the DLRN hash and appends it to trunk.rdo ? 15:41:25 dmsimard, it only uses it in ci 15:41:37 weshay: define "in ci" 15:41:53 weshay: because we're essentially moving "trunk-primary" and we want to know what to pay attention to 15:42:58 ya.. I'm checking the configs now 15:43:02 dmsimard, weshay i think it uses trunk.r.o https://github.com/openstack/tripleo-quickstart/blob/master/config/release/centosci/pike-current-tripleo.yml#L10 15:43:07 tq/config/release/centosci 15:43:54 i think it should be ok 15:44:11 ya.. looks like I'm out of date 15:44:22 https://github.com/openstack/tripleo-quickstart/blob/master/config/release/centosci/master.yml 15:44:48 ok, 15:45:11 so the plan is to start doing the migration on 16-aug 15:45:33 unless there is any issue with that date 15:45:56 weshay: cool, thanks 15:47:03 amoralej: I added some notes to the plan etherpad 15:47:07 #action jpena to send a mail to rdo-cloud about the migration planned date 15:47:14 s/rdo-cloud/rdo-list/ 15:47:15 work for jpena ^ :) 15:48:52 ok, let's move on 15:49:10 #topic We still haven't done any work towards shipping stable release of containers through CentOS registry 15:49:39 dmsimard, ^ 15:49:43 yeah 15:50:08 weshay, Slower, EmilienM, jschlueter ^ 15:50:26 Pike release is near and we have zero work done towards releasing what we would call stable container images 15:50:34 here "stable release" means cloudsig repos, right? 15:51:02 yes, and unless we change our mind, it means getting the containers build out of the CentOS registry pipeline 15:51:11 it's unfortunate that apevec isn't here to discuss this.. 15:51:47 considering the container images are tripleo specific, I wonder if it's still relevant to publish them under the "cloug sig" brand 15:51:56 dmsimard, we can add it to next week meeting also 15:52:05 or if we want to just publish them on dockerhub or something 15:52:37 building the container images out of the centos registry pipeline would be an interesting experience because it is somewhat close to the downstream approach but it might turn out to be a lot of work 15:52:58 amoralej: apevec is back next week ? 15:53:01 i don't have context about that discussion but using centos registry for that sounds as a good idea 15:53:16 ups, i don't think so 15:54:10 dmsimard, using centos registry requires to do anything special to build images? 15:54:16 or we could use kolla-build 15:54:19 amoralej: yeah, it's like downstream 15:54:31 amoralej: we need to provide pseudo distgit repos with Dockerfiles 15:54:32 registry is just registry 15:54:48 amoralej: and then they have jenkins automation to build those Dockerfiles and push them to the registry 15:54:52 so, it's not only at push time, but at build time 15:55:23 it sounds as quite different to what we have :( 15:55:35 amoralej: same thing as downstream with brew afaict 15:55:39 can kolla only create dockerfiles instead of doing the images? 15:56:07 but kolla has some idea about parent/child images, right? 15:56:27 that needs to be followed when building the images, i mean 15:57:34 dmsimard, so, what'd be the action about that? 15:58:06 I have absolutely no idea, from the different discussions I've had this can go in any direction 15:58:18 It doesn't sound like TripleO was interested in having RDO as their upstream for containers 15:58:41 i'm thinking in users, not in upstream for tripleo, tbh 15:58:49 so in that sense, it would mean that TripleO would be responsible for building and making their stable containers available 15:59:06 amoralej: kolla can generate just the dockerfiles, yes 15:59:21 amoralej: and yes, there is the notion of relationship between images 15:59:31 amoralej: jschlueter has some script to build the things in the right order 15:59:51 so, tripleo can build the images and provide a registry from undercloud, right? 16:00:15 that could be an option to users until there is some public registry with images 16:00:33 that gets a bit too in-depth for me to answer, I know there is a registry somewhere but I don't know if users are expected to build new images themselves 16:01:22 dmsimard, amoralej: I do have some of that logic. 16:01:39 dmsimard: no, we are supposed to ship them at some point but not necessarily for Pike 16:01:58 number80: who we, and what them 16:02:03 number80: it's an important nuance 16:02:14 we == RDO 16:02:29 RDO is accountable for building and shipping tripleo containers ? 16:02:36 them == containers usable by tripleo 16:02:59 I would see RDO accountable for shipping vanilla containers 16:03:07 AFAIK, providing artefact usable by tripleo is part of RDO Definition of Done 16:03:34 * number80 is not pushing for this version of DoD but until it's changed, this is the only one we have 16:03:46 ok, we'll have to discuss with the folks involved in tripleo 16:04:04 we're over time for now 16:04:11 let's make it a topic next week too. 16:04:17 although I won't be there, /me on PTO next week 16:04:55 dmsimard: that's a discussion for the mailing, it will be a long long one :) 16:05:08 yeah, i think we need to involve other folks 16:05:12 number80: it's an important distinction to make 16:05:21 number80: RDO provides packaging and the distribution of that packaging 16:05:40 number80: those are containers specific to the tripleo project 16:06:00 we have to be careful about what we do and why we do it 16:06:27 I'll fight for RDO to stay as neutral and agnostic as possible even if that means I'll come across as the bad guy :) 16:06:47 Yes, but we made that definition of DoD before containers, we are accountable for these artefacts until we change it 16:07:17 * number80 playing Devil Advocate here 16:07:47 sure 16:08:14 I'm not found of the idea of providing external stuff either, but we have to be consistent and not have RDO rely on *random* third-party containers 16:08:39 (let's close the meeting and someone chairing the next one) 16:08:45 *have 16:08:47 yeah 16:09:14 #topic volunteers to chair next meeting? 16:10:13 no volunteers? 16:10:41 ok, i'll take it again 16:10:50 #action amoralej to chair next meeting 16:11:02 we are out of time, but there is anything for open floor? 16:11:07 #topic open floor 16:11:49 ok, i'll close the meeting 16:11:56 thanks everyone for joining! 16:12:03 #endmeeting