15:01:10 <jpena> #startmeeting RDO meeting - 2017-03-29
15:01:11 <openstack> Meeting started Wed Mar 29 15:01:10 2017 UTC and is due to finish in 60 minutes.  The chair is jpena. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:15 <openstack> The meeting name has been set to 'rdo_meeting___2017_03_29'
15:01:35 <jpena> the agenda is available at https://etherpad.openstack.org/p/RDO-Meeting, feel free to add last-minute topics
15:01:39 <jpena> #topic roll call
15:01:43 <amoralej> o/
15:01:49 <radez> o/
15:02:05 <jpena> #chair radez amoralej
15:02:05 <apevec> o/
15:02:05 <openstack> Current chairs: amoralej jpena radez
15:02:17 <rdogerrit> Alfredo Moralejo created openstack/neutron-distgit: Update to 9.3.0  https://review.rdoproject.org/r/6079
15:02:41 <rdogerrit> Jakub Libosvar created openstack/neutron-distgit: Workaround orphaned neutron-rootwrap-daemon  https://review.rdoproject.org/r/6080
15:02:42 <rdogerrit> Alfredo Moralejo created openstack/neutron-fwaas-distgit: Update to 9.0.1  https://review.rdoproject.org/r/6081
15:03:48 <jlibosva> ajo: ^^ when you're back :) I need to test it though
15:04:21 <jpena> not many people around, or most are lurking :). Let's move on
15:04:28 <jpena> #topic trunk.rdoproject.org proposed maintenance for Thu 30 Mar, 0900 UTC / 1100 CET
15:05:16 <jpena> The public-facing DLRN instance needs some downtime to be patched. It should take < 1 hour tomorrow
15:05:17 <amoralej> jpena, that will affect upstream CI
15:05:35 <jpena> right, during reboot some jobs could fail, and just need to be rechecked
15:05:47 <amoralej> EmilienM ^
15:06:31 <number80> o/
15:06:36 <jpena> #chair number80
15:06:36 <openstack> Current chairs: amoralej jpena number80 radez
15:07:10 <EmilienM> ack
15:07:33 <jpena> I'll send an e-mail to rdo-list as a reminder
15:07:44 <jpena> #action jpena to send e-mail to rdo-list about trunk.rdo maintenance
15:08:03 <jpena> next topic?
15:08:44 <chandankumar> \o/
15:09:10 <jpena> #chair chandankumar
15:09:10 <openstack> Current chairs: amoralej chandankumar jpena number80 radez
15:09:12 <jpena> #topic triple-o aarch64 support progress report
15:09:19 <jpena> radez ^^
15:09:28 <radez> going to paste a bit, typed up ahead of time
15:09:35 <radez> Been working on aarch64 support for triple-o in RDO
15:09:37 <radez> I've been able to apply small patches to get disk images for undercloud and overcloud built and to get an openstack undercloud install to complete.
15:09:46 <radez> This work is being done in the context of OPNFV apex, though all the patches that I'm generating will be posted upstream as I post them to APEX too
15:09:51 <radez> Next step is to attempt to run an overcloud deploy and start collecting patches needed to get an overcloud deployed.
15:09:57 <radez> May thx to number80 for his support with packaging and on the CentOS side jperrin and jbastian for their support with CentOS and ARM.
15:10:01 <radez> Planning to send an email with these updates later this week too.
15:10:02 <radez> fin
15:10:05 <radez> quiestions?
15:10:12 <trown> o/
15:11:54 <jpena> #chair trown
15:11:54 <openstack> Current chairs: amoralej chandankumar jpena number80 radez trown
15:12:08 <dmsimard> \o sorry I'm late
15:12:17 <jpena> #chair dmsimard
15:12:18 <openstack> Current chairs: amoralej chandankumar dmsimard jpena number80 radez trown
15:12:24 <jpena> ok, let's move on
15:12:35 <jpena> #topic mirroring of trunk.rdoproject.org
15:12:37 <trown> radez: nice one, definitely ping me on any TripleO patches you need reviews for
15:12:39 <dmsimard> pabelanger: ^
15:13:01 <radez> trown: thx, only one out there right now is the dib patch
15:13:14 <radez> ian has reviewed so far
15:13:36 <radez> more coming csince the undercloud install has completed now
15:13:38 <dmsimard> Not sure if pabelanger is around but I can speak on his behalf if he's not
15:13:48 <pabelanger> yes
15:13:51 <pabelanger> here!
15:14:02 <dmsimard> pabelanger: floor is yours, I'll jump in as necessary :p
15:14:13 <pabelanger> So, basically. I would like to rsync things from trunk.rdoproject.org to openstack-infra
15:14:27 <pabelanger> which means, rdo project needs to enable rsync on your servers
15:14:36 <pabelanger> I'm curious how to go about doing that
15:14:38 <jpena> what would you want to sync?
15:14:48 <pabelanger> all of trunk.rdoproject.org
15:14:53 <jpena> all?
15:14:54 <pabelanger> or parts of it
15:15:09 <pabelanger> basically, what ever tripleo is consuming, we need to mirror
15:15:37 <jpena> mmm, tripleo is consuming the current/ symlink in some cases, isn't it EmilienM?
15:15:41 <dmsimard> jpena: yes
15:15:42 <EmilienM> yes
15:15:43 <amoralej> yes
15:15:48 <EmilienM> YES
15:15:59 <jpena> ok, so it's a bit more complex than it seems, then
15:16:02 <EmilienM> for tripleo projects only
15:16:08 <EmilienM> (and puppet modules I believe)
15:16:17 <pabelanger> rsync can handle symlinks
15:16:20 <dmsimard> jpena: it could be scripted to get the hash first and then sync the hash
15:16:29 <dmsimard> jpena: versus rsync'ing /current/ directly
15:16:52 <jpena> would it be possible to do it the other way around? I mean, just like we're doing today from the internal instance to the public one
15:17:09 <pabelanger> not sure I follow
15:17:16 <jpena> ok, let me step back a bit
15:17:44 <jpena> we currently have 2 instances: one inside the ci.centos.org network (not public), which builds the packages, and a public one (trunk.rdoproject.org)
15:18:08 <jpena> for every package we build, we rsync the contents of the newly created repo (including symlinks) from the internal instance to the public one
15:18:28 <jpena> kind of a continuously incremental backup
15:19:07 <pabelanger> k
15:19:16 <jpena> so if we want to mirror that structure from the outside, we could either run rsync on a cron (which means scanning through an awful lot of directories)
15:19:17 <dmsimard> jpena: do we plan on keeping things the same once we move to rdo-cloud, though ? Even without considering the nodepool build setup, I remember discussing about potentially exposing rsync to make it easy to mirror trunk.rdo and then provide a mirrorlist instead of a mirror so have built-in HA
15:19:56 <jpena> dmsimard: that would depend on storage performance, and how long it takes to traverse all directories for rsync
15:20:03 <pabelanger> jpena: yes, this is how we do it today with all out sources. Crontab every x mins, then rsync.
15:20:30 <dmsimard> jpena: ephemeral performance on rdo cloud is better than ci.centos and volume is almost SSD levels
15:20:41 <dmsimard> (better than NFS)
15:21:14 <pabelanger> this would also lower stress on trunk.rdoproject.org too, since openstack would no longer hit it, but use our mirror
15:21:51 <jpena> pabelanger: that would also mean that there would be some lag between a package being built and available for upstream jobs
15:22:05 <apevec> pabelanger, there would be rsync stress on trunk.rdo
15:22:06 <jpena> I'm fine with that if tripleo is fine
15:22:31 <pabelanger> there is lag will all our mirrors today in openstack-infra. CUrrently 2 hours, but we can reduce that lower
15:22:37 <apevec> to traverse all the folder tree
15:22:49 <jpena> that traversal is what I'm afraid of
15:23:08 <jpena> we can run a quick check and see if it's severe or not
15:23:17 <pabelanger> it could be made more efficient, we a subset of directories need to be mirrored
15:23:26 <pabelanger> either by providing an index file
15:23:35 <trown> 2 hours is not much if it means greater stability in the jobs
15:24:09 <trown> it does create a window where CI is broken if we merge patches with Depends-On though
15:24:20 <pabelanger> yes, that is basically a recheck if a job fails because it failed to download something
15:25:06 <apevec> it's hashed dir tree, so you cannot have subset
15:25:26 <pabelanger> apevec: but you can provide an index of the hash
15:25:34 <pabelanger> that is how we mirror things like pypi
15:25:51 <pabelanger> we can setup an incremental rsync, as long as hashes don't change
15:25:53 <amoralej> can rsync follow symlinks?, we could only sync content of current, current-tripleo, etc.. and only the files symlinked, not the entire hashed structture
15:25:58 <jpena> pabelanger: I don't think we can. Each hashed directory has symlink to other hashed directories
15:26:01 <apevec> not sure what do you mean by index of the hash?
15:26:13 <apevec> yeah, you need them all
15:26:29 <apevec> b/c of cross-symlinks
15:27:10 <dmsimard> need to brb
15:27:15 <pabelanger> so, your concern is HDD preformence on trunk.rdoproject.org?
15:27:35 <jpena> I'm more concerned about CPU/memory usage during rsync directory traversal
15:27:43 <apevec> let jpena measured, as he suggested
15:27:47 <apevec> measure it*
15:27:51 <jpena> so, let's make a plan
15:27:55 <apevec> not measure jpena :)
15:28:04 <jpena> 1- I'll setup rsync and try once, to see the overhead
15:28:06 <pabelanger> yes, agree. if we setup rsync, we can do a test and see results
15:28:19 <jpena> 2- if it works fine, let's enable it and monitor for a while
15:28:39 <apevec> jpena, ad1) you could just test rsync via ssh
15:28:58 <jpena> apevec: I'd rather test with the same tools we will use at the end
15:29:18 <apevec> true that
15:29:33 <jpena> btw, are we syncing only for master (pike), or also stable branches?
15:29:38 <apevec> but cpu/io is the same, ssh vs rsyncd
15:29:47 <pabelanger> jpena: everything that tripleo uses
15:29:49 <apevec> stabel too
15:30:00 <jpena> ok
15:30:01 <pabelanger> depending on network is what we are trying to avoid
15:30:02 <apevec> yeah, tripleoci needs it ok
15:30:02 <trown> tripleo uses stable from current
15:30:14 <apevec> trown, even in ocata ?
15:30:27 <apevec> there is ocata/current-tripleo now
15:30:36 <pabelanger> is there an estimate on when the test / results for this could be done?
15:30:43 <pabelanger> trying to plan out things I need to work on upstream
15:30:55 <pabelanger> also, how much data is there?
15:30:56 <trown> apevec: it has to... it is at least a day before promotion can happen
15:31:05 <pabelanger> 1TB?
15:31:22 <trown> if any depends-on patches merge CI will be broken until promotion
15:31:26 <jpena> we have a 300 GB volume, currently using ~200GB
15:31:41 <pabelanger> okay, that isn't bad
15:31:55 <jpena> I'll get it tested before the end of the week
15:32:28 <pabelanger> okay, can we have an action to have something for next meeting?
15:32:33 <jpena> sure
15:32:35 <pabelanger> and follow
15:32:43 <pabelanger> if this fails, what would be the backup plan?
15:33:17 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-head-current @ http://tinyurl.com/hcnq3ll |#| Build failure on centos7-master-head/current: ironic-ui, python-openstackclient, networking-bagpipe: http://trunk.rdoproject.org/centos7-master-head/report.html
15:33:34 <jpena> is there a chance to do push instead of pull? We could do small rsyncs using SSH from the internal instance to anywhere, just like we do to the public instance
15:34:04 <pabelanger> no, we wouldn't allow external systems access to our AFS
15:34:20 <pabelanger> I mean, if DLRN was building this upstream in openstack-infra we would
15:34:33 <pabelanger> but not from an external service
15:35:36 <jpena> ok, then let's see if it works as-is, and think of alternatives in the meantime
15:35:43 <jpena> #action jpena to test rsync overhead on DLRN instance
15:36:01 <jpena> #action jpena to follow up with pabelanger after tests, to setup rsync with openstack infra
15:36:14 <pabelanger> okay, cool. Ya, this is a high priory to keep tripleo jobs working, as any help is great
15:36:56 <jpena> should we move to the next topic?
15:37:11 <pabelanger> ++
15:37:22 <jpena> #topic Enabling unit tests in complex packages
15:38:06 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-head-current @ http://tinyurl.com/hcnq3ll |#| Build failure on centos7-master-head/current: ironic-ui, python-openstackclient, networking-bagpipe: http://trunk.rdoproject.org/centos7-master-head/report.html
15:38:25 <jpena> earlier today, I had to propose a patch to nova-distgit (https://review.rdoproject.org/r/6062) to skip unit tests for the nova package
15:38:51 <jpena> enabling them makes package build time to be ~45 mins, and we had about 30 patches merged to Nova in March 29, so...
15:38:59 <jpena> flepied ^^
15:39:20 <apevec> heh just missed him
15:40:00 <apevec> so the problem is b/c we are currently single-threaded in dlrn
15:40:17 <jpena> well, we could have multiple threads, the code is there
15:40:19 <apevec> but also, could unittest be optimized
15:40:37 <apevec> how long does unittest job take upstream?
15:40:43 <trown> how useful are the unittests at the package build level? have we caught any major issues there?
15:40:56 <apevec> they would
15:41:05 <jpena> we catch several new build requirements with unit tests
15:41:12 <apevec> it actually exercises the code paths
15:41:13 <EmilienM> (dependencies I think)
15:41:19 <trown> ah right dependencies
15:41:48 <amoralej> http://logs.openstack.org/37/451137/1/check/gate-nova-python27-ubuntu-xenial/5db6606/console.html < 10mins
15:42:02 <amoralej> with 8 workers
15:42:14 <amoralej> Ran: 14816 tests in 534.0000 sec.
15:42:55 <jpena> we also catch cross-project dependencies, for example https://bugs.launchpad.net/horizon/+bug/1676298
15:42:55 <openstack> Launchpad bug 1676298 in OpenStack Dashboard (Horizon) "Need to remove usage of novaclient.v2.security_group_rules" [Undecided,Confirmed]
15:42:55 <EmilienM> 9 minutes. But do we have 8 workers in rdo infra?
15:43:23 <jpena> the DLRN instance has 8 cpus, but that means 100% cpu usage just for that package build (and no other workers)
15:43:39 <EmilienM> ouch
15:43:57 * jpena accepts hardware donations
15:44:27 <apevec> jpena, rdocloud!
15:44:38 <jpena> I can't wait to have it available :D
15:45:16 <jpena> anyway, this was mainly FYI. I know there was some initiative in the works to enable unit tests in packages, but we'll have to wait a bit
15:45:53 <jpena> next?
15:47:32 <jpena> #topic Automation of CBS tagging using gerrit workflow
15:47:38 <jpena> amoralej, is that yours?
15:47:41 <amoralej> yes
15:47:59 <amoralej> that's mainly for awareness, we are working to automate CBS tagging
15:48:34 <amoralej> that we'd use both for dependencies and stable builds of openstack packages
15:48:50 <amoralej> we'll add new metadata to rdoinfo and a separate file for dependencies
15:49:15 <amoralej> and i guess some changes in rdopkg
15:49:24 <amoralej> jruzicka ^ i'll need your help
15:49:55 <amoralej> the idea is to run CI gate jobs before pushing tags
15:50:19 <rdogerrit> Merged openstack/cinder-distgit: Update to 9.1.3  https://review.rdoproject.org/r/6070
15:50:29 <apevec> actually, we may move some code from rdopkg to rdoinfo repo
15:51:49 <amoralej> i'm preparing a first proposal for metadata and adding deps, i'll send it for review soon
15:52:28 <amoralej> that's it from me, unless there is any question
15:52:59 <trown> cool
15:53:36 <jpena> #topic chair for next meeting
15:53:41 <jpena> any volunteer?
15:54:56 <trown> I can chair next week
15:55:03 <trown> its been a while :P
15:55:07 <jpena> great! thx :)
15:55:19 <jpena> #action trown to chair next meeting
15:55:22 <jpena> and finally
15:55:23 * EmilienM chair rotation is cool. I'll probably steal the idea for tripleo :P
15:55:29 <jpena> #topic open floor
15:55:36 <trown> EmilienM: totally should
15:55:50 <EmilienM> trown: cool. Thanks for chairing tripleo meeting next week :D
15:55:54 <trown> lol
15:56:05 <trown> maybe not next week since I am chairing this on
15:56:06 <trown> one
15:57:49 <jpena> if there is nothing else to talk about, we can get 2 minutes back
15:57:56 <jpena> 3
15:57:57 <jpena> 2
15:57:57 <jpena> 1
15:58:02 <jpena> #endmeeting