15:02:20 <chandankumar> #startmeeting RDO meeting - 2017-03-08
15:02:21 <openstack> Meeting started Wed Mar  8 15:02:20 2017 UTC and is due to finish in 60 minutes.  The chair is chandankumar. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:24 <openstack> The meeting name has been set to 'rdo_meeting___2017_03_08'
15:02:30 <number80> o/
15:02:41 <chandankumar> Please add your meeting agenda on meeting etherpad
15:02:51 <chandankumar> #link https://etherpad.openstack.org/p/RDO-Meeting
15:02:56 <dmellado> o/
15:03:01 <chandankumar> #topic rollcall
15:03:06 <rbowen> \0/
15:03:08 <dmsimard> \o
15:03:13 <dmellado> \o/
15:03:13 <amoralej> o/
15:03:20 <chandankumar> #chair number80 dmellado rbowen dmsimard amoralej
15:03:20 <jpena> o/
15:03:21 <openstack> Current chairs: amoralej chandankumar dmellado dmsimard number80 rbowen
15:03:29 <chandankumar> #chair jpena
15:03:34 <openstack> Current chairs: amoralej chandankumar dmellado dmsimard jpena number80 rbowen
15:03:53 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-head-current @ http://tinyurl.com/hcnq3ll |#| Build failure on centos7-master-head/current: diskimage-builder: http://trunk.rdoproject.org/centos7-master-head/report.html
15:05:08 <chandankumar> so starting with the first topic
15:05:23 <chandankumar> #topic RDO Pike testdays
15:05:42 <chandankumar> tentative schedule here: https://www.redhat.com/archives/rdo-list/2017-March/msg00013.html
15:05:43 <chandankumar> Review for conflicts (major holidays, etc)
15:06:01 <chandankumar> rbowen: please go ahead
15:06:18 <rbowen> I posted tentative dates, each one one week out from milestones.
15:06:35 <rbowen> I just need people to have a look and ensure that there's no conflict with major holidays, etc.
15:06:42 <rbowen> I didn't see any.
15:07:01 <rbowen> And, as always, we need to continue improving the test day "playbook" and test instructions, so that more people can participate.
15:07:02 <rbowen> Thanks.
15:07:38 <chandankumar> rbowen: Thanks,
15:07:46 <chandankumar> Any queries on dates?
15:08:11 <rbowen> #link https://www.redhat.com/archives/rdo-list/2017-March/msg00013.html
15:08:15 <jpena> I haven't found any conflict
15:08:26 * chandankumar not also
15:08:35 <rbowen> ok, thanks.
15:09:16 <chandankumar> i have just one query, how we are planning to test openstack upstream installers docs with the test day?
15:10:03 <rbowen> That's a good suggestion. We should put that in the test matrix.
15:10:12 <chandankumar> i mean this doc https://docs.openstack.org/project-install-guide/ocata/rdo-services.html
15:10:48 <rbowen> We should link directly to that doc in the test matrix, and ask that people step through each one of those items as part of test day.
15:11:13 <chandankumar> that would be a good idea
15:11:33 <rbowen> This is a really substantial document, and it's important that the steps be verified for each new release, for sure.
15:12:06 <rbowen> #action Rich will add upstream install doc https://docs.openstack.org/project-install-guide/ocata/rdo-services.html to RDO test day instructions.
15:12:25 <chandankumar> moving to next topic
15:12:43 <chandankumar> #topic CentOS infrastructure planned outage March 14th
15:12:54 <dmsimard> That's mine
15:12:58 <chandankumar> This is a reminder CentOS infrastructure planned outage March 14th
15:13:11 <chandankumar> https://lists.centos.org/pipermail/ci-users/2017-March/000502.html
15:13:18 <chandankumar> dmsimard: please go ahead
15:13:24 <dmsimard> This is yet another reminder because it's a significant outage and you won't be able to way you weren't warned :)
15:13:41 <dmsimard> CentOS infrastructure bits like CBS or ci.centos.org will be unavailable
15:13:57 <dmsimard> weshay, trown|outtypewww, myoung, EmilienM, adarazs ^
15:14:08 <dmsimard> Please prepare accordingly (i.e, oooq gate jobs)
15:14:29 <adarazs> ack, thanks!
15:14:52 <dmsimard> Nothing else on that topic
15:15:00 <rbowen> Super short agenda today.
15:15:48 <chandankumar> dmsimard: and next is also yours!
15:15:56 <chandankumar> moving to next topic
15:15:59 <rbowen> Oh, it grew. :-)
15:15:59 <dmsimard> yeah.
15:16:03 <chandankumar> #topic Beta testing RDO cloud
15:16:17 <dmsimard> In case you didn't know, there's this RDO cloud thing that will solve all of our problems
15:16:24 <dmsimard> All of them
15:16:41 <myoung> dmsimard: ack thx
15:16:43 <dmsimard> and we're beta testing it right now, we reported a few things here and there, things are progressing
15:17:01 <dmsimard> I'll be testing nested virtualization CI on it today
15:17:54 <dmsimard> On that end, a sub-topic would be a FYI that we're working on moving part of the ci.centos.org workload to RDO cloud (weirdo jobs)
15:18:26 <dmsimard> weirdo jobs do not require bare metal hardware to run and the ci.centos.org mixed hardware yields unreliable results -- this will help make those jobs more consistent and reliable while also freeing up additional capacity for tripleo-based jobs
15:19:11 <dmsimard> that's it for me on that topic unless people have questions
15:20:30 <chandankumar> dmsimard: just stupid question do we have public url to access RDO cloud and participate in beta testing of RDO cloud?
15:21:01 <dmsimard> RDO cloud is a public cloud but the scope of the testing is limited to certain people/systems right now, it's not an "open beta"
15:21:20 <chandankumar> dmsimard: Thanks :-)
15:21:38 <dmsimard> I don't really have details on when/how the availability might be expanded in the future as the rollout progresses
15:22:11 <chandankumar> if no any queries on this topic moving to next topic
15:22:25 <chandankumar> #topic updates on tempest
15:22:35 <chandankumar> that's mine
15:22:55 <chandankumar> oooq-extras got the support of tempest rpm and new python-tempestconf
15:23:05 <chandankumar> https://github.com/openstack/tripleo-quickstart-extras/commit/a1aab3eacc3b086ca16476ee696220463c69f1cb
15:23:29 <chandankumar> blog post related to all tempest rpm story is coming soon.
15:24:25 <chandankumar> that's it from this topic
15:24:43 <chandankumar> Any queries related to tempest oooq-extras patch?
15:25:26 <chandankumar> if not then moving ahead with next topic
15:25:42 <chandankumar> #topic Enabling unit tests for RDO packages
15:26:23 <chandankumar> In Pike release, we are targetting to enable unittests  for all RDO python packages
15:26:43 <chandankumar> for that i have created an etherpad to track the initial thoughs
15:26:54 <chandankumar> #link https://review.rdoproject.org/etherpad/p/enable_unittest
15:27:16 <chandankumar> Feel free to add your queries there so that we can plan together and finish it
15:28:39 <chandankumar> Any queries related to this topic ?
15:29:35 <jpena> just one: be careful with large projects. keystone or neutron have lots of unit tests, and build times can be quite high
15:29:45 <jpena> we've seen that in the upstream rpm-packaging project
15:30:35 <chandankumar> jpena: how we are solving the above issue in upstream rpm-packaging project?
15:30:48 <chandankumar> for larger project
15:31:31 <jpena> for neutron we're only using a subset of the unit tests (using --black-regex) to keep build time reasonable, we found issues when unit tests were running for more than 1h
15:32:22 <jpena> in general, you'll need several CPUs to run unit tests in parallel
15:32:32 <chandankumar> yes
15:32:44 <jpena> so the more CPU power the merrier
15:33:33 <chandankumar> so basically we need to figure out what sets of unit tests we need to run without using more cpus.
15:34:10 <jpena> kind of. It we see that build time goes too high, we'll have to find a balance
15:34:44 <chandankumar> yup.
15:35:42 * chandankumar takes a note of that.
15:35:50 <migi> chandankumar: I had some comparison and adding more then 4vCPUs for unit is not going to speed things dramatically
15:37:14 <chandankumar> migi, one way might be optimizing openstack services unit tests code?
15:37:35 <migi> chandankumar: don't understand what you mean
15:38:37 <chandankumar> i was saying about running all tests and check which one takes much time then doing code optimization might reduces some time
15:39:08 <jpena> chandankumar: maybe, but that's probably an overkill. It's just that there are many tests being executed
15:39:11 <migi> chandankumar: ah ok, it's part of each run so it's already there
15:39:23 <migi> but it's overkill as jpena said
15:40:08 <chandankumar> yup,
15:40:32 <chandankumar> that's it from my side on this topic.
15:40:47 <migi> chandankumar: see: http://logs.openstack.org/99/428199/9/check/gate-nova-python27-ubuntu-xenial/f919e28/console.html
15:40:57 <migi> 2017-03-08 13:49:51.574019 | Slowest Tests
15:42:28 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| Build failure on centos7-newton/current: diskimage-builder: http://trunk.rdoproject.org/centos7-newton/report.html
15:42:43 <chandankumar> migi: 19.0 for a nova.tests.unit.db.test_migrations.TestNovaMigrationsPostgreSQL.test_walk_versions
15:42:51 <chandankumar> too much time
15:42:58 <jruzicka> py.test --profile-svg ^^
15:43:33 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/zqowcwb |#| Build failure on centos7-master/current: diskimage-builder: http://trunk.rdoproject.org/centos7-master/report.html
15:43:59 <chandankumar> migi: while enabling unit tests, let's find the what tests to be blacked list and what to run in %check.
15:44:10 <chandankumar> during review process
15:44:14 <chandankumar> what you say?
15:47:04 <migi> chandankumar: don't think we should do that, it's decision based on timing and not importance of the test
15:47:49 <migi> chandankumar: better off log the bug upstream to fix the root cause (slow test) and possibly fix it if it's real issue
15:48:07 <chandankumar> migi: sure
15:48:41 <migi> chandankumar: as this example see: https://bugs.launchpad.net/keystone/+bug/1602407
15:48:41 <openstack> Launchpad bug 1602407 in OpenStack Identity (keystone) "MySQL tests significantly slower than PostgreSQL tests" [Undecided,Expired]
15:50:23 <chandankumar> migi: yes, i will take a look.
15:51:04 <chandankumar> moving to next topic if not further queries
15:51:05 <apevec> migi, is that good example, bug expired :)
15:51:35 <apevec> but let's move on,
15:51:39 <migi> apevec: heh - the importance of such things
15:51:41 <apevec> and move discussion to etherpad
15:51:45 <chandankumar> #topic What I Did In Ocata" interviews
15:51:58 <chandankumar> rbowen: please go ahead
15:52:00 <rbowen> I did 23 interviews at the PTG, on the topic of "what I did on Ocata"
15:52:09 <rbowen> I've edited 7 so far, and they are appearing at https://www.youtube.com/playlist?list=PLOuHvpVx7kYksG0NFaCaQsSkrUlj3Oq4S
15:52:22 <rbowen> If you would like to do an interview and missed that opportunity, please ping me.
15:52:43 <apevec> rbowen, should we also extend to RDO users?
15:52:46 <apevec> not just developers
15:52:47 <rbowen> These interviews will be turned into blog posts, as well as an over all "what's new in Ocata" post that will run several places.
15:52:56 <apevec> i.e. interesting RDO implementations
15:53:01 <rbowen> Yes, I would very much like to have a sampling of users, too. That would be AWESOME.
15:53:10 <apevec> mnaser, ^ :)
15:53:25 <rbowen> So email me - rbowen@redhat.com - and we'll do the interview online, and add it to the queue.
15:53:28 <rbowen> Thanks.
15:53:32 <rbowen> #link https://www.youtube.com/playlist?list=PLOuHvpVx7kYksG0NFaCaQsSkrUlj3Oq4S
15:53:35 <mnaser> o/
15:53:37 <apevec> mnaser has internal DLRN running iirc
15:54:02 <apevec> which might be interesting for other to hear about
15:54:04 <mnaser> i think we discussed this once before, but yes, we run rdo all across (deployed with ansible) and we have internal dlrn for specific packages (horizon being the one we use the most)
15:54:09 <apevec> for horizon customizations
15:54:14 <rbowen> We're looking for roughly 10 minutes - not a major thing
15:54:35 <mnaser> i can do that.  i'll drop an email
15:54:46 <rbowen> This way, folks that don't have time/inclination to blog about stuff still have an opportunity to get their story published. :-)
15:54:59 <rbowen> </done>
15:55:00 <apevec> blog-aaS ?
15:55:05 <rbowen> Exactly. :)
15:55:34 <rdogerrit> Alfredo Moralejo created openstack/tripleo-image-elements-distgit: Update to 6.0.0  https://review.rdoproject.org/r/5659
15:55:43 <chandankumar> rbowen: Thanks :-)
15:56:04 <rdogerrit> Alfredo Moralejo created openstack/tripleo-heat-templates-distgit: Update to 6.0.0  https://review.rdoproject.org/r/5660
15:56:14 <chandankumar> before moving to open floor, one last topic
15:56:22 <chandankumar> #topic chair for next meeting
15:56:30 <rdogerrit> Alfredo Moralejo created openstack/instack-undercloud-distgit: Update to 6.0.0  https://review.rdoproject.org/r/5661
15:56:30 <apevec> #info tripleo Ocata final coming ^ :)
15:56:53 <rdogerrit> Alfredo Moralejo created openstack/tripleo-puppet-elements-distgit: Update to 6.0.0  https://review.rdoproject.org/r/5662
15:57:04 <jpena> chandankumar: I can chair the next meeting
15:57:10 <apevec> amoralej, you're spaming the meeting minutes :)
15:57:12 <chandankumar> ah cool!
15:57:19 <amoralej> it's my script, sorry
15:57:25 <chandankumar> #action jpena to chair for next meeting
15:57:33 <chandankumar> #topic open floor
15:57:54 <chandankumar> If any topic is missed feel free to bring it now.
15:58:08 <mnaser> i have a small thing i'd like to ask: we run into a lot of really weird issues a lot of times with openstack that generally would have fixes backported into stable
15:58:32 <mnaser> depending on the project, takes a while to get a release out and some of these issues can be weird edge conditions
15:59:01 <rdogerrit> User yrobla proposed openstack/diskimage-builder-distgit: Adapt package to work with new dib changes  https://review.rdoproject.org/r/5651
15:59:06 <mnaser> example: we ran into an issue upgrading nova to newton that was an edge case which was fixed in stable/newton but not out yet, is there any consideration to try and get those in (i'd say as a service for other rdo users)
15:59:22 <mnaser> or is it a "lets be a bit more persistent on getting tarball releases out"
15:59:39 <apevec> mnaser, if patch is merged on stalbe, you could use RDO trunk stable builds as a  hotfix
15:59:56 <mnaser> apevec that completetly escaped my mind at the time, that's an excellent suggestion
15:59:57 <apevec> trunk NVRs should be higher than release
16:00:13 <jpena> mnaser: what apevec said. We try hard to ensure package NVRs allow upgrades
16:00:19 <apevec> and when next stable point release comes, it's again higher b/c it's -1
16:00:23 <apevec> and trunk is -0
16:00:26 <mnaser> you're right, that's probably the best way to go about doing it
16:01:09 <jpena> I tried to document that use case in https://www.rdoproject.org/what/repos/
16:01:19 <apevec> we currently have issues with puppet-* but that's going to be fixed today
16:01:27 <mnaser> we just got a little hectic mid-upgrade, but thanks for addressing this (and thanks for the rdo folks for great packaging as usual, we're very happy!)
16:02:16 <chandankumar> times up, closing the meeting now, if anything left we can discuss after the meeting.
16:02:21 <chandankumar> #endmeeting