15:00:11 <imcsk8> #startmeeting RDO meeting (2016-05-25)
15:00:11 <zodbot> Meeting started Wed May 25 15:00:11 2016 UTC.  The chair is imcsk8. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:11 <zodbot> The meeting name has been set to 'rdo_meeting_(2016-05-25)'
15:00:12 <openstack> Meeting started Wed May 25 15:00:11 2016 UTC and is due to finish in 60 minutes.  The chair is imcsk8. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:15 <openstack> The meeting name has been set to 'rdo_meeting__2016_05_25_'
15:00:21 <imcsk8> #topic roll call
15:00:22 <number80> o/
15:00:24 <jpena> o/
15:00:27 <elmiko> o/
15:00:33 <trown> o/
15:00:36 <dmsimard> \o
15:00:44 <dmsimard> oh, openstack
15:00:45 <chandankumar> \o
15:00:46 <leifmadsen> \o\
15:00:52 <leifmadsen> /o/
15:01:01 <imcsk8> #chair number80 jpena elmiko trown dmsimard chandankumar leifmadsen
15:01:01 <zodbot> Current chairs: chandankumar dmsimard elmiko imcsk8 jpena leifmadsen number80 trown
15:01:02 <jruzicka> \o\
15:01:02 <openstack> Current chairs: chandankumar dmsimard elmiko imcsk8 jpena leifmadsen number80 trown
15:01:08 <imcsk8> #chair jruzicka
15:01:09 <zodbot> Current chairs: chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 trown
15:01:11 <openstack> Current chairs: chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 trown
15:01:21 <apevec> o/
15:01:26 <imcsk8> #chair apevec
15:01:27 <zodbot> Current chairs: apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 trown
15:01:27 <openstack> Current chairs: apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 trown
15:01:39 <imcsk8> ok, let's start...
15:01:44 <apevec> dmsimard, still not merged that infra fix?
15:01:53 <fbo> number80, It should work now please retry.
15:02:01 <number80> fbo: thanks!
15:02:13 <imcsk8> first topic:
15:02:16 <imcsk8> #topic DLRN instance migration to ci.centos infra
15:02:39 <jpena> this is a follow-up from last week
15:02:52 <apevec> still blocked on tripleoci promotion?
15:02:53 <jpena> dmsimard trown: how's the state for the promote job?
15:03:09 <dmsimard> trown said there's a review bound to be sent today
15:04:06 <dmsimard> the buildlogs cdn doesn't seem to have been updated by KB :(
15:04:14 <dmsimard> I'll poke him some more.
15:05:25 <imcsk8> anything else for this for topic?
15:05:56 <apevec> actions
15:06:10 <apevec> #action dmsimard will poke KB and trown some more
15:06:14 <rbowen> Sorry, I am here, but everything crashed.
15:06:25 <imcsk8> #chair rbowen
15:06:25 <zodbot> Current chairs: apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 rbowen trown
15:06:26 <openstack> Current chairs: apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 rbowen trown
15:06:55 <imcsk8> ok, let's continue with next topic. suddenly there are a lot!
15:06:57 <dmsimard> When the tripleoci new promotion is setup
15:07:09 <dmsimard> We need to have tripleo switch to the buildlogs repo
15:07:20 <dmsimard> I don't know what's involved in taht
15:07:24 <dmsimard> derekh would probably know
15:07:45 <dmsimard> tripleo can be entirely moved by themselves, I think they'll have all they need
15:08:36 <apevec> I think trown and derekh are closely syncing
15:08:49 <dmsimard> we can move on
15:08:52 <apevec> but I can doulbe-check
15:09:11 <apevec> #action apevec to check w/ derekh about tripleoci
15:09:12 <imcsk8> apevec: should we add an action for that?
15:09:18 <apevec> taken
15:09:21 <imcsk8> ok, let's continue...
15:09:31 <imcsk8> #topic Updates from RDO Bug Triage day
15:09:40 <chandankumar> RDO Bug Triage went well.
15:09:40 <amoralej> sorry o/
15:09:49 <chandankumar> Some stats :
15:09:50 <chandankumar> 433 Bugs affected on Bug Triage Day.
15:09:50 <chandankumar> 398 Bugs closed on Bug Triage Day out of which 365 EOL Bugs closed by automation script.
15:09:50 <chandankumar> 35 Bugs triaged on Bug Triage Day and most of the EOL triaged bugs closed due to EOL automation script
15:09:50 <chandankumar> 22 people participated in the RDO Bug Triage Day.
15:09:50 <chandankumar> EOL automation script needs some improvement to skip components and check for version while closing.
15:09:52 <chandankumar> I will be moving these scripts to rdostats very soon.
15:09:58 <chandankumar> https://www.rdoproject.org/blog/2016/05/rdo-bug-triage-day-event-report/
15:10:06 <imcsk8> #chair amoralej
15:10:06 <openstack> Current chairs: amoralej apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 rbowen trown
15:10:06 <zodbot> Current chairs: amoralej apevec chandankumar dmsimard elmiko imcsk8 jpena jruzicka leifmadsen number80 rbowen trown
15:10:09 <chandankumar> that's it from my side.
15:10:28 <apevec> re. post-mortem, let's do it now and here
15:10:43 <derekh> apevec: iirc trown is going to switch our promotion to talk to jenkins instead of directly ssh'ing to the delorean node, so it should be sorted soon
15:10:48 <chandankumar> * first is mitaka component is missing.
15:10:49 <apevec> points from chandankumar message: EOL script needs improvements
15:11:08 <imcsk8> apevec: sorry i don't understand what doing post-mortem means
15:11:18 <apevec> imcsk8, retrospective,
15:11:24 <apevec> what went good and what wrong
15:11:27 <imcsk8> oh ok, thanks
15:11:30 <chandankumar> we need to get it as soon as possible.
15:11:31 <tosky> imcsk8: literaly "after death", it's latin
15:11:43 <imcsk8> tosky: hehehe that i get
15:12:04 <apevec> chandankumar, yes, I'm always inches away from filing that bugzilla-request
15:12:24 <chandankumar> apevec, need a way to sync version also in bugzilla
15:12:25 <tosky> imcsk8: right, latin languages for you too
15:12:37 <chandankumar> for each target release
15:12:57 <apevec> chandankumar, it needs more, I was discussing w/ number80 to have a script syncing rdoinfo and generating bugzilla-requests tickets
15:12:59 <rdobot> [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://uchiwa.monitoring.rdoproject.org/#/client/rdo-monitoring/master.monitoring.rdoproject.org?check=check-delorean-newton-current |#| Build failure on centos7-master/current: os-testr: http://trunk.rdoproject.org/centos7-master/report.html
15:13:23 <apevec> it also needs syncing new components, updated default maitainers
15:13:41 <apevec> this will be one of automation goals this cycle
15:13:48 <chandankumar> apevec, feel free to create comment in the gist https://gist.github.com/chkumar246/23237abe5203d963054092fd989a2b88
15:14:02 <number80> first step is to have a reporting page to detect discrepancies (no bz component or wrong default assignee)
15:14:11 <chandankumar> so that i can update the script based on the requirements.
15:14:29 <apevec> chandankumar, re. script - let's move it to rdostats as-is then have people send PRs there
15:14:54 <chandankumar> #action chandankumar to move rdo bug triage scripts to rdostats
15:14:54 <imcsk8> guys, i think we're drifting a little from the topic
15:15:12 <apevec> it has been moved to https://github.com/rdo-infra/rdostats
15:15:18 <apevec> so it's official now :)
15:15:44 <apevec> imcsk8, right, let's stick to triage day
15:15:52 <apevec> any other observations?
15:16:12 <imcsk8> i think we could do continuous triage
15:16:15 <number80> exempt Packages Reviews from the script :)
15:16:16 <chandankumar> apevec, we need to be more specific what we need to triage
15:16:33 <chandankumar> the rdo bug triage is outdated
15:16:45 <apevec> number80, CVE trackers too
15:16:49 <chandankumar> *the rdo bug triage wiki
15:16:51 <number80> *nods*
15:17:47 <chandankumar> one more improvement would be linking patches to bugzilla once merged get it closed automatically
15:18:43 <number80> Yes
15:18:55 <apevec> yeah, need to look at how it is done upstream w/ launchpad
15:19:03 <chandankumar> just a query are we keeping the triage definition of RDO similar to fedora one?
15:19:20 <apevec> iirc it's post commit zuul job ?
15:19:25 <imcsk8> can we close bugs without creating a package?
15:19:41 <apevec> chandankumar, I think we could simplify it
15:19:50 <amoralej> i think review.rdoproject.org is not available in external but trackers, would it be possible to add it?
15:20:21 <amoralej> for some cases, required changes are in distgits, no upstream
15:20:24 <apevec> amoralej, I'll ask bugzilla-requests
15:20:47 <apevec> #action apevec ask bugzilla-requests to add review.rdoproject.org in external trackers
15:21:07 <imcsk8> can we continue to the next topic?
15:21:18 <chandankumar> we need action for triage definition?
15:22:18 <dc> I have an issue deploying overcloud (mitaka) - it keeps failing at ControllerOvercloudServicesDeployment_Step4.
15:22:27 <imcsk8> chandankumar: should we follow up that on the ML?
15:22:36 <chandankumar> imcsk8, will work
15:22:37 <dc> Log references Error: Could not find dependent Service[keystone] for File[/etc/keystone/ssl/certs/signing_cert.pem] - any ideas?
15:22:48 <imcsk8> dc: sorry, we're on the middle of a meeting
15:23:15 <imcsk8> chandankumar: should i set the action to you?
15:23:37 <chandankumar> #action chandankumar to simplify triage definition for RDO through ML
15:23:51 <chandankumar> move to next topic?
15:23:58 <imcsk8> yes
15:24:09 <imcsk8> #topic Increase timeout for Packstack CI jobs
15:24:31 <jpena> this is something we've been discussing earlier today
15:24:44 <jpena> some Packstack CI jobs are taking longer than the current 60-minute timeout, and failing
15:25:07 <apevec> right, I don't think blindly increasing timeout is the right thing
15:25:26 <number80> (btw, when people edit agenda, please update etherpad color code)
15:25:49 <number80> (and try to use different colors)
15:26:08 <dmsimard> apevec: +1
15:26:27 <imcsk8> do we have more information on why are this timeouts happening?
15:26:29 <dmsimard> Jobs should be ending under 45 minutes, if they're not, it's an issue and we need to address it
15:26:50 <jpena> dmsimard: we may try to find some correlation between the timeouts
15:26:54 <jpena> e.g. cloud provider
15:27:02 <dmsimard> I've tried to reproduce the issue with puppet openstack scenario003 but without any success (it has been timing out >90 minutes)
15:27:28 <dmsimard> jpena: puppet scenario003 times out >90 minutes on ci.centos but I ran the exact same job manually on a ci.centos node and couldn't reproduce it
15:27:46 <jpena> dmsimard: do you have any logs from that?
15:27:53 <dmsimard> I will give some more attempts, now that you mention it, ci.centos does have a mix of amd and intel hardware
15:28:16 <dmsimard> jpena: no because the timeouted jobs are killed pretty hard apparently
15:28:26 <dmsimard> that's why I was hoping to reproduce the issue outside of a job
15:28:28 <amoralej> yes, dmsimard, that's one of the issues
15:28:29 <jpena> ok, so not even timestamps for each stdout line
15:28:37 <amoralej> no logs when jobs timed out
15:28:42 <dmsimard> no, because ansible only flushes the output once the task is finished
15:28:47 <dmsimard> and the task doesn't finish, it's killed
15:28:54 <dmsimard> you would have logs on logs.o.o, though
15:29:00 <dmsimard> since it's flushed for each line for puppet
15:29:30 <imcsk8> dmsimard: would - debug: var=out.stdout_lines be helpful?
15:29:37 <amoralej> but we at least have some timestamps
15:29:39 <amoralej> http://logs.openstack.org/62/318762/3/gate/gate-packstack-integration-scenario003-tempest-centos-7/3328605/console.html
15:29:53 <jpena> ok, let's try something: why don't we try to gather all CI-failed runs due to timeout, and try to find some correlation?
15:29:57 <dmsimard> The last time job run times went out of control like this, it was neutron including a bunch of tests in smoke
15:30:02 <dmsimard> Maybe it's something similar
15:30:38 <imcsk8> i'll try to reproduce on my end
15:30:48 <amoralej> in this case, we had 45 mins to run packstack, before starting to run empest
15:31:22 <dmsimard> That sounds slow, just the installation taking 45 mins.
15:31:46 <dmsimard> Considering tempest is usually finished by then
15:31:50 <jpena> that example screams "slow VM"
15:32:03 <imcsk8> slow network?
15:32:08 <dmsimard> A path to explore could be the introduction of openstack-infra of the new centos and rdo mirrors
15:32:26 <dmsimard> but I don't believe we use thesed
15:32:37 <dmsimard> damnit
15:32:51 <dmsimard> packstack master still uses mitaka current-passed-ci
15:33:14 <imcsk8> any actions?
15:33:28 <imcsk8> so we can proceed to the next topic
15:33:35 <openstackgerrit> David Moreau Simard proposed openstack/packstack: Bump repository to newton for master branch  https://review.openstack.org/321074
15:33:36 <dmsimard> my action ^
15:34:18 <imcsk8> #action dmsimard introduction of openstack-infra of the new centos and rdo mirrors
15:34:23 <dmsimard> nonono
15:34:27 <dmsimard> #undo
15:34:27 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x9746b90>
15:34:27 <zodbot> Removing item from minutes: ACTION by imcsk8 at 15:34:18 : dmsimard introduction of openstack-infra of the new centos and rdo mirrors
15:34:53 <dmsimard> I meant to say that openstack-infra recently added mirrors that are consumed by the VMs inside their infrastructure
15:35:07 <imcsk8> ohhh
15:35:10 <apevec> that should have speed up things :)
15:35:17 <imcsk8> indeed
15:35:23 <dmsimard> Yeah, but the matter of fact is that we don't use these (at least for RDO)
15:35:29 <dmsimard> UNLESS, they are conflicting
15:35:37 <dmsimard> i.e, they used to setup epel by default
15:36:31 <dmsimard> nah, doesn't seem to be any repo interference according to http://logs.openstack.org/59/318759/3/check/gate-packstack-integration-scenario003-tempest-centos-7/705e67c/logs/diag/yum_repolist_-v.txt.gz
15:36:52 <dmsimard> ok, I guess actions are general troubleshooting around jobs timing out
15:37:53 <imcsk8> do you think this should be followed up in the ML?
15:38:06 <imcsk8> that can be the action
15:38:34 <rdobot> [sensu] RESOLVED: master.monitoring.rdoproject.org - check-delorean-mitaka-current @ http://uchiwa.monitoring.rdoproject.org/#/client/rdo-monitoring/master.monitoring.rdoproject.org?check=check-delorean-mitaka-current |#| No build failures detected: http://trunk.rdoproject.org/centos7-mitaka/report.html
15:38:56 <apevec> yeah, we have a long list of topics today
15:39:15 <imcsk8> ok, let's continue
15:40:15 <imcsk8> #topic packaging guide status?
15:40:45 <number80> So what's blocking the draft?
15:40:48 <number80> jruzicka: ?
15:41:00 <jruzicka> nothing blocking
15:41:04 <leifmadsen> I'm happy to review anything, as I am interested in getting up to speed on packaging
15:41:05 <number80> it looks good enough to be published
15:41:18 <jruzicka> I was waiting for reviews
15:41:19 <leifmadsen> I can follow through directions, make sure it all works
15:41:24 <jruzicka> and then got busy with rdopkg
15:41:41 <jruzicka> "looks good enough to be published" sounds like a +1
15:41:44 <leifmadsen> jruzicka: link me and I will review after lunch
15:41:46 <number80> jruzicka: yeah
15:41:57 <leifmadsen> I'll +1 the hell outta it
15:42:20 <jruzicka> leifmadsen, https://github.com/yac/website/commit/ec80c29b378d1fef43b89589b65b188531465aa5
15:42:36 <leifmadsen> thanks, on my agenda for this afternoon
15:42:42 <number80> #action jruzicka send a PR for updated packaging guide
15:42:46 <number80> ;)
15:42:50 <leifmadsen> +1
15:42:50 <jruzicka> oh, apevec actually did review it :)
15:42:52 <leifmadsen> :D
15:42:58 <jruzicka> all right
15:43:11 <imcsk8> can we move on to the next topic?
15:43:13 <jruzicka> so I planned to move the current guid to -old
15:43:18 <number80> wfm
15:43:24 <jruzicka> for historic reference
15:43:24 <number80> imcsk8: yes
15:43:27 <jruzicka> or remove it alltogether?
15:43:31 <leifmadsen> remove
15:43:39 <leifmadsen> I hate duplicate docs :)
15:43:41 <rbowen> Remove. It's in revision control if people want old versions.
15:43:44 <leifmadsen> +1
15:43:45 <rbowen> Having duplicate can be very confusing.
15:43:56 <jruzicka> ack
15:44:03 <number80> #agreed remove old packaging doc
15:44:12 <jruzicka> very well
15:44:15 <imcsk8> ok, moving on...
15:44:28 <imcsk8> #topic stable updates
15:45:13 <number80> Liberty Keystone is blocked until upstream tags the repo
15:45:27 <number80> we have new trove/keystone update for Mitaka coming up
15:45:56 <number80> #action number80 cleanup the stable updates stack
15:46:28 <number80> EOF if nobody else have comments
15:46:38 <apevec> I have
15:47:08 <apevec> we need promotion pipeline against -testing repo, trown was working on jjb changes for that
15:47:31 <apevec> for Mitaka release we had a temporary pipeline, now we need to make it proper
15:47:36 <trown> ya, I needed to do some major cleanup of jjb first, which I accomplished yesterday
15:47:43 <apevec> cool!
15:47:56 <trown> next on my list of jjb fun is to make those -testing jobs
15:48:07 <apevec> #info trown did  major cleanup of jjb
15:48:18 <apevec> trown, jjb is fun?
15:48:44 <imcsk8> ok, moving on to the next topic
15:48:49 <apevec> imcsk8, one more
15:49:11 <trown> apevec: no I hate it... I can never get it right the first try either :P
15:49:15 <apevec> trown, related: w/ this pipeline we'll be able to publish updated CBS based 3o image on buildlogs right?
15:49:24 <trown> apevec: ya
15:49:41 <apevec> GA images went missing :)
15:49:56 <apevec> but no point reproducing them imho, we should just provide updated ones
15:50:03 <trown> ya, I just realized they are on buildlogs, so I can put them back
15:50:10 <number80> trown: I'll ping you later to figure out how we can retrieve CI status for stable promotion
15:50:21 <apevec> trown, any RCA why thing went missing on buildlogs?
15:50:26 <trown> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/mitaka/cbs/
15:50:32 <apevec> oh they were moved somewhere else?
15:50:47 <trown> apevec: no RCA from centos folks...
15:51:04 <apevec> ok, that's all, let's move on
15:51:05 <trown> no they were not moved... buildlogs syncs from artifacts
15:51:12 <imcsk8> #topic RDO bookmark update: new CLI - https://github.com/redhat-openstack/website/tree/master/source/images/bookmark
15:51:15 <rbowen> I need to do a new run of the RDO CLI bookmarks. I would appreciate it if a few people can look at the old one, make sure it's still correct, and suggest what else/different/less we should have on there.
15:51:17 <rbowen> #link https://github.com/redhat-openstack/website/tree/master/source/images/bookmark
15:51:21 <rbowen> EOL
15:51:39 <apevec> rbowen, how to send patches?
15:51:40 <flepied> misc: Duck: you should have root access to blogs.r.o
15:52:00 <apevec> it's binary blobs :)
15:52:01 <number80> #action everyone help review new RDO CLI bookmarks
15:52:07 <rbowen> That link is the git repo. But the file is a LibreOffice file, so it's not patches so much as just new versions. So ... I'm not sure.
15:52:13 <number80> https://github.com/redhat-openstack/website/tree/master/source/images/bookmar
15:52:14 <rbowen> I suppose you could email proposed changes and I can manage the process.
15:52:27 <rbowen> Yeah, binary blob is annoying for this.
15:52:50 <apevec> should we put it in google doc?
15:52:50 <rbowen> Also see the README for the fonts that you have to install to make it render correctly.
15:52:56 <apevec> or would formatting fall apart?
15:53:04 <rbowen> Yes, the formatting would fall apart.
15:53:11 <rbowen> But I could transcribe it, and then put it back afterwards.
15:53:27 <rbowen> I'll follow up to the mailing list thread with a google doc address. good idea.
15:53:53 <imcsk8> next topic guys, we're almost top of the hour
15:54:03 <apevec> #action rbowen  follow up to the mailing list thread with a google doc
15:54:06 <rbowen> +1
15:54:07 <apevec> yes, next
15:54:15 <imcsk8> #topic review and clean old issues from website
15:54:38 <rbowen> Not much to discuss here, I think. We're just reviewing old tickets and closing the ones that are no longer relevant. we're down to 9 now.
15:55:05 <imcsk8> rbowen: i'll take a look and see if i can help
15:55:09 <apevec> thanks, let's move on
15:55:19 <imcsk8> #topic Newton 1 test day: https://www.rdoproject.org/testday/newton/milestone1/
15:55:30 <apevec> I think dates are not right: https://github.com/redhat-openstack/website/commit/2bebcf0e1310dc2220721e5ad1c2b347016836a0#commitcomment-17615469
15:55:42 <apevec> upstream is next week, so RDO is week after
15:55:49 <rbowen> As usual, help is appreciated updating the test cases.
15:55:54 <rbowen> Oh, did I get the dates wrong. Crap.
15:55:55 <apevec> June 9/10
15:55:57 <trown> ya, I thought about that
15:56:09 <rbowen> ok, I'll fix asap and fix scheduled tweets etc.
15:56:13 <trown> tripleo is trying to have something by end of next week for N1
15:56:30 <apevec> #info  tripleo is trying to have something by end of next week for N1
15:56:38 <imcsk8> ok, last topic...
15:56:38 <rbowen> #action rbowen to fix dates on Newton 1 test days.
15:57:00 <trown> and by something I mean actual releases!
15:57:00 <imcsk8> #topic [info] rdopkg-0.37 released with bugfixes and improvements, please test
15:57:37 <apevec> jruzicka++ nice job on hotfix release
15:57:55 <apevec> also good to see contributions from !jruzicka
15:58:29 <jruzicka> \o/
15:58:31 <jruzicka> indeed
15:58:38 <imcsk8> jruzicka++
15:58:39 <zodbot> imcsk8: Karma for jruzicka changed to 6 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:58:42 <jruzicka> biggest highlight would be
15:58:50 <jruzicka> ktdreyer's RFE
15:59:12 <imcsk8> ok, let's move on so we can finish on time
15:59:15 <jruzicka> yup
15:59:19 <imcsk8> #topic chair for next meeting
15:59:56 <imcsk8> anybody?
16:00:04 <amoralej> i can do it
16:00:05 <apevec> there should be meetbot command to pick random nick from all present :)
16:00:16 <imcsk8> hehehe
16:00:26 <number80> it would be easy to cheat then :)
16:00:34 <imcsk8> #action amoralej to chair next meeting
16:00:40 <imcsk8> can we finish?
16:00:47 <jruzicka> we have to!
16:00:51 <apevec> number80, bot could remember nic renames :)
16:00:54 <imcsk8> #endmeeting