17:01:27 <krtaylor> #startmeeting ironic_qa
17:01:28 <openstack> Meeting started Wed May 25 17:01:27 2016 UTC and is due to finish in 60 minutes.  The chair is krtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:29 <DuncanT> geguileo: debug is fine I guess. Not much real difference
17:01:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:32 <openstack> The meeting name has been set to 'ironic_qa'
17:01:42 <vdrok> o/
17:01:43 <rpioso> o/
17:01:46 <mjturek1> o/
17:02:12 <watanabe_isao> o/
17:02:17 <TheJulia> o/
17:02:25 <krtaylor> hi everyone, I'm hosting this week, jlvillal is away
17:02:42 <mjturek1> hey krtaylor :)
17:02:46 <krtaylor> As always the agenda is at: https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:02:49 <krtaylor> hi mjturek1
17:02:59 <krtaylor> its a pretty light agenda this week
17:03:20 <krtaylor> #topic Announcements
17:03:33 <krtaylor> so any announcement this week?
17:03:40 <watanabe_isao> krtaylor, may I give a Info?
17:03:45 <vdrok> one announcement from me - we now have tests for microversions in tempest - https://review.openstack.org/#/c/260358/
17:04:05 <krtaylor> watanabe_isao, sure, you have the floor
17:04:14 <krtaylor> vdrok, excellent!
17:04:15 <watanabe_isao> #Info Fujitsu iRMC CI gave its first test result to Ironic, yesterday (5/25). Currently the CI is not so stable due to some intra network issue. And we set it to non-voting for now. We will try our best to fix the issue, and start to vote ASAP.
17:04:56 <krtaylor> watanabe_isao, it should be non-voting for a while, until it proved to be robust
17:05:08 <krtaylor> but thats good news!
17:05:12 <rloo> great work watanabe_isao!
17:05:17 <jroll> \o/
17:05:31 <watanabe_isao> krtaylor, sure.
17:05:37 <watanabe_isao> rloo, thank you.
17:05:56 <krtaylor> ok, any other announcements?
17:06:17 <krtaylor> #topic Grenade
17:06:28 <krtaylor> great progress this week!
17:06:50 <krtaylor> #link https://etherpad.openstack.org/p/ironic-newton-grenade-whiteboard
17:07:29 <krtaylor> anyone want to make any comments or raise any topics on grenade?
17:07:44 <jroll> just that we need reviews, sounds like things are passing just fine :)
17:07:55 <jroll> patches are on the etherpad there
17:08:04 <krtaylor> yep, looks like 8 patch sets are left
17:08:08 <TheJulia> awesome
17:08:14 * devananda sneaks in the back, catches up
17:08:40 <vdrok> morning devananda ! :)
17:09:11 <krtaylor> anything else?
17:09:32 <krtaylor> #topic Functional testing
17:09:48 <krtaylor> I'm not aware of any progress here, anyone?
17:10:01 * jroll hasn't heard anything
17:10:09 <rloo> sorry can i ask a dumb question about grenade tests.
17:10:25 <rloo> once those patches are *in*, we have grenade tests working, right?
17:10:32 <rloo> then do we add more tests to grenade?
17:10:59 <vdrok> rloo: yep, grenade is pretty stable, grenade partial is not atm
17:11:09 <vdrok> and I think we don't add tests to grenade
17:11:14 <vdrok> it just runs smoke
17:11:35 <rloo> vdrok: ok, so next thing is to get grenade partial working?
17:11:43 <vdrok> maybe we'll add something in resorce_create and resource_verify, but dunno
17:11:47 <jroll> well, I'd like to make sure smoke covers everything we care about, e.g. rebuild
17:11:49 <rloo> vdrok: I mean, after you party and take a break :)
17:12:08 <jroll> but yes, the next would be grenade partial working. which implies getting rolling upgrades working. :)
17:12:16 <jroll> (well, and making grenade a voting job, ofc)
17:12:22 <vdrok> rloo: yes, it mostly works, I guess the reason it's unstable is concurrency 2
17:12:33 <rloo> ok, just want to make sure there is a list of things-to-do
17:13:28 <jroll> cool
17:13:31 <jroll> shall we move on?
17:13:35 <krtaylor> ok, so I think we are past functional
17:13:36 <vdrok> jroll: isn't partial just cold upgrading ironic ironic without everything else?
17:13:40 <devananda> what are the chances we'll get grenade testing on some of hte current stable branches?
17:13:55 <devananda> vdrok: there are two forms of partial upgrade that I think we need to cover
17:14:01 <devananda> 1. upgrade some ironic services
17:14:09 <jroll> vdrok: it's making sure new ironic-conductor works with old ironic-api, which is the basis for rolling upgrades
17:14:11 <devananda> 2. upgrade some openstack services but not others
17:14:29 <jroll> devananda: (2) is completely different, and something we should work on, but as a separate task
17:14:31 <jroll> (imo)
17:14:35 <krtaylor> lets jump back to this topic in the open discussion
17:14:39 <devananda> jroll: agreed. but it's also a "partial upgrade"
17:14:43 <devananda> krtaylor: sure, sorry
17:14:49 <jroll> krtaylor: +1
17:14:49 <vdrok> aha, the reason I ask is I saw it passing a couple of times :)
17:15:17 <krtaylor> #topic 3rd party CI
17:15:39 * jroll is excited about this topic
17:15:43 <krtaylor> I just sent an email for more eyes on the status of the teams
17:15:51 <krtaylor> in the driver wiki
17:15:54 * rloo likes to see jroll excited
17:15:59 <[1]cdearborn> o/
17:16:24 <watanabe_isao> krtaylor, do I need to provide any Info (of iRMC CI) to you to update the driver wiki?
17:16:25 <krtaylor> I have talked to a few teams and made a couple of changes, but I doubt it is complete
17:16:46 <jroll> so I see CI systems that say they're reporting on every patch, that are not reporting on every patch
17:16:56 <jroll> e.g. irmc just started reporting today, I don't think that's enough data
17:17:04 <krtaylor> watanabe_isao, yes, please review the ironic/driver page
17:17:10 <jroll> oneview reports on patches touching ironic/drivers/ only, apparently
17:17:19 <watanabe_isao> krtaylor, sure sir.
17:17:20 <krtaylor> #link https://wiki.openstack.org/wiki/Ironic/Drivers
17:17:43 <krtaylor> jroll, I don't doubt it
17:17:43 <jroll> cisco must also be filtering based on file, I don't see it on e.g. https://review.openstack.org/#/c/318497/
17:17:44 <TheJulia> jroll: I believe that is hardware capacity issues combinded with providing timely feedback
17:17:47 <thiagop> true, I talked about it on the Summit with some people. We just don't have enough hardware to run on every patch
17:18:21 <krtaylor> jroll, I'd like to get this table sanity checked for sure
17:18:26 <jroll> well, those systems need to get there IMO
17:18:29 <jroll> from the spec: "It runs the expected tests for every patch set that is not excluded. Reasons for this exclusion will be documented and approved by the ironic team."
17:18:54 <jroll> I don't remember seeing any exclusions documented, let alone approving them
17:19:16 <krtaylor> yes, and we also said that it didn't have to be in the 4 hour window to start
17:19:27 <jroll> yep
17:19:32 <jroll> the spec also says "unless that change is to code or documentation that can not impact the driver"
17:19:39 <jroll> certainly ironic/utils/ can impact drivers
17:19:43 <jroll> almost certainly*
17:19:55 <krtaylor> 4hrs  is the recommended time to turn around a comment
17:20:08 <jroll> well, 8 hours in Newton, 4 in Ocata
17:20:18 <thiagop> Maybe we should relax this requirement? Ironic testing requires hardware...
17:20:19 <krtaylor> ironic patches usually don't come in so fast to make that a problem  :)
17:20:33 <jroll> right, there aren't a lot of patches
17:20:40 <jroll> this was a known caveat when we introduced it
17:20:53 <jroll> vendors said it would be fine, given the estimates on number of patches
17:21:20 <krtaylor> exactly, lets see commenting and if it is past 8hrs evaluate then
17:21:33 <jroll> +1
17:21:33 <thiagop> I'm trying to reduce our testing time, but yesterday I was with 18 patches on my queue at some point...
17:21:51 <jroll> with the addition that I expect every patch running in CI
17:21:55 <devananda> some of the large patch sets that get rebased periodically might be affecting this right now
17:21:57 <jroll> or rather, following the rules upstream CI does
17:21:58 <devananda> eg, the network integration
17:22:29 <devananda> jroll: what was the estimated # of patches per day mentioned at the summit? do you recall?
17:22:37 <krtaylor> I don't think teams should just give up, run through the queue and see how long it takes
17:22:37 <jroll> sure, we won't panic if some CI can't handle the load of a large series being uploaded
17:22:40 <jroll> I don't, sorry
17:22:48 <thiagop> I think it was 30
17:22:57 * krtaylor looks at his notes
17:23:14 <jroll> this is the exclusion list for upstream CI https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L971-L979
17:23:14 <watanabe_isao> yesterday, there were 45 patches for 24 hours.
17:23:28 <rloo> that's 30 revisions to patches.
17:23:42 <krtaylor> but that is abnormal
17:24:42 <devananda> jroll: that exclusion list seems fine for 3rd party too, IMO
17:24:51 <jroll> devananda: yep, that's my thought
17:25:33 <krtaylor> agreed
17:25:58 <jroll> so yeah, just wanted to point out the requirements, and that there are CIs that say they fulfill the requirement but don't in reality
17:26:07 <krtaylor> well, I'll be happy to correct info on the wiki status table, but I probably shouldn't be the one policing it
17:26:19 <jroll> krtaylor: yeah, I'm happy to fix it too
17:26:25 <TheJulia> Agreed, thiagop, any chance you can update your filter to match the above link of sorts, excluding the other vendor drivers so we can see how long repsonses are in?
17:26:37 <jroll> krtaylor: just curious where the "reports on everything" came from :)
17:26:39 <krtaylor> jroll, thanks!
17:26:45 <devananda> jroll: have we stated an expectation that 3rdparty CI test other projects in the ironic group?
17:27:01 <jroll> devananda: we have not, and I don't expect to need that
17:27:12 <krtaylor> jroll, like I said at summit, most of the info was from either the etherpad, infra account info, etc
17:27:13 <devananda> eg, ironic-lib
17:27:23 <jroll> devananda: perhaps make it optional if they're concerned about it breaking
17:27:26 <thiagop> TheJulia: I can, yes
17:27:35 <jroll> devananda: we have depends-on if we as developers think a patch may be problematic
17:27:46 <TheJulia> thiagop: That would be awesome :)
17:28:14 <thiagop> TheJulia: but the average time for builds on the current CI is 80min
17:28:51 <thiagop> TheJulia: on the new CI, using nodepool, I dropped it to 36min so far
17:28:55 <jroll> O_o
17:29:01 <jroll> ah, that sounds better
17:29:06 <TheJulia> oh that does sound far better
17:29:17 <devananda> thiagop: nice!
17:29:33 <thiagop> but it's not stable yet...
17:29:48 <thiagop> working on it today/tomorrow
17:29:57 * krtaylor needs to see if the CI dashboard will show some historical info to make a determination which teams really have CI
17:30:06 <jroll> what was the old CI using, just curious?
17:30:36 <thiagop> jroll: it was building devstack on a KVM in a custom script
17:30:44 <jroll> ah
17:30:49 <thiagop> nodepool caching really REALLY helps to cut time
17:30:54 <jroll> yep
17:31:32 <krtaylor> thiagop, mmedvede from my team has some really good optimization experience
17:32:21 <krtaylor> so did we get to a conclusion on CI topics?
17:32:21 <thiagop> krtaylor: I'll talk to him/her as soon as my tests are passing. Thanks!
17:32:51 <krtaylor> thiagop, sure np, he now runs the third party CI working group meetings
17:33:02 <jroll> well, conclusions I've seen: 1) follow the guidelines, 2) myself or kurt need to update the wiki thing
17:33:03 <rloo> if most vendors have same issues wrt optimization, would it help to have some documentation about best practise or whatever?
17:33:20 <watanabe_isao> thiagop, what CI cloud are you using for your nodepool, please? KVM?
17:33:44 <krtaylor> rloo, I don't see why not, that makes sense
17:34:23 <krtaylor> mjturek1, ^^^
17:35:03 <krtaylor> I think that is an area that we can share our experience and at least start docs for ironic CI
17:35:25 <jroll> ++
17:35:29 <krtaylor> so lets move on
17:35:34 <thiagop> watanabe_isao: the nodepool is in our internal cloud of the lab
17:35:45 <krtaylor> #topic General QA
17:35:49 <thiagop> krtaylor: sure thing
17:35:56 <krtaylor> other topics?
17:36:15 * krtaylor thinks we should combine general and open topics in future meetings
17:36:22 <jroll> +1
17:36:32 <watanabe_isao> +1
17:36:52 <krtaylor> well then, lets open the floor to open discussion
17:36:53 <rloo> so remind me, what's the purpose of https://wiki.openstack.org/wiki/Ironic/Drivers
17:37:05 <krtaylor> #topic Open Discussion
17:37:07 <rloo> vs the marketplace one vs there is some other list of drivers
17:37:24 <krtaylor> rloo, not much, and the info there is old
17:37:28 <devananda> rloo: some of the historical data on the wiki should, IMO, be removed
17:37:46 * jroll doesn't like wikis either
17:37:50 <krtaylor> rloo, I was using it as a place to create a table (because no etherpad tables)
17:37:55 <rloo> so let's decide/describe what is in each page, so we don't duplicate and whatever.
17:38:11 <thiagop> or eliminate some maybe :)
17:38:19 <devananda> krtaylor: the last table on the wiki looks like where you're tracking that status, yes?
17:38:20 <krtaylor> but that is true for a lot of our wiki sub pages
17:38:22 <thiagop> consolidate information...
17:38:34 <krtaylor> devananda, yes
17:38:59 <krtaylor> wiki cleanup - maybe a good workday kinda activity
17:39:26 <devananda> let's remove the rest of the data on that page, replacing with a link to stackalytics drivers page and some info on process to update it ?
17:39:27 <rloo> krtaylor: even after cleaning up the wiki, is the wiki meant to be kept up-to-date, and if so, who keeps it up-to-date?
17:39:54 <krtaylor> devananda, I'd second that
17:40:05 <rloo> or do as devananda suggests )
17:40:06 <krtaylor> devananda, that info is next to the table too
17:40:11 <devananda> and at the end of Newton, let's remove that table, too
17:40:21 <krtaylor> yes
17:40:30 <krtaylor> then just CI docs
17:40:49 <krtaylor> then we get a gerrit trail for changes
17:41:00 <krtaylor> and reviews, etc
17:41:01 <devananda> yup
17:41:25 <jroll> yay peer review
17:41:48 <krtaylor> so, do we need to revisit grenade/rolling upgrades discussion?
17:41:59 <devananda> I'd be in favor of basically ceasing to use the wiki as much as possible, especially for documentation
17:42:09 <jroll> devananda: ++
17:42:19 <krtaylor> ++
17:42:30 <devananda> for tracking things like krtaylor has been doing, it's a little better than an etherpad, but not much
17:42:58 <devananda> krtaylor: would you like to do the honors of trimming that page down? :)
17:43:08 <krtaylor> there is a table plugin for etherpads  :)
17:43:16 <devananda> krtaylor: there is?!
17:43:22 <krtaylor> devananda, sure and yes
17:43:45 <krtaylor> we use the etherpad table plugin internally
17:43:55 <devananda> mmmmmm. I need to find that ...
17:43:58 <devananda> anyway :)
17:44:00 <krtaylor> hm, maybe an upgrade to infro etherpad is in order
17:44:02 <devananda> rolling upgrades
17:44:37 <devananda> jroll: we've talked about the different combinatorial sets of upgrade testing we want, eg. in the summit 'pad
17:44:38 <thiagop> lol
17:44:47 <jroll> mhm
17:46:01 <devananda> I think it'd be helpfu, when folks say 'partial upgrades', to specify which one
17:46:33 <jroll> well, so grenade-partial has historically always meant "partial upgrade of the service under test"
17:47:02 <jroll> so I prefer to keep that meaning
17:47:07 <devananda> fair 'nuf
17:47:20 <jroll> no projects in openstack currently test "full upgrade of a single service in the cloud"
17:47:45 <devananda> I'm pretty sure I recall that, when ironic was going through graduation process, we were required to have an "upgrade ironic before nova" test
17:47:49 <jroll> so I'm not really concerned with testing that (yet)
17:47:49 <devananda> so we did, at one point
17:48:12 <jroll> interesting
17:48:19 <jroll> I'd be curious if nova folks still care about that
17:48:22 <devananda> thus my ongoing confusion when folks talk about it as though it never was
17:48:26 <devananda> me too :)
17:48:53 <jroll> I don't remember that fwiw
17:49:25 <devananda> i'll dig around for a second ...
17:49:37 <rloo> devananda: if we had that test, what happened to it?
17:50:03 <rloo> devananda: or do you mean, they wanted that test, but we didn't get it done. is that what nova keeps asking us for?
17:50:04 <jroll> rloo: if it's grenade-partial-ironic, then it sat not working for the last two years
17:50:17 <jroll> nova keeps asking us for tempest-full
17:50:21 <jroll> (as they should)
17:50:48 <rloo> ironic is a teenager :D
17:50:57 <krtaylor> hehheh
17:51:17 <devananda> jroll: https://review.openstack.org/#/c/111859/
17:51:31 <devananda> jroll: yea, grenade-sideways
17:51:41 <jroll> devananda: oh, that, that was for the migration from nova-bm though
17:51:54 <devananda> right
17:51:57 <devananda> so it was :)
17:52:26 <devananda> Ah. so we never had that after the nova-bm migration
17:52:48 <devananda> never mind then :)
17:52:53 <jroll> right, that was a one time thing
17:53:39 <jroll> so idk, I think ironic vN with nova vN-1 is a good thing to test, but let's take one thing at a time and get rolling upgrades working
17:53:48 <jroll> I *think* that will actually test that, too
17:53:48 <devananda> jroll: agreed
17:53:56 <jroll> well, not quite, meh
17:54:05 <rloo> so it would be good if someone would write down what grenade-partial means/types of tests it will include, etc.
17:54:09 <jroll> we could add a "finish the ironic upgrade" step
17:54:17 <devananda> jroll: ++
17:54:20 <jroll> rloo: it will be ==grenade, but only upgrade ironic-conductor
17:54:59 <rloo> QA folks, is there documentation somewhere about the types of tests that ironic has?
17:55:29 <krtaylor> no that I'm aware of
17:55:40 * krtaylor scribbles down a note
17:56:29 <krtaylor> ok, feels like this is winding down, maybe we continue this in -ironic?
17:56:59 <jroll> wfm
17:57:13 <krtaylor> anything else?
17:57:30 <krtaylor> Thanks everyone
17:57:40 <rloo> thx krtaylor
17:57:49 <krtaylor> #endmeeting