17:00:32 <krtaylor> #startmeeting ironic-qa
17:00:33 <openstack> Meeting started Wed Nov 18 17:00:32 2015 UTC and is due to finish in 60 minutes.  The chair is krtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:37 <openstack> The meeting name has been set to 'ironic_qa'
17:00:48 <krtaylor> anyone here for ironic-qa?
17:00:54 <cdearborn> o/
17:00:54 <sinval> o/
17:00:57 <MattMan> o/
17:00:59 <rpioso> o/
17:01:46 <krtaylor> hi everyone
17:02:19 <sinval> hey
17:02:21 <krtaylor> I'll chair again this week since jlvillal is busy
17:02:34 <krtaylor> here is the agenda for the meeting:
17:02:38 <krtaylor> #link https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:03:18 <krtaylor> I'll start with actions, I had one to talk to lucas about the spec
17:03:24 <krtaylor> and that was done
17:03:37 <krtaylor> any quick announcements?
17:03:49 <dtantsur> o/
17:04:14 <krtaylor> ok, well, then let's jump in
17:04:19 <sinval> so I assume that the spec will land soon?
17:04:25 <rajinir> hi
17:04:32 <krtaylor> #topic grenade testing
17:04:50 <krtaylor> sinval, hope so, lets talk about that in the CI section
17:04:56 <KevinCalman> o/
17:05:19 <sambetts> o/
17:05:20 <krtaylor> I don't have anything advancements or status here, anyone?
17:05:24 <rajinir> o/
17:06:21 <krtaylor> we'll have to wait until jlvillal gets back for a grenade update I guess
17:06:33 <liliars> sorry guys, little late
17:06:52 <krtaylor> hi liliars, no worries
17:06:58 <krtaylor> #topic functional testing
17:07:10 <krtaylor> same here for me, nothing to report, anyone?
17:07:55 <liliars> more like related to the third party ci things, but I talked to people over here and we are good to publish our functional tests
17:08:18 <sinval> well, for oneview driver, we created something like a "functional test suite", there is a bunch of code that was developed and could be reused for other CI
17:08:19 <krtaylor> good, that needs a nudge to get started
17:08:20 <liliars> I'm hoping someday next week *possibly*
17:08:41 <sinval> I thought that maybe we could work on it and make something like a piece of code that would be good to reuse in other CI, even for Ironic functional testing, what do you guys think about that?
17:08:58 * jroll apologizes for being late
17:09:33 <krtaylor> sinval, I'd like to understand it, could you present the basic ideas next week?
17:09:41 <jroll> is there functional testing for oneview that doesn't apply to other things?
17:09:47 <jroll> I'd rather just have those in the main tree
17:09:59 <sinval> krtaylor, sure
17:10:00 <liliars> yeah that's what we talked about last week
17:10:12 <liliars> (for a few seconds)
17:10:18 <sinval> Actually, I think that the basic functional test suite for each driver could be totally reusable, something like: "if you are developing a driver CI: here is a lib for functional testing in Ironic that your CI can reuse to test you driver, and in this lib there is a bunch o test cases already implemented that can be configured and you can reuse and take care of more specific test cases of your driver", maybe this lib can be useful in Ironic func
17:10:19 <sinval> tional testing module
17:10:40 <jroll> yeah, I'd rather that just go up into ironic
17:10:53 <sinval> nice
17:11:01 <liliars> cool
17:11:04 <sambetts> +1000
17:11:08 <liliars> we'll plan on it
17:11:59 <krtaylor> sinval, liliars - please add to the agenda for next week
17:12:04 <liliars> will do
17:12:23 <sinval> We'll formalize that on something like a RFC for you guys
17:12:36 <jroll> well
17:12:44 <jroll> is it just api commands to play with power control and such?
17:13:00 <sinval> yep, kind of
17:13:10 <liliars> plus some management stuff
17:13:17 <liliars> deploy is on its way
17:13:18 <sinval> but we did some tricky test scenarios, and setup things to test drivers
17:13:22 <jroll> yeah, I'd just propose the code honestly
17:13:36 <sinval> sounds great
17:13:44 <jroll> maybe a quick etherpad if you want to lay out what it does first
17:13:45 <krtaylor> yes, that will be good to understand
17:13:52 <jroll> krtaylor: ^ you good with that?
17:14:00 <sinval> jroll, ok
17:14:27 <liliars> I also think it's more reasonable, most of the tests are pretty easy to understand anyway
17:14:30 <krtaylor> +1, and an explanation if needed next week should get everyone on the same page
17:14:35 <liliars> (I hope hah)
17:15:15 <krtaylor> cool, anything else on functional testing?
17:15:18 <liliars> cool :)
17:15:40 <krtaylor> #topic 3rd party CI
17:15:48 <krtaylor> so, the spec is coming along
17:15:52 <krtaylor> good comments
17:16:09 <sinval> awesome
17:16:23 <krtaylor> I think the rest of the details can be documented in documentation patches
17:16:27 <jroll> ++
17:16:27 <krtaylor> and refined there
17:16:49 <krtaylor> main thing is to get all the drivers contacted and aware of the schedule
17:16:56 <rajinir> is there a link to the spec? sorry I'm new to the meeting. What spec are we talking about
17:16:56 <liliars> agreed
17:17:16 <liliars> docs can have more details, but I understand folks concerns over there
17:17:30 <rajinir> Spec: https://review.openstack.org/#/c/241294 ?
17:17:31 <krtaylor> then get the driver teams in the infra CI meetings, understanding zuul/jenkins etc
17:17:53 <liliars> yeah rajinir
17:18:05 <sinval> krtaylor, +1
17:18:15 <krtaylor> yes, thats it
17:18:30 <aarefiev> rloo, jroll: please take a look, is it right way to add notes https://review.openstack.org/#/c/226201/
17:19:08 <aarefiev> missed
17:19:16 <krtaylor> aarefiev, can we move review requests to #openstack-ironic?
17:19:30 <aarefiev> krtaylor: sorry
17:19:38 <krtaylor> aarefiev, np
17:20:19 <krtaylor> so, anything else on CI? everybody gearing up?  :)
17:20:58 <krtaylor> we (IBM PowerKVM) should turn on our ironic ci testing soon
17:21:12 <liliars> yep! :)
17:21:14 <krtaylor> then we can help document what we did
17:21:24 <rajinir> Is everyone following asselin's docs to setup the CI?
17:21:24 <sinval> o/
17:21:32 <krtaylor> I was hoping to get the doc structure out soon
17:21:53 <MattMan> That would be great, evn though our driver is not (yet) in tree I am planning on getting CI going as soon as I can
17:21:55 <sambetts> We (Cisco) have a plan :)
17:21:58 <rajinir> https://review.openstack.org/#/c/227584/9/contrib/README.md ?
17:22:09 * jroll gets excited
17:22:15 <krtaylor> rajinir, a better way to ask might be is everyone using upstream infra for their test services?
17:22:19 <liliars> krtaylor: cool, feel free to ping me when you decide to \o/
17:22:53 <krtaylor> liliars, will do
17:23:24 <rajinir> krtaylor:  ok,sure (this is Dell, we just getting started)
17:23:47 <liliars> also, are we ready to talk about the definition of a reliable ci?
17:24:22 <krtaylor> rajinir, get involved in the infra CI meeting, a great place for getting statred help
17:24:24 <liliars> last meeting we kinda talked about it but no conclusions
17:24:32 * krtaylor runs away...
17:24:37 <sinval> hahahah
17:24:56 <liliars> wait!!
17:24:57 <krtaylor> well, I finally did get everyone to agree on a CI dashboard
17:24:59 <liliars> hah
17:25:08 <rajinir> krtaylor: sure, thx
17:25:19 <sambetts> Thats something that we'd like to know too, as well how many events we should be expected so we know how to scale our CI
17:25:21 <krtaylor> so we have one solution for that now, but there is so much more
17:25:43 <krtaylor> sambetts, we have been using satckalytics and guessing
17:25:45 <sinval> I have some thinking about that, but maybe is better to have this kind of discussion during the doc patches...
17:25:58 <liliars> sambetts +1 for scaling
17:26:36 <liliars> first I thought we were going to consider every ironic patch but I saw that changed in the spec
17:26:40 * jroll tries to get real numbers
17:26:49 <krtaylor> sambetts, sinval - agreed, a recommendation of what is required and a rough average on patch counts would be good for planning
17:26:51 <jroll> wait, that changed?
17:26:55 <jroll> it should still be every patch
17:27:04 <krtaylor> no, didnt change
17:27:10 <krtaylor> patch = change
17:27:15 <liliars> 1 sec
17:27:40 <krtaylor> liliars, every change = every patch
17:27:50 <krtaylor> it was a wording change
17:28:03 <sambetts> We are currently looking at ordering hardware for our CI, so knowing how much parralisation we need is important :)
17:28:19 <liliars> from the spec: *..to test the driver against every code change proposed for Ironic, unless that change is to code or documentation that is irrelevant to the driver.*
17:28:42 <sinval> thanks liliars
17:28:55 <krtaylor> sambetts, it was an evolving sizing for us, we are still getting it right after 2+ years
17:29:06 <liliars> dont't know if I misunderstood that
17:29:30 <krtaylor> liliars, change = patch, should I make that more clear?
17:29:32 <sinval> "irrelevant" for me here is OneView CI should not test iLO implementation changes
17:29:43 <krtaylor> exactly
17:29:49 <sambetts> liliars: every change to ironic could cause a breakage, so every change is relevant
17:30:11 <krtaylor> every core change, that is what that clarified
17:30:13 <jroll> well, I think we need to start with all changes and narrow down from there
17:30:29 <sinval> krtaylor, agree
17:30:43 <sambetts> I agree, iLo could do something weird that for some reason breaks my code, and I'd want to test against that
17:31:20 <sinval> have you guys decided about the time to test things?
17:31:38 <liliars> yeah I get it
17:31:40 <krtaylor> but if that driver is not enabled, generally speaking, isn't that a no-op anyway?
17:31:42 <sinval> we are facing ~40 mins to run a deploy interface test case...
17:32:04 <liliars> I think I misunderstood the meaning of "irrelevant" here
17:32:23 <liliars> thanks for the clarifications guys krtaylor sambetts
17:32:23 <afaranha> in order to run all the core patches we would need a few more machines for the CI (I'm from OneView Team)
17:32:45 <krtaylor> liliars, let's refine that wording in the requirements documentation  :)
17:32:57 <afaranha> and with what we have thing would stack and one patch could take very long to start running
17:33:00 <jroll> sinval: the gate also takes ~40 minutes to run the pxe_ipa job
17:33:05 <liliars> krtaylor: yeah! will make a note on that :)
17:33:31 <krtaylor> sinval, that is about right time-wise
17:34:00 <sinval> jroll, krtaylor nice :)
17:34:10 <sambetts> I havn't got any data on ours yet, but I know that the boot time on our servers is not fast
17:34:32 <krtaylor> 40 minutes + setup/teardown * daily patch average = machines needed for peak load, ymmv
17:34:33 <sinval> sambetts, feel you pain :(
17:34:42 <MattMan> Our SPARC servers are also quite slow to boot
17:36:23 <krtaylor> let's table this for now, we can continue to share timing/sizing estimates in future meetings
17:36:47 <sambetts> cool :)
17:36:50 <krtaylor> #topics General QA topics
17:37:10 <krtaylor> all drivers case - testing conflicts
17:37:23 <krtaylor> does someone want to lead that discussion?
17:37:31 <sambetts> krtaylor: I don't think the topic changed
17:37:39 <krtaylor> oops
17:37:47 <krtaylor> #topic General QA topics
17:37:52 <krtaylor> much better
17:38:52 <krtaylor> anyone?
17:39:07 <sambetts> Ok, so in the ironic meeting on Monday, it was brought up, should we be testing the case of enabling all the drivers to check for
17:39:12 <sambetts> conflicts
17:39:18 <sambetts> ?
17:39:41 * krtaylor is trying to come up with a use case
17:39:45 <jroll> so I thought more about that and want to defer doing things with this until pavlo files the bug he promised
17:39:47 <liliars> yeah that one, was trying to remember the context, thanks sam
17:39:58 <jroll> and it's actually about installing dependencies, not necessarily enabling the drivers
17:40:08 <jroll> packaging is a good use case for this
17:40:13 <sinval> what were the drivers involved on this problem? IRMC and iLO?
17:40:38 <krtaylor> is it a pre-req versioning test? or a namespace test?
17:41:49 <jroll> it's a "make sure services run with all the things installed"
17:42:22 <krtaylor> jroll, but would that ever happen in real life?
17:42:43 <jroll> krtaylor: a distro may ship with all the driver deps installed, yes
17:42:52 <sambetts> I believe their bug was they installed a dependecy and it blows up the conductor, which should be covered by a third party CI one right?
17:43:06 <krtaylor> jroll, ah, thats the use case I needed, thanks
17:43:24 <jroll> yeah, third party CI should also cover this
17:43:34 <jroll> I'm not inclined to make this a priority, at least until someone files a bug
17:44:19 <krtaylor> so, the test is to install all packages and see if the install worked, seems easy enough
17:44:53 <jroll> right, it would need to stay non-voting so packages not in g-r can't explode the gate
17:44:55 <jroll> but yeah
17:45:11 <krtaylor> ok, well we can defer this until the need arises
17:45:18 <sambetts> not all drivers reqs are installable via PyPi, so we'd have to have a bunch of special sauce too
17:45:26 <krtaylor> thanks for the clarification, I understand the need now
17:46:10 <krtaylor> anything else on this?
17:46:25 <krtaylor> #topic Open Discussion
17:46:31 <jroll> I have a thing here
17:46:41 <jroll> http://bit.ly/1OQD1mv <- graph of ironic patches + rechecks per day
17:46:55 * krtaylor look
17:47:03 <jroll> so this seems roughly like what a third party CI needs to support
17:47:38 <rajinir> max:30
17:47:46 <krtaylor> ok, and other systems handle this within a set time frame
17:47:56 <krtaylor> so you don't need a system that can run 30 at once
17:48:10 <krtaylor> other projects say within 4 hours
17:48:15 <krtaylor> nova for instance
17:48:15 <rajinir> what is the interval ?
17:48:24 <jroll> a second thing: I'm in talks with some folks, it tentatively appears that OSIC can provide some hardware for ipmitool third party testing. I can't promise that I can maintain the third party CI for that, but I can probably gather some help from either other rackspace people or the community
17:48:25 <rajinir> and any predictions on the growth rate?
17:49:00 <jroll> better graph, covers the past year: http://graphite.openstack.org/render/?width=586&height=308&_salt=1447868663.13&lineMode=connected&from=00:00_20141115&until=23:59_20151118&target=summarize%28stats_counts.zuul.pipeline.check.openstack.ironic.total_changes,%20%271d%27%29
17:49:24 <sambetts> krtaylor: so 4 hours * 30 / how long it takes to run your test = how many machines we'll need
17:49:26 <krtaylor> jroll, I know there are other discussions for ipmi too
17:49:40 <liliars> cool! thanks jroll, thats helpful
17:50:23 <jroll> krtaylor: cool, I've got a soft yes on it - they want to see what we need from a network perspective
17:50:31 <krtaylor> sambetts, we haven't set a timeframe for ironic testing requirements, but if we used nova's for example, it would need to report results within 4 hours after a patch is submitted
17:50:34 <jroll> so we/I need to write something up there
17:50:36 <rajinir> jroll:Thanks
17:51:07 <sambetts> krtaylor: Makes sense :) thats something we'll need to define it the whole "what is a reliable CI?" convo then
17:51:18 <liliars> +1
17:51:18 <jroll> yeah
17:51:23 <krtaylor> sambetts, exactly
17:51:33 <jroll> and of course we can aim to improve over time, start looser and tighten as we go
17:51:40 <jroll> or get some CIs up and see what's reasonable
17:51:55 <sinval> what about logs? how much time to keep them?
17:52:00 <sinval> 1 month?
17:52:01 <krtaylor> cinder started with a much larger number at first, it makes the sizing estimates easier
17:52:27 <krtaylor> sinval, the infra requirement was 6 months, now down to a month
17:52:35 <liliars> yeah also makes sense, and give us some time to adjust
17:52:47 <sambetts> krtaylor: larger number of expected patch sets? or time to post comment?
17:52:52 <sinval> krtaylor, cool
17:53:06 <krtaylor> sambetts, time to post CI result comment
17:53:29 <sambetts> krtaylor: Ok that makes sense :)
17:53:35 <krtaylor> so start with 8 hours for N, then 4 hours for O, etc
17:54:05 <krtaylor> as teams get running CI systems, they will be better able to see what they need to hit a 4 hour window
17:54:13 <jroll> ++
17:54:21 <sambetts> ++
17:54:51 <krtaylor> our (powerkvm) goal has always been to beat jenkins  :)
17:55:04 <krtaylor> and we do often
17:55:05 <jroll> heh
17:55:17 <sambetts> gamify it :D
17:55:28 * jroll 5 minute warning
17:55:30 <krtaylor> YES
17:55:45 <krtaylor> ok, are we winding down here?
17:56:02 <krtaylor> last call for topics
17:56:26 <krtaylor> ok, thanks everyone, another good meeting!
17:56:32 <sambetts> Thanks krtaylor o/
17:56:34 <jroll> \o/ thanks for leading, krtaylor
17:56:45 <krtaylor> #endmeeting