17:00:03 <jlvillal> #startmeeting ironic_qa
17:00:04 <openstack> Meeting started Wed Feb 24 17:00:03 2016 UTC and is due to finish in 60 minutes.  The chair is jlvillal. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:07 <openstack> The meeting name has been set to 'ironic_qa'
17:00:21 <krtaylor> o/
17:00:24 <jlvillal> Hello everyone
17:00:37 <rpioso> o/
17:00:48 <jlvillal> As always the agenda is here: https://wiki.openstack.org/wiki/Meetings/Ironic-QA
17:00:54 <krtaylor> hi jlvillal, may be a light meeting today
17:00:58 <jlvillal> #topic Announcements
17:01:04 <jlvillal> krtaylor: I wouldn't be surprised.
17:01:22 <jlvillal> I don't have any announcements. Does anyone else?
17:01:55 <jlvillal> #info TinyIPA job has been added to the gate as non-voting
17:02:04 <jlvillal> Okay, moving on
17:02:11 <jlvillal> #topic Grenade testing of Ironic
17:02:41 <jlvillal> #info jlvillal has created an environment to simulate the devstack-gate  https://github.com/JohnVillalovos/devstack-gate-test
17:03:39 <jlvillal> So myself, mgould, and a co-workers have used that environment to test the ironic grenade job
17:04:30 <jlvillal> #info Have proposed being able to base the Tempest REGEX down through Grenade. Currently getting push-back on proposal from sdague
17:05:14 <jlvillal> #info Alternative idea would be to flag out every test that isn't currently being run when we use the baremetal REGEX when running Ironic.
17:05:54 <jlvillal> #info Idea there would be to try to get same tests to run in 'smoke' as currently run with the 'baremetal' REGEX
17:06:10 <krtaylor> jlvillal, so not be able to do skips via REGEX?
17:06:30 <jlvillal> krtaylor: At the moment we use the REGEX for our normal Ironic gate.
17:06:43 <jlvillal> But when doing Grenade, it doesn't allow usage of the REGEX.
17:06:58 <jlvillal> Let me find my patches
17:07:06 <krtaylor> right, but that is acceptable in check/gate but not grenade?
17:07:18 <jlvillal> Well I guess it is a matter of opinion :)
17:07:20 <krtaylor> seems like a simple fix
17:07:33 <jlvillal> I would think we should run the same tests in both our normal gate as in Grenade.
17:07:45 <krtaylor> ++
17:07:58 <jlvillal> #link https://review.openstack.org/241018
17:08:07 <jlvillal> #link https://review.openstack.org/241044
17:08:21 * krtaylor looks
17:08:59 <jlvillal> So have made pretty good progress. But running into an issue where openstackclient is spitting out a wonky error.
17:09:11 <krtaylor> ah, yes, I remember the 3 VM discussion
17:09:21 <jlvillal> #info Running into bug: https://bugs.launchpad.net/devstack/+bug/1549095
17:09:21 <openstack> Launchpad bug 1549095 in Ironic "devstack fails while running Ironic grenade job: init__() got an unexpected keyword argument 'token'" [Undecided,In progress] - Assigned to John L. Villalovos (happycamp)
17:09:47 <jlvillal> I have reached out to an openstackclient contact in my company to see if they can help.
17:09:51 * devananda comes in a bit late, catches up on scrollback
17:10:05 <krtaylor> hi devananda
17:10:10 <jlvillal> I have no idea why getting that error. But it happens in my test environment and also in the gate.
17:10:41 <jlvillal> My theory/wild-a**-guess is that it is something is broken between stable/liberty and new libraries.
17:10:55 <jlvillal> As we don't see that error in our currently running gate.
17:11:59 <jlvillal> #action jlvillal to try to resolve bug with assistance from openstackclient developer.
17:12:08 <jlvillal> So that's all I got. Any questions?
17:12:10 <krtaylor> jlvillal, I know we also had stable branch problems, I'll ask our powerkvm CI team for help
17:12:12 <jlvillal> And hello devananda
17:12:24 <jlvillal> krtaylor: Thanks
17:12:35 <devananda> jlvillal: re: which tests to run - yes, I agree we should run the same functional & integration tests in both the existing gate jobs and the grenade jobs. It sounds like the issue is how to trigger the selection of those jobs
17:13:30 <jlvillal> devananda: Agreed. I'm okay proposing a patch to flag-out all the tests that run in 'smoke' that we don't currently run in our gate.
17:13:51 <devananda> rather than flag out every test we don't want to run -- because that will result in any NEW test potentially breaking our gate -- we should add a decorator that says "this is a test we SHOULD run against Ironic"
17:13:56 <devananda> ie, make it opt-in instead of opt-out
17:14:03 <jlvillal> If we do that and get it merged, then I think we should switch to using smoke in our gate, so that we stay in sync.
17:14:18 <devananda> so that we (the ironic team) need to actively agree to adding new tests that run in our gate
17:14:43 <jlvillal> devananda: I'm not sure if that is how tempest works or not. I'll have to look again. It has been more than a month since I looked at the tempest code...
17:14:43 <krtaylor> devananda, ++ decorator for opt-in
17:15:01 <jlvillal> I remember things to opt-out.
17:15:30 <devananda> that only makes sense for an integrated gate
17:15:44 <devananda> if another project can add a test to tempest, and that will automatically be run in our gate ==> problem
17:15:51 <krtaylor> or, just support skips in grenade  :)
17:15:52 <devananda> I do not believe tempest was written to behave that way
17:15:54 <jlvillal> Agreed!
17:16:34 <jlvillal> #action jlvillal to investigate if Tempest tests can be opt-in for bare-metal, using a decorator.
17:16:43 <jlvillal> Anything else?
17:16:59 <jlvillal> Okay moving on.
17:17:05 <jlvillal> #topic Functional testing
17:17:20 <jlvillal> Does anyone have any updates?
17:17:40 <mjturek1> hey jlvillal, nothing from me unfortunately. maurosr did you make any progress there?
17:17:56 <maurosr> mjturek1: no, I was focused on ci only
17:18:00 <mjturek1> I am wondering if we could discuss coverage a bit more
17:18:21 <jlvillal> mjturek1: What do you mean?
17:18:35 <mjturek1> are there tests in mind that belong in functional testing
17:18:58 <jlvillal> mjturek1: Do you mean current tests? Or tests in general?
17:19:11 <mjturek1> tests in general
17:19:50 <jlvillal> mjturek1: I would think we should look first at the API and see if we can exercise the API.
17:19:55 <jlvillal> ironic-api
17:20:05 <mjturek1> jlvillal: I beleive devananda wasn't for that
17:20:29 <jlvillal> Another thing might be functional testing that calls the ironic-conductor directly via RPC.
17:20:36 <jlvillal> Those are ideas of mine.
17:20:51 <mjturek1> ahh, that makes sense
17:21:31 <jlvillal> mjturek1: I'm open to suggestions :)
17:22:13 <mjturek1> jlvillal: db tests possibly? maurosr wasn't that one idea?
17:22:32 <jlvillal> mjturek1: And could do a thing where call the API and maybe have a fake conductor that responds. So make sure the API is calling the conductor how we think???
17:22:39 <jlvillal> I'm just brain-storming
17:23:13 <mjturek1> right
17:23:32 <jlvillal> Maybe DB tests too. I haven't thought about that much. But I could see a call to the API which goes to the conductor and then we check that the database has what we expect.  Possibly that???
17:23:45 <mjturek1> jlvillal: do you think moving current tests from tempest should be the main focus though?
17:24:19 <jlvillal> I don't like the word 'moving'
17:24:35 <mjturek1> heh, migrating?
17:24:43 <jlvillal> I don't like that either :)
17:24:48 <jlvillal> 'copying'
17:24:54 <krtaylor> I think the idea was duplicating first
17:24:58 <krtaylor> yes
17:25:00 <mjturek1> ahhhh, got it
17:25:03 <jlvillal> I don't like a word that implies that we remove a test from tempest.
17:25:22 <mjturek1> gotcha
17:25:35 <jlvillal> But yes, using current tempest tests as a framework/ideas on tests to put into functional testing, is a good idea.
17:26:15 <mjturek1> fair enough, so we could copy over tests we think are appropriate for initial functional testing
17:26:31 <jlvillal> mjturek1: That sounds good to me.
17:26:38 <jlvillal> As a first step.
17:26:43 <mjturek1> yep, totally
17:26:52 <mjturek1> thanks jlvillal!
17:26:57 <jlvillal> mjturek1: Thank you.
17:27:03 <jlvillal> Anything else before moving on?
17:27:08 <mjturek1> I'm good
17:27:15 <jlvillal> Okay, moving on
17:27:30 <jlvillal> #topic 3rd Party CI (krtaylor)
17:27:39 <jlvillal> krtaylor: All yours
17:27:54 <krtaylor> I did a initial scrub for infra requirements for our CI systems
17:28:19 <krtaylor> basically trying to verify the M2 milestone
17:28:35 <krtaylor> all results were posted here:
17:28:38 <krtaylor> #link https://etherpad.openstack.org/p/IronicCI
17:28:48 <krtaylor> most lined up properly
17:29:15 <krtaylor> there are still a few questions I have on in-tree drivers, systems that are testing
17:29:28 <krtaylor> I'll sort those out
17:29:41 <krtaylor> I have already talked to a few teams and fixed up some info
17:29:55 * mgould finally catches up on scrollback
17:30:06 <krtaylor> other than that, that is all I have this week for CI
17:30:30 <krtaylor> I really need to put all this into a table
17:30:43 <krtaylor> it will be MUCH easier to digest
17:31:05 <krtaylor> something like the nova hypervisor support table
17:31:16 <krtaylor> all in time
17:31:35 <krtaylor> any questions? comments?
17:31:42 <jlvillal> krtaylor: I gave you contact info for tanlin in regard to AMT. I added it to the Etherpad.
17:31:46 <devananda> krtaylor: ++ to an ironic driver matrix table
17:32:10 <jlvillal> krtaylor: Might have to be a Wiki page. Not sure if Etherpad does tables...
17:32:43 <krtaylor> jlvillal, it does, but with another plugin
17:33:00 <krtaylor> but I'm thinking wiki page anyway
17:33:14 <jlvillal> Anything else?
17:33:50 <krtaylor> till next week then
17:33:57 <mgould> has anyone worked with the openstack-puppet test infrastructure before?
17:34:09 <jlvillal> mgould: Let's wait until the next topic :)
17:34:20 <mgould> oh, sorry, I thought we'd moved on to AOB
17:34:34 <jlvillal> #topic Open Discussion / General QA topics
17:34:41 <jlvillal> mgould: go! :)
17:34:56 <mgould> hurrah!
17:35:22 <krtaylor> mgould, there are a bunch of test teams that have, what did you need?
17:35:25 <mgould> has anyone worked with the openstack-puppet test framework before?
17:35:26 <jlvillal> I have not used it.  Except for in my devstack-gate-test it runs some puppet stuff to setup the environment
17:36:07 <mgould> krtaylor, my current question is "why does it keep checking out the master branch whenever I try to run the unit tests?"
17:37:18 <jlvillal> mgould: Might be able to get some help over at #openstack-qa or #openstack-infra
17:37:43 <mgould> ie, commit changes -> run tests -> tests pass -> oh, that's because it was testing a different branch
17:37:51 <mgould> jlvillal, I'll try that, thanks
17:37:56 <krtaylor> mgould, I'd need more info, btw, openstack-infra is a good place, or I set up a channel for third-party-ci
17:38:15 <krtaylor> called #openstack-third-party-ci   :)
17:38:21 <jlvillal> Anything else to discuss?
17:38:23 <mgould> sensible name :-)
17:39:16 <jlvillal> Okay I think that does it.
17:39:21 <jlvillal> Thanks everyone!
17:39:28 <jlvillal> #endmeeting