#openstack-meeting log

22:00:13 <mtreinish> #startmeeting qa
22:00:14 <openstack> Meeting started Thu Jun 26 22:00:13 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:17 <openstack> The meeting name has been set to 'qa'
22:00:23 <mtreinish> hi who's here today?
22:00:35 <asselin> hi
22:00:38 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_June_26_2014_.282200_UTC.29
22:00:40 <dpaterson> hi
22:00:44 <mtreinish> ^^^ Today's agenda
22:00:59 <mlavalle> hi
22:01:01 <masayukig> hi
22:02:03 <mtreinish> sdague: you around?
22:02:04 <mtreinish> ok well let's get started
22:02:05 <dkranz> Any one here?
22:02:17 <mtreinish> #topic Juno specs proposal cutoff date? (mtreinish)
22:02:20 <dpaterson> hi
22:02:50 <mtreinish> so this is something I've been thinking about because the reviews on the qa-specs repo have been moving slowly
22:03:19 <mtreinish> as a way to try and increase velocity there we set a cutoff date for when we stop accepting specs targetted for juno
22:03:42 <asselin> when is that?
22:04:03 <mtreinish> asselin: I was thinking the end of July, but I wanted some feedback before we started enforcing it
22:04:08 <dkranz> mtreinish: I don't think we should stop accepting specs until there  is not time to implement
22:04:08 <mtreinish> does it sound like a good idea?
22:04:14 <sdague> mtreinish: oh, yeh
22:04:29 <dkranz> mtreinish: what is the purpose?
22:04:50 <mtreinish> dkranz: I'm just trying to think of ways to motivate reviews on the specs repo
22:05:09 <mtreinish> it goes both ways because a bunch of reviews have sat there with -1's and no response for a while
22:05:19 <sdague> mtreinish: nova just did a specs review day, that was handy
22:05:20 <dkranz> mtreinish: I think that is an issue for the core team. Something is making us not want to review them even as we do other stuff
22:05:39 <dkranz> sdague: I think that is a great idea
22:06:03 <sdague> dkranz: sure, honestly, I'll say my overall review level has dropped in the last few weeks as I've been fire fighting a lot
22:06:17 <mtreinish> sdague: yeah I think that's a good idea, although we don't have nearly the same amount of specs proposals for qa-specs
22:06:56 <sdague> mtreinish: sure, but if we carve out a day, we can go through them all. We could even post a hangout or something and talk through them as well in a social way
22:06:59 <dkranz> mtreinish: but we have enough to notice it is an issue
22:07:08 <dkranz> sdague: +++
22:07:47 <dkranz> a little talk can go a lot further than communicating in gerrit with +1/-1
22:08:08 <mtreinish> ok, I think we can do a spec review day, hopefully it will start moving things there
22:08:21 <mtreinish> I'll send a ML thread out about doing
22:09:08 <mtreinish> #action mtreinish to schedule a specs review day
22:09:37 <mtreinish> ok, I guess we'll give this a try and we can revisit having a proposal deadline if we still have a problem
22:09:55 <dkranz> great
22:09:57 <masayukig> mtreinish: great
22:10:04 <mtreinish> let's move on to the next topic then
22:10:14 <mtreinish> #topic Specs Review
22:10:23 <dkranz> :)
22:10:29 <mtreinish> ok does anyone have any specs that they would like to bring up for discussion?
22:10:51 <mtreinish> dkranz: heh, I'm glad someone noticed that :)
22:10:57 <dpaterson> #link https://review.openstack.org/#/c/91725/
22:11:42 <dkranz> dpaterson: I think Matt was just asking for clarification
22:11:46 <mtreinish> dpaterson: ok I gave it a -1 because of an inconsitency on how the preserved state will be defined
22:12:25 <mtreinish> you mentioned both a JSON file and using the config file. You should only need one, and if it's JSON you should clarify what the format is
22:12:51 <dpaterson> let me take a look, state is json only
22:13:18 <mtreinish> dpaterson: you can follow up with me in -qa after the meeting
22:13:33 <mtreinish> are there any other specs to discuss?
22:14:30 <mtreinish> ok then let's move on
22:14:30 <dkranz> mtreinish: we are making no so much progress on the generate config spec
22:15:02 <mtreinish> dkranz: yeah I noticed that. Do you want to just start writing a separate one
22:15:32 <dkranz> mtreinish: I may do that. I've been trying to wait because boris-42 told me they were actually doing stuff.
22:16:12 <mtreinish> yeah, I mean we're already into J-2 and I was going to mark that bp as essential when the spec was approved
22:16:20 <mtreinish> I just want to make sure we have enough time for implementation
22:16:28 <dkranz> mtreinish: Right
22:16:48 <dkranz> But Juno does not really have as much meaning for tempest
22:17:05 <dkranz> BUt we should get going. I have a feeling they may be coding away.
22:17:11 <mtreinish> we still frame our dev cycle on the same cadence though, even if the milestone doesn't mean anything
22:17:44 <mtreinish> although I will tag a release then too
22:17:49 <dkranz> mtreinish: I will ping him again
22:17:53 <mtreinish> dkranz: ok
22:18:07 <dkranz> mtreinish: I don't really want to work on a spec only to have some one else submit code right after it is done.
22:18:31 <mtreinish> dkranz: but then we'll -2 the code if doesn't align with the spec we worked on
22:18:42 <mtreinish> the whole point of this was to decide a technical direction before code is submitted
22:18:45 <mtreinish> so we don't waste time
22:18:56 <dkranz> mtreinish: I know that, but not every one buys into that.,
22:19:11 <dkranz> mtreinish: anyway, I will talk to him and decide
22:19:34 <mtreinish> ok, are there any other specs? Otherwise let's move on to the next topic
22:20:15 <mtreinish> #topic Blueprints
22:20:28 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance
22:20:36 <mtreinish> so I wanted to discuss what we should do about this BP
22:20:44 <mtreinish> since the V3 api isn't going to exist in the same form
22:21:09 <mtreinish> I don't think reorganizing the nova api tests like proposed in that bp makes much sense anymore
22:21:15 <sdague> agreed
22:21:27 <dkranz> right
22:21:40 <sdague> especially as we're still working out what the nova api evolution strategy is
22:22:01 <mtreinish> masayukig: is Ken'ichi around?
22:22:11 <masayukig> no..
22:22:32 <mtreinish> sdague: yeah, I'm was going to close the BP, but I wanted to bring it up first
22:22:36 <masayukig> oh, he's just come!
22:23:01 <mtreinish> masayukig: ok cool, I just wanted to make sure he wasn't opposed to closing it, since he's the owner
22:23:31 <mtreinish> once the nova api evolution strategy is worked out we can rework the tempest test structure based on that
22:23:35 <dkranz> IMO we need to get v3 out of base classes
22:23:47 <dkranz> mtreinish: Is it not certain that v3 is gone?
22:24:34 <mtreinish> dkranz: I think it's going to exist, just not as v3
22:24:41 <sdague> dkranz: it is agreed that it's not going to be what it is
22:24:55 <sdague> but what the strategy going forward is still.... under discussion
22:25:28 <dkranz> sdague: ok, but the current structure tries to map two monolithic apis. And that is definitely off the table for v3, right?
22:25:31 <sdague> there are 2 competing definitions of micro versions at the moment (one is mine, so it's still evolving)
22:25:39 <sdague> dkranz: maybe
22:25:44 <dkranz> sdague: Is this written up somewhere?
22:25:50 <sdague> yeh, nova-specs
22:25:58 <dkranz> sdague: ok, I should look at that
22:26:13 <mtreinish> I think in the meantime the status quo for tempest is ok
22:26:14 <sdague> https://review.openstack.org/#/c/101648/ and https://review.openstack.org/#/c/101648/
22:26:17 <sdague> oops
22:26:29 <sdague> https://review.openstack.org/#/c/96139
22:26:37 <dkranz> sdague: got it
22:26:43 <mtreinish> we won't do any more of the refactor until the nova direction is decided
22:26:48 <sdague> I would agree, status quo until this shakes out
22:27:07 <dkranz> sdague: ok. I see dan smith -1 both :)
22:27:13 <oomichi> mtreinish: agree
22:27:13 <mtreinish> oomichi: any opposition? ^^^ since it's your bp
22:27:18 <mtreinish> oomichi: ok, cool
22:27:46 <mtreinish> ok then for the other bps are there any updates?
22:28:07 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/branchless-tempest
22:28:12 <mtreinish> sdague: ^^^ anything on that
22:28:17 <mtreinish> since it's marked as essential :)
22:28:33 <sdague> nothing new
22:29:02 <mtreinish> It's still just the finishing up the feature matrix stuff right?
22:29:09 <sdague> yes
22:29:20 <mtreinish> ok
22:29:26 <sdague> honestly, what it might be worth doing is closing out that as done
22:29:39 <sdague> and creating an new spec/bp with the extensions part of the feature matrix
22:29:50 <sdague> then it would be documented at least
22:30:22 <mtreinish> sdague: ok, that makes sense to me. I'll defer to you on how to organize it
22:30:29 <sdague> because realistically we haven't really had angry villagers yet
22:30:54 <mtreinish> do you want to write up a spec for the extensions and optional features part of the feature matix
22:31:17 <mtreinish> I'll close out the current bp when that's up
22:31:19 <dkranz> sdague: I don't think any one will be angry until they have to backport something to icehouse due to it.
22:31:53 <sdague> mtreinish: works for me
22:32:02 <mtreinish> sdague: ok cool
22:32:14 <sdague> I'll make tomorrow my writing day, as I think I don't have anything on fire atm :)
22:32:15 <mtreinish> #action sdague to draft a spec for finishing the feature matrix
22:32:34 <mtreinish> ok are there any other bps to discuss?
22:33:17 <mtreinish> #topic Grenade
22:33:31 <mtreinish> sdague: the floor is yours
22:34:07 <sdague> We got the volume support for javelin2 in
22:34:25 <sdague> EmilienM is going to be looking at using javelin2 tomorrow
22:34:57 <mtreinish> ok cool
22:35:04 <sdague> there is also a devstack change that merged https://review.openstack.org/#/c/101004/
22:35:13 <mtreinish> did the patch merge? I know "we" approved it
22:35:25 <sdague> which *should* help on the service not starting on the new side
22:35:29 <sdague> mtreinish: good question
22:35:47 <sdague> nope
22:35:50 <mtreinish> sdague: has the devstack change made a noticeable difference? or is too soon to tell?
22:35:56 <sdague> https://review.openstack.org/#/c/100105/
22:36:06 <sdague> mtreinish: we'll need ES back on track to see
22:36:16 <sdague> right now I think we're close to 24hrs behind
22:36:39 <mtreinish> yeh, that sucks
22:36:44 <mtreinish> we've become very dependent on ES
22:37:02 <sdague> yep
22:37:10 <sdague> Bug 1331274 - Logs are not getting created for services in Grenade
22:37:11 <uvirtbot> Launchpad bug 1331274 in grenade "Logs are not getting created for services in Grenade" [Undecided,New] https://launchpad.net/bugs/1331274
22:37:14 <sdague> is the tracking bug for that
22:37:38 <sdague> though realistically, that seems to flatline within our signal range
22:37:42 <sdague> so we might be fixed there
22:38:05 <sdague> that's probably about it
22:38:08 <mtreinish> ok, cool
22:38:20 <mtreinish> does anyone else have something to discuss about grenade?
22:38:52 <mtreinish> #topic Neutron Testing
22:38:57 <mtreinish> mlavalle: around?
22:40:07 <mtreinish> ok we can come back to this topic if he shows up
22:40:34 <mtreinish> salv-orlando email had a good update on the status of parallel full neutron jobs
22:40:46 <mtreinish> which was what we discussed during last weeks meeting
22:40:59 <mtreinish> let's move on
22:41:02 <mtreinish> #topic Bugs
22:41:04 <salv-orlando> ping me if you need more info on the matter.
22:41:10 <dkranz> mtreinish: one question about that
22:41:26 <sdague> salv-orlando: yeh, status on that would be cool actually
22:41:38 <dkranz> mtreinish: I'm going to be at the mid-cycle for neutron. Is there anything we want me to pay attention to from our standpoint?
22:41:40 <mtreinish> salv-orlando: heh I figured it was too late for you
22:41:53 <salv-orlando> it’s never too late. These are my productive hours… so
22:42:14 <salv-orlando> the status is simple we have identified the failures which occur in the full job but not in the smoketests
22:42:33 <salv-orlando> funny thing we noticed was that these failures occured on smoke tests - but only in the full job
22:42:45 <salv-orlando> and then we realized we never enabled parallelism on the smoke job
22:43:16 <salv-orlando> which is a shame because back in april parallel tests had less than 5% failure rate
22:43:39 <mtreinish> salv-orlando: oh yeah I guess we never did
22:43:41 <salv-orlando> what we are seeing are two things: a race in nova due to the nova/neutron notification mechanism
22:43:58 <salv-orlando> mtreinish: we had that patch back in january that was lost after that massive gate outage
22:44:13 <salv-orlando> and there is a patch for that - for which I need to add a UT
22:44:51 <salv-orlando> the other problem we’re seeing are notifications not sent at all from neutron; which could mean two things - either the notification system has some bug which shows up only in parallel jobs - or the l2 agent is broken again
22:45:22 <salv-orlando> that’s all. I’m looking logs and by next monday I will have root cause for all bugs and hopefully patches
22:45:26 <salv-orlando> one last thing
22:45:37 <mtreinish> salv-orlando: I could see either likely being the case :)
22:46:36 <salv-orlando> the lock wait timeout error… that’s another error that is more likely to happen with a parallel job. We have already people working on fixing those - but you know how it is with an opinionated developer community. So even if those failures are not fixed yet I would still enable the job and make it voting
22:46:59 <salv-orlando> mostly because we’re now merging DVR patches which do a lot of changes in L2 and L3 agents - I expect a lot of regresssions as well
22:47:06 <salv-orlando> and I’d love to catch them at the gate
22:47:12 <salv-orlando> now it’s really all
22:47:18 <mtreinish> salv-orlando: that sounds good to me
22:47:30 <sdague> salv-orlando: yeh, honestly, I'm kind of ok with the idea of full job on neutron and not on others
22:47:53 <sdague> that you proposed, which I know mtreinish didn't like :)
22:47:58 <mtreinish> sdague: I dunno asymetrical gating always makes me nervous
22:48:04 <salv-orlando> sdague: that’s a debate we already had in the past asymettric vs symmetric
22:48:10 <dkranz> sdague: +1 Least of evils IMO
22:48:24 <mtreinish> everytime we've tried we end up blocking things
22:48:25 <sdague> mtreinish: sure, though if we are looking at potentially a big regression in neutron
22:48:37 <salv-orlando> let’s take it offline - I thing we should go symmetric again once the failure rate is such that we don’t cause a lot of failures in other projects.
22:48:44 <sdague> yeh
22:49:23 <mtreinish> salv-orlando: ok yeah we can discuss it later
22:49:25 <mtreinish> let's move on then
22:49:35 <mtreinish> I did start the bugs topic
22:49:48 <mtreinish> but given time let's move on to critical reviews :)
22:49:57 <mtreinish> #topic Critical Reviews
22:50:08 <mtreinish> so does anyone have any reviews that they would like to bring up?
22:50:46 <mtreinish> the only one I have that just needs another +2 is this spec readme update:
22:50:49 <mtreinish> #link https://review.openstack.org/102314
22:51:15 <mtreinish> and probably this one to stop dumping tracebacks in the logs on ping retries:
22:51:18 <mtreinish> #link https://review.openstack.org/95815
22:52:04 <sdague> https://review.openstack.org/#/c/101702/ - mostly I wanted to get concensus on this hacking extension locally before spending any effort on cleaning it up
22:52:25 <sdague> there are currently about 250 assertTrue/assertFalse in our code base with no msg param
22:52:34 <mtreinish> sdague: I'm +1 on doing that
22:52:49 <mtreinish> I also think assertEqual/NotEqual might be useful to add there too
22:53:10 <sdague> I seem to keep tripping over these things in debugging, True != False is no good
22:53:11 <mtreinish> but sometimes it's fine without it, so just True/False is probably a good starting point
22:53:27 <mtreinish> we can always add another rule if we need it
22:53:47 <sdague> yeh, assertEqual is often ok, I wouldn't hard enforce on that
22:53:55 <sdague> but assertTrue is completely undebuggable
22:54:03 <sdague> when it's false
22:54:37 <mtreinish> sdague: just make sure to put something in HACKING.rst when you add the rule for real
22:54:45 <sdague> yeh
22:54:59 <mtreinish> ok, does anyone else have any reviews?
22:55:22 <afazekas> https://review.openstack.org/#/c/98065/
22:55:56 <mtreinish> afazekas: I thought we skipped the LBaas test
22:56:13 <mtreinish> and I'm not comfortable merging anymore tests for the L7 neutron stuff until we get the full job gating
22:56:22 <afazekas> mtreinish: it was unskipped
22:56:55 <sdague> well it is a fix for an existing test
22:56:55 <mtreinish> afazekas: ok, I'll take a close look at that then
22:56:56 <afazekas> mtreinish: It fixes the lbaas test issue: http://logstash.openstack.org/#eyJzZWFyY2giOiJCYWRTdGF0dXNMaW5lIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDAzODIzMzU0OTEyfQ==
22:57:15 <mtreinish> sdague: yeah, I thought it was a fix/unskip
22:57:44 <mtreinish> afazekas: do you have a bug link?
22:58:08 <afazekas> https://bugs.launchpad.net/tempest/+bug/1326607
22:58:09 <uvirtbot> Launchpad bug 1326607 in tempest "TestLoadBalancerBasic fails in Tempest" [Undecided,In progress]
22:58:33 <mtreinish> ok, thanks
22:59:04 <mtreinish> ok well with 1 min. left, I just wanted to remind people about the midcycle meetup
22:59:16 <mtreinish> if you're planning to attend please put your name on the wiki page
22:59:26 <mtreinish> #link https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
22:59:45 <mtreinish> and if you have ideas for a good topic to discuss face to face ping me
22:59:46 <tpatil_> #link  https://review.openstack.org/#/c/79549
23:00:01 <mtreinish> tpatil_: ok, I'll take a look
23:00:06 <mtreinish> ok well that's time
23:00:09 <mtreinish> thanks everyone
23:00:11 <mtreinish> #endmeeting