17:01:43 <mtreinish> #startmeeting qa
17:01:44 <openstack> Meeting started Thu Dec  5 17:01:43 2013 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:47 <openstack> The meeting name has been set to 'qa'
17:02:00 <mtreinish> hi who do we have here today?
17:02:03 <sdague> o/
17:02:07 <mlavalle> hi
17:02:07 <dkranz> o/
17:02:20 <andreaf> hi
17:02:28 <andreaf> hi
17:02:32 <giulivo> hi
17:02:37 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_December_5_2013
17:02:38 <adalbas> hi
17:02:43 <mtreinish> ^^ Today's agenda
17:02:57 <mtreinish> let's get started
17:03:05 <mtreinish> #topic Meeting time change proposal
17:03:10 <dkranz> afazekas: meeting
17:03:23 <mtreinish> so I sent out an email to the ml about this
17:03:30 <sdague> mtreinish: yep, thanks for that
17:03:41 <mtreinish> we need to change the meeting time, I suggested 2200 utc
17:03:48 <mtreinish> which is a bit difficult for people in Europe
17:04:04 <sdague> I just wanted to make sure to highlight it to people so they got on that thread if they weren't already
17:04:10 <afazekas> hi
17:04:17 <mtreinish> so I was thinking we could alternate between this time and then (on Thursday)
17:04:18 <giulivo> mtreinish, 22utc is good for me though (europe)
17:04:26 <mtreinish> giulivo: ok cool
17:04:32 <giulivo> is other "europe" against it?
17:04:42 <mtreinish> I'd prefer to keep one time
17:04:47 <sdague> giulivo: mkoderer said it was going to be pretty late for him
17:04:48 <mtreinish> mkoderer was hesitant
17:04:51 <dkranz> mkoderer was not thrilled but said he could do it
17:04:57 <giulivo> oh okay sorry
17:05:03 <giulivo> afazekas ?
17:05:10 <dkranz> What about 1 hour earlier?
17:05:18 <mtreinish> it's too early for Japan then
17:05:28 <mtreinish> 6am
17:05:31 <dkranz> I thought they said they could do 6am
17:05:45 <sdague> dkranz: it also ends up on top of the nova meeting then, right?
17:05:57 <dkranz> sdague: Don't know
17:05:58 <giulivo> yet, isn't it three of us in europe? afazaks will make the difference
17:06:12 <dkranz> But 22:00 is midnight in europe most of the year
17:06:49 <mtreinish> well anyway we can take this back out to the list, and do a final vote next week
17:07:02 <giulivo> dkranz, indeed it is late but altenating meetings means we would loose 1 in 2 anyway ... will midnight be better or worst than that?
17:07:12 <sdague> yep, I'm honestly going to be more +1 on oscillating just because it won't discourage new .eu people
17:07:30 <mtreinish> sdague: I was just worried about the disconnect then
17:07:31 <dkranz> I agree. And if giulivo wants to go to the other at midnight that is ok
17:07:46 <dkranz> But it is a tough call
17:07:57 <dkranz> Making it 1 doesn't help if people don't show up
17:07:59 <afazekas> giulivo: I will try to solve to attending the meeting at whatever time
17:08:14 <mtreinish> how about next week we do the 22 UTC and start oscillating?
17:08:25 <sdague> mtreinish: that works for me
17:08:33 <sdague> want to propose that back to the list?
17:08:38 <dkranz> Sounds reasonable
17:08:47 <mtreinish> sdague: yeah will do
17:08:51 <mtreinish> and I'll update the wikis
17:08:54 <adalbas> it can start oscillating and then check if stick to a single time
17:09:00 <mtreinish> if it doesn't work well we can always revisit it again
17:09:05 <sdague> +1
17:09:11 <mlavalle> +1
17:09:19 <dkranz> 22:00 is the best fallback we have so +1
17:09:19 <giulivo> +1
17:09:28 <mtreinish> #action mtreinish to update meeting times to oscilate between current time and 2200 UTC
17:09:37 <mtreinish> ok then lets move on
17:09:46 <mtreinish> #topic Getting additional folks signed up as point people
17:09:51 <mtreinish> sdague: this is your's
17:09:54 <sdague> yep
17:10:24 <sdague> so basically I'm realizing that there are too many discreet things to keep an eye on in the QA program for me to handle it all well
17:10:28 <sdague> or any future PTL
17:10:34 <dkranz> Agreed
17:10:44 <sdague> so I'd like to carve off point people for specific things
17:11:11 <sdague> which isn't that they'd own all the work, they would just be the point person to make sure we stay on top of it, and harass folks when we don't
17:11:34 <sdague> I figured 3 things that could be carved off this way are: meeting management, blueprint triage, and bug triage
17:11:55 <sdague> mtreinish volunteered for meeting management (hence why he's running the meeting and handling the time change)
17:12:06 <sdague> but I'd still like to get volunteers for the others
17:12:10 <dkranz> I think the problem is the things you are carving off are things that in most engineering orgs are done by managers. We should recruit managers to do them.
17:12:29 <sdague> dkranz: well our community doesn't work that way
17:12:40 <sdague> people that do the work, get the trust
17:12:50 <dkranz> sdague: Why not? If a company wants to donate manager time for this...
17:13:20 <sdague> dkranz: ok. But we don't see those folks as contributors today
17:13:27 <dkranz> sdague: I know
17:13:38 <dkranz> sdague: But we have not tried to get them either
17:13:39 <sdague> so I don't think that's a productive direction for the conversation
17:13:58 <sdague> because honestly, point people need to be folks the group trusts
17:14:32 <dkranz> sdague: ok, i'll stop here. But I don't know why being a good hacker makes you more trustworthy at bug triage
17:14:32 <sdague> because if they don't trust them, then it's sort of useless
17:15:06 <dkranz> I'm not talking some random manager but some one who is engaged.
17:15:07 <afazekas> looks like we still have bugs in Fix Committed status, does anyone has objection against moving them to the Fix released status ?
17:15:25 <giulivo> afazekas, I wanted to ask that too, that's a +1 from me
17:15:27 <sdague> dkranz: I'm not saying it does. I'm saying there are no non developer candidates in our community that are doing work
17:15:29 <dkranz> That was supposed to happen automiatically but there was a window where it didn't
17:15:37 <mtreinish> afazekas: that's kinda off topic, but they should be moved (we do it by default now)
17:15:52 <dkranz> sdague: RIght. I just had some idea to try changing that
17:15:54 <sdague> so... this is exactly why we need a point person :)
17:16:09 <dkranz> sdague: Agreed. And it should be one of us.
17:16:17 <sdague> because now the conversation went off into the weeds on bp details, which have gotten lost, because there wasn't one
17:16:38 <sdague> or more specifically, the person was me, but it's about #12 down my priority list
17:16:44 <sdague> so effectively will never get done
17:17:20 <giulivo> so I won't bother, but I don't like this "single point of failure" thing
17:17:31 <adalbas> sdague, do you expect this point person to be amongst the core people, or not necessarily?
17:17:57 <sdague> adalbas: not necessarily, especially for bugs
17:18:00 <dkranz> let's separate bug triage from blueprints because they are a different magnitude of task
17:18:04 <sdague> yep
17:18:14 <sdague> and different levels of trust needed I think
17:18:35 <giulivo> but my long term idea is to do some hard work now and later block new blueprints to not find ourselves in the same situation again
17:18:37 <sdague> so our bug group is open, anyone can triage, so I really think any volunteer on bug triage point person would be welcomed
17:18:50 <afazekas> The problem with random failure bugs, in most cases we cannot do anything with them in tempest, how to handle them ?
17:19:04 <giulivo> afazekas, that's why I'd drop non triaged
17:19:07 <sdague> afazekas: ok, again, moving into the details
17:19:19 <sdague> because the point person would get to make some of those calls
17:19:43 <sdague> giulivo: so it sounds like you are actually taking a run at the blueprint cleanup anyway
17:19:47 <adalbas> sdague, could you point what the bug triage person would be doing (responsibilities)?
17:20:02 <giulivo> sdague, hehehe yes I'd like to indeed!
17:20:04 <sdague> you want to be the point person for that for now
17:20:14 <sdague> giulivo: I think yuo'd be great at it :)
17:20:40 <sdague> adalbas: QA bug czar (Spearheading Tempest Bug triage)
17:20:40 <sdague> * focus on driving down Tempest bug queue to a useful thing
17:20:40 <sdague> * nag about top QA bugs in tempest meeting
17:20:40 <sdague> * organize bug triage days regularly to help with this
17:21:27 <giulivo> but really, why do we need new blueprints if not discussed here or by a core? does "closing" blueprints look so terrible?
17:22:05 <sdague> giulivo: I don't think so. Again we just need someone to stay on top of it and make sure it doesn't get out of hand
17:22:16 <dkranz> That is the direction nova is going in
17:22:16 <sdague> hint hint, you sound like a great volunteer for this :)
17:23:17 <giulivo> dkehn, you mean closing "new blueprints to trusted" ?
17:23:21 <giulivo> dkranz, ^^
17:23:37 <mtreinish> ok, well lets move to the next topic now
17:23:44 <dkehn> ??
17:23:48 <mtreinish> sdague can pester people to volunteer after the meeting
17:23:49 <dkranz> giulivo: No, just the idea of exerting more control over accepted blueprints sooner
17:23:49 <sdague> yeh, I guess we still aren't getting volunteers on it
17:23:51 <adalbas> sdague, if anyone else from the core team are not able to be the point for that and if you are all ok with that, I would be ok to help with the bug triage thing
17:23:52 <sdague> yep
17:24:10 <mtreinish> #topic Blueprints
17:24:12 <sdague> adalbas: yes, +1. I'm totally happy if yuo wanted to lead the bug triage effort
17:24:42 <mtreinish> oops maybe jumped the gun a bit there
17:24:53 <sdague> mtreinish: nope, it's all good
17:25:02 <sdague> let's move on
17:25:03 <mtreinish> are there any outstanding blueprints that need attention?
17:25:09 <andreaf> I have https://blueprints.launchpad.net/tempest/+spec/input-scenarios-for-scenario
17:25:44 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/input-scenarios-for-scenario
17:26:11 <andreaf> it's something I suggested we could do at the summit, sdague mentioned it would be a good idea, so I started working on it
17:26:27 <mtreinish> andreaf: ok, cool
17:26:50 <giulivo> so I'm moving that into accepted
17:26:50 <mtreinish> andreaf: so just need the bp approved?
17:26:59 <mtreinish> do you have a WIP up?
17:27:17 <sdague> giulivo: +1
17:27:36 <mtreinish> giulivo: you just volunteered...
17:27:37 <andreaf> mtreinish: I only have it locally, I can upload  a WIP
17:28:16 <mtreinish> andreaf: ok cool, that would help my understanding of what you're doing there
17:28:17 <sdague> andreaf: cool, would be great
17:28:23 <mtreinish> but it sounds like everyone is on board
17:29:10 <andreaf> sdague, mtreinish: ok cool
17:29:22 <mtreinish> ok are there any other bp status updates that people have
17:29:49 <dkranz> mtreinish: Just that the negative test stuff is not really moving quickly
17:30:04 <dkranz> mtreinish: There will probably not be any progress until after the new year
17:30:15 <sdague> dkranz: is there a bp registered for that one?
17:30:28 <dkranz> sdague: I thought there was but I'll check
17:30:50 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/negative-tests
17:30:50 <giulivo> dkranz, , https://blueprints.launchpad.net/tempest/+spec/negative-tests ?
17:31:03 <dkranz> Yes
17:31:27 <adalbas> dkranz, i m working with kein omich and others to get the separate files for negative tests
17:31:35 <dkranz> I've been swamped and Marc is out until new year
17:31:40 <adalbas> the patches are added on that blueprint
17:31:41 <sdague> so we should probably make 2 blueprints
17:31:44 <dkranz> adalbas: Yes, but that is not what this is about
17:31:53 <sdague> adalbas: can you make another blueprint on the separation?
17:31:54 <dkranz> There should be 2
17:32:04 <adalbas> sdague, dkranz , yes
17:32:08 <sdague> we'll put them both into an approv state
17:32:11 <adalbas> i can separate them
17:32:24 <adalbas> i ll line that up with the others working on that
17:32:44 <giulivo> sdague, adalbas I can draft and approve it
17:32:58 <giulivo> you meant to have two one for "autogeneration" and one for "separating negative files" right
17:33:01 <giulivo> ?
17:33:06 <adalbas> yes giulivo
17:33:11 <dkranz> right
17:33:57 <mtreinish> ok, are there any other blueprints we need to discuss, otherwise let's move on to the next topic
17:34:37 <mtreinish> #topic Neutron testing
17:34:46 <mtreinish> ok mlavalle your name is on this one
17:34:58 <mlavalle> yeap
17:35:16 <mlavalle> Over the past few days I've been conducting a gap analysis for API tests
17:35:37 <mlavalle> I put the results in the API section of https://etherpad.openstack.org/p/icehouse-summit-qa-neutron
17:36:11 <mlavalle> I've finished with core and l3 api's and plan to continue with L4 - L7 over the next few days
17:36:18 <sdague> great
17:36:47 <sdague> so are we at the point of turning on the full runs non voting on normal (non experimental) neutron runs?
17:36:59 <mlavalle> I will also create a wiki page for new contributors, walking them through the process of writing api tests for Neutron
17:37:01 <sdague> i.e. how many fails are we down to now?
17:37:10 <mtreinish> sdague: I thought we weren't going to do that until parallel worked
17:37:34 <sdague> ah, right, it will get too long
17:37:47 <mlavalle> That wiki will be done early next week and when it';s done i'll advertise it in the ML
17:37:49 <mtreinish> mlavalle: yeah, that would be helpful. You've got ~52 tests listed there right?
17:37:57 <sdague> mlavalle: great
17:38:31 <mlavalle> Last nigh I made a little change to the Neutron wiki https://wiki.openstack.org/wiki/Neutron
17:38:51 <mlavalle> Look at the Related Projects section close to the top
17:40:03 <mlavalle> I am also attending every week the Neutron IRC meeting, so I keep them abreast of what is going on with QA
17:40:38 <sdague> so do we know where we stand on parallel
17:40:49 <mtreinish> salv-orlando: ^^^
17:40:51 <afazekas> Do we able to create a shared network in parallel safe way? Most of tempest test assumes there is only one network visible in the tenant
17:40:56 <sdague> i.e. agent rewrite
17:41:03 <salv-orlando> hi
17:41:05 <mlavalle> salv-orlando is working on several patches to Neutron to enable that
17:41:34 <mlavalle> he sent an email to the ML a couple of day ago with an update on the status
17:41:44 <salv-orlando> I know where we stand. To cut a long story short we have fixed the OVS agent, which does not have anymore tremendous lag for wiring ports, but..
17:42:35 <salv-orlando> … there is also an issue with the DHCP agent, which takes about 40 seconds in updating the hosts file under load and this causes a failure since the cirros vms send 1 dhcp discover per minute
17:43:05 <salv-orlando> I am working on that too. So far I am pushing all the stuff I have in a single patch, where I run the parallel job
17:43:21 <salv-orlando> as soon as I have 10 green runs in a row I will push the patches for review with unit tests and stuff
17:43:24 <salv-orlando> end of the updated.
17:43:53 <mtreinish> salv-orlando: ok, very cool
17:43:56 <mlavalle> salv-orlando: based on my work with bug 1251448, we also need to worry when we devete ports
17:43:58 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] https://launchpad.net/bugs/1251448
17:44:00 <mlavalle> delete
17:44:22 <salv-orlando> and btw, I reckon that with armax's fix for tempest not relinquishing public IPs + another proposed fix for making the public network a /24 we shall be enable to enable full isolation
17:44:35 <salv-orlando> mlavalle: update my outside of the meeting with this issue, please
17:44:48 <mlavalle> salv-orlando: cool
17:44:54 <mtreinish> salv-orlando: ok, when that goes through let me know and we can push through turning that on again
17:45:41 <mtreinish> is there anything else on the neutron testing front?
17:45:51 <mlavalle> not on my side
17:45:59 <mtreinish> ok then let's move on
17:46:07 <mtreinish> #topic Critical Reviews
17:46:19 <mtreinish> does anyone have any reviews they'd like to bring up?
17:46:35 <mtreinish> I actually have one today: https://review.openstack.org/#/c/58604/
17:46:45 <dkranz> mtreinish: I don't think we should discuss it now but please look at https://review.openstack.org/#/c/54948/
17:47:02 <mtreinish> #link https://review.openstack.org/#/c/54948/
17:47:07 <dkranz> mtreinish: which is a disupte about parallelism and tenant isolation requirements
17:47:09 <mtreinish> #link https://review.openstack.org/#/c/58604/
17:47:30 <mtreinish> dkranz: ok, I'll look at it after the meeting
17:48:12 <mtreinish> on the review I posted it has 2 -1's but I responded to them with no response so if someone else could take a look at them I'd appreciate it
17:48:22 <mtreinish> (the -1s probably mean that no one looks at it)
17:48:30 <mtreinish> are there any other reviews to bring up?
17:48:34 <sdague> https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/nova-v3-api-tests,n,z
17:48:49 <sdague> it would actually be nice to move those forward, they have been there for a while
17:49:00 <sdague> and remain about 1/3 of our queue
17:49:10 <mlavalle> mtreinish: Over he weekend I will submit a patch to add tenant isolation to the Neutron base test. We will need to approve that quickly to enable new contributors to work on Neutron API tests. I'll ping you when it's in Gerrit
17:49:12 <sdague> oh, only 20% now
17:49:27 <dkranz> sdague: I have been reviewing these
17:49:28 <mtreinish> mlavalle: that is already up on gerrit
17:49:43 <dkranz> sdague: And will continue and hopefully be done by tomorrow
17:49:44 <mtreinish> mlavalle: https://review.openstack.org/#/c/53459/
17:49:57 <mlavalle> oh, so maybe someone picked it up from the ether pad…. great!
17:49:58 <sdague> I guess the only other issue is we really need to not be +2ing heat patches that we can't run
17:50:10 <sdague> I've been -1ing the stuff that modifies heat slow jobs
17:50:19 <sdague> because that doesn't run at all yet
17:50:39 <dkranz> sdague: The problem is that they don't run due to an infrastructure issue
17:50:45 <sdague> hopefully the f20 release next week will let stevebaker get the infrastructure working
17:50:48 <dkranz> sdague: Tempest tests are a high priority for that team
17:50:57 <dkranz> sdague: I hope so
17:51:10 <sdague> dkranz: then getting the infrastructure working to run them, or running 3rdparty testing needs to be a priority
17:51:16 <dkranz> sdague: If not then we either have to accept them or force people out of upstream
17:51:30 <dkranz> sdague: I'm not really sure what the blocker is
17:51:38 <dkranz> sdague: I'll inquire about that and get a status
17:51:44 <mtreinish> dkranz: we can't really add tests until we can confirm they work. We've had issues with that in the paste
17:51:51 <mtreinish> s/paste/past/
17:51:52 <sdague> so we actually went through it on IRC
17:52:08 <sdague> disk-image-builder needs pip 1.5 to install
17:52:12 <sdague> which is not yet released
17:52:13 <dkranz> mtreinish: We have, but this is a little different because they only run in the expewrimental queue
17:52:20 <sdague> because they need real guests
17:52:21 <dkranz> mtreinish: ANd won't be lit up until they all run
17:52:34 <sdague> dkranz: no it's not different
17:52:47 <sdague> the point is we'd have a ton of tests that we have no proof in the community they ever ran
17:53:13 <sdague> it's the same reason we don't accept tests that start skipped
17:53:31 <mtreinish> dkranz, sdague: lets save this for after the meeting. We're down to 7 min. and dkranz still has a topic on the list
17:53:36 <sdague> sure
17:53:39 <dkranz> sdague: I don't completely agree but let's see if it can be resolved soon and made moot
17:53:51 <mtreinish> #topic How can we make sure reviews don't slide >1 week
17:54:00 <dkranz> mtreinish: I think I can withdraw my topic for now since we are doing ok
17:54:00 <mtreinish> dkranz: you're up
17:54:14 <mtreinish> dkranz: heh, ok cool
17:54:21 <mtreinish> then let's open the floor
17:54:22 <dkranz> We should just all make sure to focus on old reviews first
17:54:29 <mtreinish> #topic open discussion
17:54:30 <dkranz> even thought it is a pain
17:54:42 <mtreinish> dkranz: yeah I always start at the bottom
17:54:42 <dkranz> mtreinish: Question about neutron
17:54:57 <mtreinish> dkranz: ok, shoot
17:55:06 <sdague> yeh, I do as well. though I'm trying to actually start with my review list, so I'm not missing coming back to things
17:55:07 <dkranz> How many of us are comfrotable reviewing neutron complex scenarios for functional correctness?
17:55:22 <dkranz> I'm not
17:55:33 <sdague> dkranz: I typically try to ask someone from neutron core to comment
17:55:54 <mtreinish> dkranz: yeah the same with me, I'm not super familiar with it at this point
17:56:08 <mlavalle> dkranz: I can help with it as well
17:56:08 <dkranz> sdague: That is the right answer but it might help to have a slightly more formal way to spread the load on the neutron team for this
17:56:10 <sdague> that's actually on my list after sorting out volunteers on the tempest items
17:56:24 <dkranz> sdague: ok, great
17:56:42 <dkranz> sdague: I don't want reviews to slide because we are not comfortable pushing them forward
17:56:43 <sdague> dkranz: so I think the actual solution there is getting one or more point people in each core project that are there to help
17:56:50 <dkranz> sdague: Right
17:57:02 <giulivo> sdague dkranz +1 from on that
17:57:07 <dkranz> sdague: That's the only way that scales
17:57:08 <giulivo> *from me
17:57:22 <sdague> which I'm going to run on, once I have volunteers signed up for the tempest tasks. We need our house in order first before we ask others for things
17:57:24 <dkranz> sdague: Never mind our three new projects :)
17:58:18 <malini> dkranz, sdague: o/  potential marconi-tempest contributor snooping
17:58:25 <sdague> malini: great
17:58:46 <sdague> ok, we're about to loose the room
17:58:58 <mtreinish> yeah, lets end it here for today
17:59:01 <mtreinish> thanks everyone
17:59:07 <mtreinish> #endmeeting