09:00:23 <gmann> #startmeeting qa 09:00:24 <openstack> Meeting started Thu Mar 23 09:00:23 2017 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:00:28 <openstack> The meeting name has been set to 'qa' 09:00:39 <gmann> who all here today? 09:00:45 <masayukig> \o 09:00:50 <ltosky[m]> o/ 09:00:57 <chandankumar> \o 09:01:09 <martinkopec> o/ 09:01:28 <gmann> hello everyone 09:01:30 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_March_23rd_2017_.280900_UTC.29 09:01:35 <gmann> ^^ today agenda 09:01:38 <felipemonteiro_> o/ 09:01:58 <zhufl> hello 09:01:58 <gmann> #topic Previous Meeting Action review 09:02:14 <gmann> there was action item for jordan for ML on mem allcoation 09:02:22 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114235.html 09:02:48 <gmann> seems like there are many patches to shrink the mem failure peak 09:03:14 <gmann> but most of the failure from API tests mainly attach/detach volume 09:03:41 <gmann> ll discuss that in gate topic 09:03:50 <gmann> #topic The Forum, Boston 09:04:21 <gmann> so there are 2 forum/sessions in Boston 09:04:39 <gmann> 1. Onboarding sessiosn #link https://etherpad.openstack.org/p/BOS-QA-onboarding 09:05:15 <gmann> this will be 90 min slot with combined with QA/infra/stable/release team 09:06:00 <gmann> and requirement team 09:06:20 <gmann> we will have around 15-20 min slot per each team 09:06:47 <gmann> this is more to talk and show project scope and visibility for new contributors in summit 09:06:51 <gmann> #link https://etherpad.openstack.org/p/BOS-QA-onboarding 09:06:59 <gmann> ideas for that is here ^^ 09:07:08 <masayukig> gmann: ok, but it looks like the number of ideas are too much for the time.. 09:07:28 <chandankumar> gmann: are these sessions going to be recorded? 09:07:37 <gmann> yea, i also thought we should have separate 90 min but i think andreaf is ok with 15-20 min :) 09:07:58 <gmann> chandankumar: do not know. it will be classroom type sessions 09:08:24 <gmann> actually there were room shortage and had to merge with those 5 team. 09:08:33 <prateek> gmann, would those be practical hands on sessions or more on the theory side ? 09:08:37 <gmann> let's see how many new contributor we can get from there 09:08:53 <gmann> prateek: theory due to time limits 09:09:01 <prateek> gmann, ok.. 09:09:06 <masayukig> gmann: yeah.. 09:09:20 <gmann> but there will be open area which we can use for team and new contributor and for hands on if they want 09:09:25 <ltosky[m]> So a big list of RTFM, given the time? :) 09:09:41 <prateek> gmann, sounds good 09:09:54 <gmann> yea we have to manage with that. 09:10:08 <gmann> 2. Brainstorming Forum 09:10:10 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114399.html 09:11:12 <gmann> these will be more cross projects, dev-ops like and each team needs to submit their proposal and selection team will schedule those by april 10 09:11:27 <gmann> deadline to submit sessions is april 2nd 09:11:40 <gmann> there is etherpad andreaf created to gether the ideas #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/114399.html 09:11:54 <gmann> please updates you ideas on that. 09:12:27 <masayukig> gmann: the link of the etherpad? 09:12:29 <gmann> as 2nd april is just after next meeting i am putting Ai for andreaf 09:12:41 <gmann> sorry 09:12:42 <gmann> #link https://etherpad.openstack.org/p/BOS-QA-brainstorming 09:13:01 <masayukig> gmann: thx :) 09:13:19 <gmann> #action andreaf to propose the Forum sessions before 2nd april on #link http://forumtopics.openstack.org/ 09:13:46 <gmann> current proposed sessions can be seen here #link http://forumtopics.openstack.org/ 09:13:55 * andreaf o/ 09:14:15 <gmann> andreaf: just put AI for you on Forum sessions submission 09:14:41 <andreaf> gmann: yeah sure 09:14:45 <gmann> andreaf: any idea how long those sessions will be ? 09:15:07 <andreaf> gmann: uhm good question I have to check - I think it said in some ML 09:15:30 <gmann> ok 09:15:56 <andreaf> gmann: it's an opportunity to talk with openstack users / operators - so I like the idea of a session about downstream use of tempest plugins and other qa tools 09:16:39 <andreaf> gmann: there's still a bit of time to propose things in the etherpad, if you do please put your name as well :) 09:16:40 <gmann> yea we can get valuable (good/bad) feedaback from them 09:17:03 <gmann> andreaf: yea 09:17:04 <andreaf> gmann: yeah thanks for proposing that idea on the etherpad 09:17:17 <gmann> thanks :) 09:17:24 <gmann> andreaf: and for Onboarding sessions, 15-20 min enough ? 09:17:27 <chandankumar> andreaf: gmann are we keeping brainstome topic on etherpad somewhere? 09:17:42 <chandankumar> i mean forum topics 09:17:42 <gmann> chandankumar: #link https://etherpad.openstack.org/p/BOS-QA-brainstorming 09:18:14 <gmann> andreaf: or we catch interested contributor in open area and explain them in more details ? 09:18:19 <chandankumar> gmann: sorry i mean forum topics 09:18:31 <andreaf> gmann: yeah well I initially thought we could use 90 min - but since it's a class type of setting I think it will be more more presentation + q&a style 09:18:36 * masayukig is worried about that some operator might say "Tempest is totally crap!" :-p 09:18:44 <gmann> chandankumar: those are Fourm topic only 09:19:06 <chandankumar> gmann: ack, i will add some more thoughts there 09:19:15 <gmann> masayukig: heh, that will be nice and opportunity to convince or improve ourself 09:19:34 <masayukig> gmann: heh, yeah 09:19:40 <andreaf> gmann: yeah my idea is to do a quick presentation and hopefully attract contributors 09:19:40 <gmann> andreaf: ok, but now we cannot get more time right 09:19:59 <gmann> masayukig: we will miss Jordan in those cases :) 09:20:00 <andreaf> gmann: yeah 15min should be enough for a presentation 09:20:16 <andreaf> LoL 09:20:20 <gmann> andreaf: cool, some fancy slides is expected :) 09:20:39 <gmann> ok so please put more and more ideas there 09:20:48 <gmann> #topic Gate Stability - status update 09:20:57 <andreaf> masayukig, gmann: we did a lot of effort over the years to make tempest better for downstream usage - so any constructive input about where to improve things is welcome 09:21:15 <gmann> yea 09:21:21 <gmann> gate status -> #link https://goo.gl/ptPgEw 09:21:21 <masayukig> andreaf: yup 09:21:34 <jordanP> (hi, I am late) 09:21:44 <gmann> jordanP: hi 09:21:58 <gmann> so its little bit high failure from tomorrow 09:22:11 <gmann> main failure on attach detach API tests seems 09:22:32 <andreaf> jordanP: right on time! do you having anything to share about the gate and memory footprint? 09:22:40 <gmann> should we short list the heavy API tests like scenario tests ? 09:23:14 <jordanP> humm there's the "reduce the number of apache workers" and "enable ksm in guest" devstack patches 09:23:23 <jordanP> don't know what the status of those are 09:23:44 <jordanP> I've not be following the gate stability lately, is it as bad as usual ? 09:23:59 <gmann> one is merged - #link https://review.openstack.org/#/c/446741/ 09:24:00 <andreaf> jordanP: well it's decent but not as good as it should 09:24:10 <gmann> yea 09:24:20 <jordanP> #link reduce number of apache workers for keystone: https://review.openstack.org/#/c/445910/ 09:24:34 <gmann> ksm one is up -> #link https://review.openstack.org/#/c/447119/ 09:25:11 <jordanP> the patch that disable nova-cert in gate jobs has been merged 09:25:20 <andreaf> another thing I saw in the conversation was that apparently the py35 job is doing much better in term of memory footprint - it does not run swift which is part of it, but perhaps py35 is doing a better job in terms of memory mgmt 09:25:24 <gmann> oh we still had that? 09:25:33 <jordanP> https://review.openstack.org/#/c/446986/ 09:25:54 <gmann> jordanP: thanks 09:26:16 <jordanP> that saves only 50Mo of RAM thought, but it's still good 09:26:34 <andreaf> I would like to propose a session in the forum about how to control / monitor over time the footprint of different services in the gate 09:27:05 <jordanP> +1 09:27:10 <gmann> andreaf: +1 09:27:10 <masayukig> ++ 09:27:16 <chandankumar> that would be awesome ++ 09:27:25 <andreaf> ideas around that are welcome, if you have please add them in the etherpad :) 09:28:25 <andreaf> atm we have a lot of data but I feel perhaps not the right one so if's not obvious what'a using what and how things are changing over time 09:28:55 <andreaf> if we had historical data we could see if a service is regressing significantly in terms of memory footprint 09:29:07 <jordanP> lot of data is good, making sense of them is harder 09:29:11 <gmann> andreaf: putting those on graphite ? to monitor periodically 09:29:12 <andreaf> the issue with few data points is that there is so much variation in the test nodes that it's hard to tell 09:29:36 <andreaf> gmann: yeah graphite of somewhere we can see that data over time 09:29:50 <jordanP> well for eachand every runnning service we can ask ouselves "do we really need this" ? 09:29:57 <ltosky[m]> but with enough data it should be possible to group by node type 09:30:52 <andreaf> ltosky[m]: yeah if we have data over time we can work with it 09:31:18 <jordanP> I suggested in a ML email to kill a couple of unecessary swift services 09:31:33 <jordanP> the ones related to replication and reconcialiation 09:31:37 <andreaf> jordanP: thanks 09:31:53 <jordanP> I have less and less time to work on openstack so I won't do it myself 09:32:03 <jordanP> but I think we can save at least 100 Mo there 09:32:25 <masayukig> jordanP: :( and :) 09:32:46 <andreaf> I'd like to think we can keep better control on the status of our test nodes on a regular basis, because now we are doing a lot of effort in addressing this from many sides 09:33:08 <gmann> how about enabling services from job definition ? may be hard to configure each job 09:33:10 <andreaf> but one we are back in good shape we should have means to stay there 09:33:20 <jordanP> sure sure, would love to have this plotted in Graphite and all, but the house is on fire :) 09:34:14 <andreaf> jordanP: yeah that's why I say this could be a session for the forum for the mid term 09:34:29 <jordanP> yep, sounds goo 09:35:16 <gmann> or shrink the default enabled services ... 09:35:35 <masayukig> so, 25 mins.. 09:35:39 <gmann> anyways let's move and keep monitor failure 09:35:42 <gmann> #topic Specs Reviews 09:35:51 <gmann> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z 09:36:18 <gmann> spec state seems same so if no one has anything we will move next ? 09:36:42 <gmann> #topic Tempest 09:36:47 <gmann> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open 09:36:54 <gmann> ^^ open reviews 09:37:33 <gmann> one update was oomichi removed the cinder v1 tests 09:37:36 <chandankumar> gmann: https://review.openstack.org/370630 09:37:42 <chandankumar> review needs feedback 09:37:44 <gmann> and working on merging v2 and v3 one 09:38:08 <chandankumar> gmann: and tempest cleanup doc patch https://review.openstack.org/#/c/444041/ 09:38:23 <gmann> chandankumar: ll check but tomorrow 09:38:28 <gmann> chandankumar: thanks 09:38:54 <gmann> Bug Triage: 09:38:59 <gmann> #link https://etherpad.openstack.org/p/pike-qa-bug-triage 09:39:21 <gmann> chandankumar: had rotation this week 09:39:24 <gmann> #link https://etherpad.openstack.org/p/tempest-weekly-bug-report 09:39:26 <gmann> chandankumar: go ahead 09:39:44 <chandankumar> gmann: some small higlights 09:39:46 <chandankumar> Bugs with no reply from assignee from last 6 months are unassigned with a comment. (Mostly from wishlist) 09:39:56 <chandankumar> Added a new tag'upstream-gate' to list only upstream tempest gate failure bugs 09:40:04 <gmann> upstream-gate ? 09:40:05 <chandankumar> New Bugs: 11 -> 8 09:40:18 <gmann> you mean failure on gate with that tag? 09:40:31 <chandankumar> gmann: failure ongate with that gate 09:40:43 <gmann> k 09:41:02 <chandankumar> https://wiki.openstack.org/wiki/Bug_Tags list bug tags for different project 09:41:12 <chandankumar> tempest is missing there so i add it there by EOD 09:41:13 <gmann> chandankumar: yea assignee things you can cleanup by leaving a comment there 09:41:28 <chandankumar> gmann: cleaned already 09:41:34 <chandankumar> as discussed in the mornign 09:41:40 <chandankumar> *morning 09:41:41 <gmann> thanks 09:41:50 <gmann> chandankumar: did see tempest tag on that wiki 09:42:18 <gmann> next rotation is prateek 09:42:26 <gmann> #topic Patrole 09:42:32 <prateek> gmann, ack 09:42:35 <gmann> #link https://review.openstack.org/#/q/project:openstack/patrole 09:42:39 <gmann> prateek: thanks 09:42:49 <chandankumar> gmann: one more thing * https://bugs.launchpad.net/tempest/+bug/1672817 09:42:49 <openstack> Launchpad bug 1672817 in tempest "Mapping client is needed for v3/OS-FEDERATION/mappings API" [Undecided,New] - Assigned to Nicolas Helgeson (nhelgeson) 09:42:51 <chandankumar> * https://bugs.launchpad.net/tempest/+bug/1504641 09:42:52 <openstack> Launchpad bug 1504641 in OpenStack Compute (nova) "Listing volumes respects osapi_max_limit but does not provide a link to the next element" [Wishlist,New] 09:42:56 <gmann> felipemonteiro_: ^^ 09:43:05 <chandankumar> needs be moved to wishlist 09:43:21 <gmann> chandankumar: ok, leave comment also ll check later 09:43:24 <felipemonteiro_> we've achieved much higher gate stability, with both gates now passing, but we're still watching them carefully 09:43:26 <chandankumar> that's it from myside 09:43:39 <jordanP> good work chandankumar, thanks 09:43:54 <gmann> chandankumar: thanks. nice effort 09:44:05 <chandankumar> jordanP: gmann you are welcome :-) 09:44:11 <felipemonteiro_> idea is to keep monitoring them, fix errant bugs, and make admin gate voting (within our project), after we are confident it is stable 09:44:24 <gmann> felipemonteiro_: any plan to run patrole tests on project side ? 09:44:47 <felipemonteiro_> gmann: what do you mean by "project side"? 09:45:03 <gmann> felipemonteiro_: on project gate job like nova 09:45:04 <jordanP> I am not huge +1 gmann on this. We would need to weight the pros and cons 09:45:22 <gmann> felipemonteiro_: so that any mis changes on policy side can be detected bu patrole at same time 09:45:31 <jordanP> like this may mean a lot of more infra VMs 09:45:42 <gmann> jordanP: sure but policy testing is really critical 09:45:59 <felipemonteiro_> gmann: ultimately yes, but only once high stability is achieved. for nova, for example, i think i've seen 1 failure related to attach volume 09:46:14 <gmann> and i am 100% sure many projects miss those currently 09:46:19 <felipemonteiro_> but nova tests are otherwise among the most stable 09:46:35 <andreaf> gmann, jordanP, felipemonteiro_: I think this is something that needs to be discussed with the projects 09:46:44 <jordanP> +1 09:46:45 <gmann> felipemonteiro_: ok let's judge that later once it is stable 09:46:46 <felipemonteiro_> andreaf: agreed 09:46:55 <andreaf> if we get one or two onboard and showcase benefits we can propose for more 09:47:13 <gmann> andreaf: yea i discussed in nova api meeting yesterday and can be decide later once patrole is stable 09:47:32 <chandankumar> andreaf: gmann can we get a nightly job which run one time with all the tempest plugins with all scenario tests to detect plugin tempest issues? 09:47:36 <chandankumar> jordanP: ^^ 09:47:36 <gmann> felipemonteiro_: i updated comment on discovery policy testing. that is kind of deprecated policy from nova side so not worth to tests 09:48:06 <gmann> chandankumar: and can that finish since morning :) 09:48:06 <andreaf> chandankumar: we are talking about patrole now? 09:48:23 <chandankumar> gmann: andreaf sorry i missed it 09:48:29 <gmann> felipemonteiro_: thanks for updates. 09:48:30 <felipemonteiro_> gmann: thanks! that was my assumption, so i hadn't invested more energy in it 09:48:37 <gmann> let's move as time is short 09:48:46 <gmann> #topic DevStack 09:48:50 <gmann> anything on devstack side? 09:49:02 <andreaf> gmann: well one sec please 09:49:19 <gmann> andreaf: sure 09:50:14 <andreaf> regarding patrole I just wanted to say that if we wait too much it will never be stable enough since project may break it 09:50:27 <andreaf> so yes we might need to wait a little bit I would start and try to get non voting jobs up for some projects 09:50:44 <andreaf> regarding chandankumar question about plugins 09:50:57 <gmann> andreaf: yea for n-v +1. but putting non stable tests it might trigger false alarm on them 09:51:18 <felipemonteiro_> andreaf: we've started discussing it with keystone, we can go to nova and other projects. but we just got "high stability" a few days ago 09:51:32 <gmann> but yea i am ok for nova to start if tests results are pretty stable 09:51:33 <andreaf> felipemonteiro_: ok thanks 09:51:52 <gmann> because we are changing the policy things there and should not break existing way 09:52:03 <gmann> felipemonteiro_: cool, thanks 09:52:08 <andreaf> gmann: sorry there was a question from chandankumar re plugins as well 09:52:31 <andreaf> chandankumar: running all plugins means running all services as well, I don't think that's feasible 09:52:32 <gmann> andreaf: yea but with all plugins i am afraid it will run till morning 09:52:51 <gmann> and night job means? its day for somewhere :) 09:53:18 <andreaf> gmann, chandankumar: and we have an issue without any plugin to keep the memory footprint in place 09:53:29 <gmann> yea 09:53:49 * gmann 7 min left 09:54:07 <chandankumar> andreaf: but i am taking about a one time a day or a week scheduled job 09:54:14 <andreaf> gmann: sorry back to you 09:54:20 <andreaf> chandankumar: I don't think that will work at all 09:54:20 <gmann> anything on devstack from anyone? 09:54:31 <gmann> chandankumar: let's discuss more on qa on this. sorry 09:54:33 <andreaf> chandankumar: let's chat in QA after the meeting 09:54:37 <gmann> yea 09:54:53 <chandankumar> andreaf: gmann sure i will ping after an hour, switching to other call 09:54:55 <gmann> let's move to next then 09:54:58 <gmann> #topic Upgrade Testing 09:55:08 <gmann> Grenade 09:55:14 <gmann> Rolling upgrade 09:55:21 <gmann> andreaf: you know if any spec up for those? 09:55:49 <andreaf> gmann: no I haven't heard anything back from the osic folks 09:55:54 <gmann> sdague: was looking for those i think 09:56:12 <gmann> luzC: ^^ 09:56:37 <gmann> ok let's jump to o-h 09:56:41 <gmann> #topic OpenStack-Health, Stackviz 09:56:49 <gmann> masayukig: ^^ any updates you want to bring in 09:57:19 <masayukig> yeah, I pushed a patch to change the default range at home page of o-h 09:57:39 <masayukig> the first page loading should be quicker than before 09:57:40 <gmann> masayukig: nice and what will be dafault now 09:57:52 <masayukig> and I pushed another patch for the rss 09:58:24 <gmann> nice 09:58:24 <masayukig> It could be useful to know the failure in the gate 09:58:35 <gmann> masayukig: link? 09:58:41 <masayukig> one sec 09:58:54 <gmann> masayukig: no prob you can give later 09:59:05 <masayukig> https://review.openstack.org/#/c/444722/ 09:59:06 <masayukig> heh 09:59:09 <masayukig> #link https://review.openstack.org/#/c/444722/ 09:59:10 <gmann> masayukig: thanks 09:59:22 <gmann> anything else? 09:59:24 <masayukig> that all 09:59:27 <gmann> #topic Destructive Testing 09:59:35 <gmann> masayukig: thanks for all nice updates 09:59:35 * masayukig one min.. 10:00:15 <gmann> on destructive testing samP updated offline that he is working with different different stack holder to get the user story 10:00:22 <gmann> and will update the spec 10:00:44 <gmann> i think time up, sorry i have to close 10:00:53 <gmann> thanks all, let's jump to QA 10:00:55 <gmann> #endmeeting QA