09:00:49 <gmann> #startmeeting qa 09:00:50 <openstack> Meeting started Thu Mar 9 09:00:49 2017 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:00:53 <openstack> The meeting name has been set to 'qa' 09:01:04 <gmann> hi, who all here today ? 09:01:06 <samP> hi o/ 09:01:13 <gmann> samP: hi. 09:01:14 <samP> This me samapth from NTT 09:01:22 <blancos> o/ 09:01:22 <zhufl> hello 09:01:28 <gmann> samP: nice to see you 09:01:40 <samP> gmann: hi.. 09:02:00 <chandankumar> gmann: toscalix samP zhufl \o/ 09:02:15 <gmann> hi everyone 09:02:31 <gmann> andreaf: masayukig jordanP ? 09:02:43 <tosky> hi 09:02:44 <gmann> masayukig is not around today 09:02:57 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_March_9th_2017_.280900_UTC.29 09:03:03 <gmann> ^^ today agenda 09:03:24 * masayukig is here but read-only-mode... 09:03:52 <andreaf> o/ 09:03:57 <gmann> we have Pike priority here - #link https://etherpad.openstack.org/p/pike-qa-priorities 09:04:17 <gmann> and most of them with assignee also. 09:04:31 <gmann> #topic Previous Meeting Action review 09:04:44 <gmann> let's checkout the action items 09:04:56 <gmann> 1. andreaf email ML with our plan on scenario tests 09:05:03 <gmann> i think andreaf did that 09:05:18 <gmann> 2. andreaf setup a scenario only job for tempest 09:05:20 <andreaf> gmann yeah just looking for a link... 09:06:11 <gmann> #link http://lists.openstack.org/pipermail/openstack-dev/2017-March/113113.html 09:06:19 <andreaf> thanks... 09:06:22 <andreaf> gmann: so for 2. 09:06:26 <andreaf> it's WIP 09:06:41 <gmann> 2. andreaf setup a scenario only job for tempest 09:06:50 <gmann> andreaf: thanks, link ? 09:06:55 <andreaf> gmann: I have a few patches in Tempest, d-g and project-config #link https://review.openstack.org/#/c/442565/ 09:07:29 <andreaf> #link https://review.openstack.org/#/c/441980/ Tempest one is merged 09:07:52 <gmann> andreaf: local.conf things from job directly not stable yet ? 09:08:04 <gmann> i was thinking we can avoid d-g flags 09:08:16 <andreaf> gmann: and project-config one is my local branch, haven't posted it yet 09:08:37 <gmann> andreaf: ok, thanks for working on those. 09:08:38 <andreaf> gmann: yes local conf works for tempest settings and I used that for the project-config bit 09:09:00 <andreaf> gmann: but which tox env to use is not under local conf control 09:09:11 <gmann> ah yea 09:09:31 <gmann> ll check all series tomorrow 09:09:33 <andreaf> gmann: I don't like new flags either, but the alternative is to not use a dedicated tox env, which I though it was worse 09:09:39 <andreaf> gmann: thanks 09:09:43 <gmann> and currently we have 11 scenario tests running on normal jobs 09:10:20 <gmann> andreaf: yea tox env use is pretty nice 09:10:33 <gmann> 3. * andreaf check with luzC about grenade specs 09:10:38 <gmann> andreaf: again for you :) 09:11:10 <andreaf> gmann: yeah luzC mentioned she was going to withdraw those specs 09:11:19 <andreaf> and start new ones 09:11:55 <gmann> withdraw old one ? 09:12:31 <gmann> or its all new one what was discussed in PTG like rolling upgrade support etc 09:13:29 <andreaf> gmann: yeah so they are not grenade specs anymore 09:13:45 <andreaf> gmann: rolling upgrade and so will be done by other means 09:14:13 <gmann> in different project or way ? 09:15:26 <gmann> anyways let's get more clarity once we have spec up 09:15:51 <gmann> next AI- 4. oomichi to look at API versions for test jobs 09:15:55 <gmann> #link https://etherpad.openstack.org/p/tempest-api-versions-in-pike 09:16:30 <gmann> i think this will be very nice step if we can get rid of deprecated and unsupported version testing from tempest 09:16:49 <andreaf> yeah there are a few patches up already oomichi has been driving this forward 09:17:09 <gmann> it will make lot of load down on maintenance as well as on gate 09:17:18 <andreaf> yeah 09:17:31 <gmann> yea, oomichi will finish those fast i know :) 09:17:48 <gmann> cool 09:17:52 <andreaf> only one note on my side - config is a stable interface, and so are default values 09:17:52 <gmann> 5. everyone - review the ethercalc @ 09:18:17 <andreaf> changing default values for a config is a backward incompatible chnage, so we need to give notice for that :) 09:18:28 <gmann> andreaf: default values also yes i think 09:18:41 <gmann> andreaf: i think i gave +1 or +2 on that 09:18:59 <gmann> but agree we should not change default value 09:19:24 <gmann> not sure deprecation process on default values change 09:19:45 <gmann> we can do just in help ? 09:19:58 <andreaf> gmann: yeah I guess so 09:20:40 <gmann> andreaf: ok. 09:21:02 <gmann> next AI: ethercalc 09:21:23 <gmann> its all done and scenario tests are sorted out 09:21:33 <gmann> 6. sdague to look into cinder v1 admin tests 09:21:49 <gmann> andreaf: we can remove even admin tests for v1 right 09:23:19 <andreaf> heh I have to check I think sdague mentioned that it was actually ok as it is, but not 100% sure 09:23:47 <gmann> humm. i want to remove api version selection config options we have 09:24:04 <gmann> anyways let's check sdague on that 09:24:06 <gmann> #topic Gate Status 09:24:34 <gmann> so after cutting off scenario tests, gate kind of stable seems.. 09:25:23 <gmann> but i am suspecting some ssh failure still even from API or scenario tests :) 09:25:37 <andreaf> gmann: the failures on the main tempest job are still too high: #link https://goo.gl/Ms7OVZ 09:26:18 <andreaf> we are still hitting libvirt crashes a lot #link http://status.openstack.org/elastic-recheck/ 09:26:18 <gmann> andreaf: oh, i think volume attach tests may be? 09:26:52 <andreaf> gmann: can you look into those? 09:27:13 <gmann> andreaf: sure, but i can do tomorrow only 09:27:43 <andreaf> I wanted to check if the libvirt crashes happen on API or scenario - I have the feeling it will be on API side, because API tests still generate good load on the system 09:28:01 <gmann> #action gmann to check libvirt crashes issues on tempest job 09:28:10 <andreaf> on the other side if it's on scenario we might be able then to isolate the issue, get a reproducer and ask libvirt folks for a fix 09:28:12 <gmann> andreaf: yea i suspect attach detach tests 09:28:37 <gmann> nice 09:28:46 <andreaf> anyways, there is still work to be done on the gate, so that stays the highest prio for me 09:29:05 <gmann> +1 09:29:13 <gmann> gate stability is main goal 09:29:21 <andreaf> in case the issue is load related, one way forward could be to split API and scenario in two jobs and reduce concurrency on the API side 09:29:35 <gmann> openstack developement should go smoothly 09:30:03 <andreaf> another approach would be to isolate "heavy" API tests and either split them up or make them scenarios 09:30:17 <gmann> andreaf: we can list heavy API tests and then figure out those to be serial or parallel 09:30:21 <andreaf> so I will keep looking at both options 09:30:45 <andreaf> gmann: yeah I started looking at subunit-describe-calls to automate that 09:31:01 <gmann> i think splitting the heavy one either APi or scenario is nice 09:31:15 <gmann> andreaf: nice. thanks that will help 09:31:22 <zhufl> +1 09:31:41 <chandankumar> gmann: one query how to determine a test is heavy or light based on time or resources? 09:31:42 <zhufl> api tests should be simple enough 09:31:47 <gmann> let's keep eyes on gate things 09:32:01 <andreaf> chandankumar: heavy from resources pov 09:32:08 <gmann> chandankumar: yea time taken and what all kind of operation it does 09:32:16 <andreaf> chandankumar: so how many VMs. volumes, networks are created by a test 09:32:23 <gmann> yea 09:32:41 <chandankumar> andreaf: gmann got that, Thanks :-) 09:32:56 <zhufl> and some scenario testcases can be removed from Tempest maybe 09:32:58 <gmann> andreaf: ll check some of them and prepare some list 09:33:10 <andreaf> gmann: ok thank you! 09:33:22 <gmann> zhufl: not sure about that. but we can go case by case 09:33:22 <zhufl> e.g, to suspend-resume-suspend-resume 09:33:42 <gmann> honestly i do not like to remove tests from tempest :) 09:34:01 <gmann> zhufl: we can figure that out where to run instead of deleting 09:34:03 <andreaf> zhufl: yeah some may not belong to tempest at all, but I guess that's a separate discussion for now 09:34:13 <gmann> but yea if those are covered on project side then yes 09:34:21 <gmann> let's move 09:34:24 <gmann> #topic Pike Goals 09:34:32 <gmann> 1. Python 3.5 https://governance.openstack.org/tc/goals/pike/python35.html 09:34:40 <gmann> i think this is completed 09:34:49 <andreaf> heh I was hoping to have some work done on these, but I have not had the time yet 09:35:04 <gmann> andreaf: on 3.5? 09:35:08 <andreaf> gmann: so the process is that I need to put up specs for each goal 09:35:25 <andreaf> gmann: and add the specs to the governance repo in the overall goal 09:35:28 <gmann> andreaf: and update patch on goverance 09:35:37 <gmann> ok 09:35:44 <chandankumar> gmann: i can check and confirm by runnign tempest on py3.5 09:35:47 <chandankumar> ? 09:35:53 <andreaf> gmann: and in each spec we need to say what needs to be done for python 3 and wsgi respectively 09:36:12 <andreaf> gmann: and this is not for tempest only but for every project under the QA umbreall 09:36:22 <gmann> andreaf: or just update status on that like - #link https://review.openstack.org/#/c/440832/ 09:36:45 <gmann> andreaf: yea, all projects 09:36:59 <gmann> blancos: how about patrole ? 09:37:12 <andreaf> so I guess there won't be much real work to be done but I just need to go through the list and make sure everything is covered 09:37:15 <gmann> do we have tests running on 3.5 job there 09:37:21 <andreaf> and what is not should go in the high prio list 09:37:25 <gmann> andreaf: yea 09:37:43 <blancos> gmann I'd have to doublecheck but I think we have a py35 gate 09:37:47 <andreaf> so I was hoping to have the list for today but I didn't manage yet, so it will be next week 09:38:11 <gmann> #action andreaf to add sepcs on Pike Goal, py3.5 and wsgi for all QA projects 09:38:20 <gmann> andreaf: ^^ :) 09:38:47 <gmann> blancos: cool, please update andreaf after checking on that. 09:39:04 <blancos> andreaf gmann I just looked, we do :) 09:39:42 <gmann> blancos: THIS? #LINK http://logs.openstack.org/17/443417/1/check/gate-patrole-python35/adfe79b/ 09:39:58 <gmann> blancos: i think that runs just unit tests on 3.5 not patrole on 3.5 09:40:15 <andreaf> so integration tests must pass when a service runs on py3.5 but it the goal does not say anything about running integration tests on py3.5 09:40:31 <gmann> andreaf: is it? 09:40:43 <andreaf> but I would like to have that as a soft goal as well, not high prio like the main goal perhaps but it would be nice 09:41:03 <blancos> gmann Oh I'm sorry you're right. We're working on CI gates right now, I'll check with my colleague who's working on that 09:41:17 <gmann> i thought tests module too should be running on 3.5 09:41:27 <gmann> ll check anyways 09:41:32 <andreaf> blancos, gmann: ok I'll prepare a spec / etherpad and we can all comment on it then 09:41:53 <gmann> andreaf: cool, thanks, i already added AI for you 09:41:55 <gmann> anyways lets speed up 09:41:59 <andreaf> gmann: yeah unit / functional tests 09:42:00 <andreaf> but we don't have functional tests 09:42:01 <gmann> #topic Specs Reviews 09:42:20 <gmann> #link https://review.openstack.org/#/q/status:open+project:openstack/qa-specs,n,z 09:42:28 <gmann> andreaf: i see 09:43:14 <gmann> i think samP created new one/restored HA one # link https://review.openstack.org/#/c/443504/ 09:43:26 <samP> gmann: yes 09:43:28 <gmann> let's talk on that separately 09:43:35 <gmann> anything else on specs? 09:43:40 <samP> gmann: sure 09:43:52 <andreaf> ok thanks I need to review that one :) 09:44:10 <andreaf> samP: is it WIP or ready for review? 09:44:12 <gmann> #topic Tempest 09:44:18 <gmann> - Warnings for internal interfaces? (andreaf) 09:44:23 <samP> adiantum: WIP for now, 09:44:35 <andreaf> samP: ok thanks 09:44:41 <samP> andreaf: sorry, WIP for now, I will update my todo items 09:44:45 <gmann> andreaf: you mean warning on each internal interface but we do not know how many in use 09:45:05 <andreaf> gmann: yeah I wanted to chat about if / how we advertise tempest internal interfaces 09:45:30 <andreaf> so something like raising a warning if an internal module is imported by a project that is not tempest 09:45:48 <gmann> andreaf: so doc on stable interface as list should be enough? and everything else is internal 09:46:04 <andreaf> I'm not sure how to implement that in way that is not too invasive and verbose, and I'm not sure if it's a good idea 09:46:05 <gmann> i know we do that currently and people still use internal variable too :) 09:46:32 <tosky> is there a way to identify them with some code analysis? It would be easier to identify the top-requested functions before starting to see warnings all over 09:46:43 <gmann> andreaf: ok, that will be good but on import warning it will be tempest too 09:46:53 <andreaf> gmann: well docs are ok I guess, but people does not necessarily read docs - which is normal 09:47:10 <tosky> gmann: can we defer the warning after identifing the required missing interfaces? 09:47:22 <gmann> tosky: we can grep on codesearch but how many, 09:47:48 <andreaf> gmann: yeah I have some scripts to find out imports 09:47:51 <gmann> tosky: yea thats main goal, to have more and more required interface as stable 09:48:13 <gmann> andreaf: nice. but list might be huge right 09:48:23 <tosky> yes, but starting with warnings before having alternatives may be... counterproductive 09:48:33 <andreaf> gmann: #link https://github.com/andreafrittoli/tempest_stable_interfaces/blob/master/data/get_deps.sh 09:48:47 <gmann> tosky: +1 i am on that side always :) but let's see how many we can provide 09:49:02 <gmann> tosky: but thing is it takes time to give complete interface as stable 09:49:21 <gmann> andreaf: cool, thanks 09:49:36 <andreaf> tosky: well apart from credential providers and perhaps test.py I think the alternative is to write the test in the plugins... 09:49:41 * gmann 12 min left 09:49:53 <gmann> next - #link https://review.openstack.org/#/q/project:openstack/tempest+status:open 09:50:08 <andreaf> tosky: well remote client perhaps as well, but I don't think we have much more high value things to move to lib 09:50:09 <gmann> ^^ are open review on tempest keep doing the good work 09:50:23 <gmann> Bug Triage: 09:50:37 <gmann> #link https://etherpad.openstack.org/p/tempest-weekly-bug-report 09:50:43 <prateek> https://review.openstack.org/#/c/443051/, can someone please review this too ? 09:50:46 <gmann> seems like nobody did last week 09:51:01 <tosky> andreaf: "get me the proper network used for the test, either created with the dynamic project or preconfigured" 09:51:03 <gmann> but we have report from luzC on previous week 09:51:10 <andreaf> gmann: lucZ sent an email about bug triage 09:51:17 <tosky> andreaf: that's required for any serious scenario tests 09:51:38 <gmann> andreaf: yea, i added in report #link https://etherpad.openstack.org/p/tempest-weekly-bug-report 09:51:43 <gmann> we have 7 new bugs 09:52:08 <gmann> jwhite is next week. ll ping him if i can reach to him 09:52:20 <gmann> anything on tempest side ? 09:52:48 <tosky> a potential issue with external plugin configuration, but we can handle it on -qa later 09:52:53 <gmann> if nothing on DevStack grenade, o-h we can skip as time is less 09:53:00 <tosky> (well, not potential) 09:53:20 <gmann> tosky: yea, do we have bug on that, saw conversation on QA channel but not fully 09:53:34 <chandankumar> gmann: do we really need this while building doc https://github.com/openstack/tempest/blob/master/doc/source/conf.py#L18 ? 09:53:53 <chandankumar> to generate a list of plugins 09:54:08 <chandankumar> or i move it to a seperate tox section? 09:54:36 <gmann> chandankumar: need to check that is for listing all plugin on sample right 09:54:39 <gmann> #topic Patrole 09:54:47 <chandankumar> gmann: ok 09:54:51 <gmann> blancos: your turn, anything you want to bring in 09:55:13 <gmann> blancos: not sure about admin things on tests which i need to check the framework first 09:55:22 <blancos> I had 3 questions related to Patrole, but I can save them for -qa since we're short on time 09:55:43 <gmann> blancos: go ahead we can see if out of time 09:56:32 <blancos> Okay. 1. I understand that at the PTG it was decided base classes would become stable interfaces. I was wondering if there was any talk about doing that for waiters as well 09:56:56 <gmann> blancos: waiter too. i will check that 09:57:08 <blancos> 2. A couple were asking at a meeting yesterday if we had an IRC room. We don't 09:57:19 <blancos> Would we use QA's or get our own? 09:57:21 <gmann> i reemember we had patch for waiter but need to check where that went 09:57:57 <gmann> blancos: we cover in QA meeting but if you feel you need more time then its all ncie to have sub meeting too 09:58:00 <gmann> we do in nova 09:58:04 <gmann> andreaf: ^^ what u say 09:58:23 <gmann> blancos: keep your 3rd question on QA :) sorry 09:58:26 <gmann> #topic Destructive Testing 09:58:40 <samP> I hv just proposed a new spec (443504) in favor of Timur’s spec. 09:58:44 <gmann> so destructive testing is being started by samP 09:58:51 <gmann> samP: big thanks on that 09:58:54 <andreaf> gmann: well in terms of IRC I would rather keep one room 09:58:57 <gmann> samP: cool 09:58:59 <samP> gmann: np 09:59:04 <samP> I will address all the comments to Timur’s spec in my new spec. 09:59:15 <andreaf> gmann: in terms of meeting, if there is a need for a submeeting (enough to discuss there) I'm fine with that 09:59:22 <gmann> samP: in barcelona we mainly wanted user story on spec side 09:59:39 <gmann> samP: not sure you have in spec, if not can you put some of them 09:59:52 <samP> gmann: I will add them 09:59:54 <gmann> samP: it will be easy to visualize 09:59:57 <gmann> samP: thanks 10:00:12 <gmann> andreaf: ok 10:00:37 <samP> gmann: I will update this spec and clear my todos soon 10:00:38 <gmann> let's move to QA 10:00:40 <andreaf> gmann: but I would keep a patrole section in the main meeting for a quick update 10:00:45 <gmann> thanks all for joining 10:00:49 <gmann> #endmeeting