09:00:11 <gmann> #startmeeting qa 09:00:12 <openstack> Meeting started Thu Jul 5 09:00:11 2018 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:00:16 <openstack> The meeting name has been set to 'qa' 09:00:26 <gmann> PING LIST- gmann, andreaf, masayukig, chandankumar, felipemonteiro 09:00:33 <gmann> who all here today? 09:00:36 <masayukig> \o/ 09:01:01 * tosky always reading the logs 09:01:08 <masayukig> But I have to go in 30mins or so 09:01:49 <gmann> sure. 09:01:59 * masayukig is tackling with tempest errors for the internal cloud. That means, I'm working :) 09:02:01 <gmann> tosky: ACK :) 09:02:15 <gmann> :) 09:02:19 <gmann> let's start 09:02:26 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours 09:02:26 <viks_> gmann: Some of the errors are like: 09:02:26 <viks_> tempest.lib.exceptions.InvalidCredentials: Invalid Credentials 09:02:26 <viks_> Details: Credentials are: {'username': None, 'project_name': None, 'project_domain_id': None, 'user_domain_id': None, 'tenant_id': None, 'user_domain_name': None, 'domain_name': None, 'tenant_name': None, 'user_id': None, 'project_id': None, 'domain_id': None, 'project_domain_name': None} Password is not defined. 09:02:26 <viks_> Some others are like: 09:02:26 <viks_> tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received 09:02:26 <viks_> Details: 503 09:02:32 <gmann> agenda for today ^^ 09:03:03 <gmann> viks_: let's discuss in open discussion of this meeting 09:03:32 <gmann> #topic Announcement and Action Item (Optional) 09:04:02 <masayukig> viks_: I recommend you to use http://paste.openstack.org/ or similar sites to put your log 09:04:10 <gmann> no announcement and no AI from last office hour - #link http://eavesdrop.openstack.org/meetings/qa/2018/qa.2018-06-14-08.59.html 09:04:35 <masayukig> gmann: k 09:04:35 <gmann> #topic Rocky Priority Items progress 09:04:48 <gmann> #link https://etherpad.openstack.org/p/qa-rocky-priorities-tracking 09:04:50 <viks_> masayukig: ok 09:05:33 <gmann> i do not see much progress on either of the item. 09:06:28 <gmann> except felipe doing lot of doc work in patrole which is good direction for stable release 09:06:46 <masayukig> ++ 09:07:02 <gmann> around 2 month left for Rocky cycle and i should be tracking the item more actively 09:07:47 <gmann> i will spend some time in next week and see what all item are good progress and what all can be asked for volunteer 09:07:50 <viks_> gmann: meeting? i have no idea about it 09:08:21 <gmann> #action gmann to track each planned item and send ML for more volunteer etc 09:08:46 <gmann> viks_: yeah, we have office hour of discussion for QA on every thursday 09:08:47 <masayukig> gmann: yeah, I raise my hand for the volunteer :) 09:09:18 <gmann> viks_: you can get more info on this wiki and discuss anything you want - #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 09:09:38 <gmann> viks_: but you do not need to wait for office hour, you can always ping any of us any time 09:09:58 <gmann> masayukig: thanks. i will compile the list and send during next week 09:10:14 <masayukig> gmann: great, thanks 09:10:35 <gmann> #topic OpenStack Events Updates and Planning 09:10:48 <gmann> Next: Denver PTG 09:11:08 <gmann> Draft Schedule -http://lists.openstack.org/pipermail/openstack-dev/2018-June/131881.html 09:11:30 <gmann> QA is help hour for monday and next 3 days for discussion 09:11:36 <gmann> this sounds good to me 09:11:58 <masayukig> gmann: yeah, I think it's enough 09:12:33 <gmann> yea and 1 day help hour is enough as we do not get much audience looking for help. 09:13:00 <gmann> and on thursday we can run code sprint kind of and friday we can attend other interested sessions 09:13:32 <masayukig> heh, yeah. (We need to recruit someone to QA :-p 09:14:09 <gmann> yeah :) 09:14:39 <gmann> let's move next 09:14:48 <gmann> #topic Sub Teams highlights (Sub Teams means individual projects under QA program) 09:14:52 <gmann> Tempest 09:15:20 <gmann> 1 bug mentioned there 09:15:23 <gmann> #link https://bugs.launchpad.net/nova/+bug/1776684 09:15:23 <openstack> Launchpad bug 1776684 in tempest "MultipleCreateTestJSON.test_multiple_create intermittently fails for cells v1 due to server name check change" [High,Fix released] - Assigned to Matt Riedemann (mriedem) 09:16:04 <gmann> that is fixed by reverting the extra checks. this might have left from last week agenda 09:16:23 <gmann> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open 09:16:34 <gmann> ^^ ongoing reivew 09:16:57 <gmann> i think we are doing good pace of review but let's improve it more as not much reviewer active for Tempest 09:17:32 <masayukig> sure, will do 09:17:50 <gmann> thanks. you already doing lot of review anyways :) 09:18:02 <gmann> Patrole 09:18:03 <masayukig> heh, not that much 09:18:11 <gmann> i can update on patrole side 09:18:22 <gmann> 1. Feature flag for supporting backwards-incompatible policies for n-1/n-2 testing 09:18:38 <gmann> this is done and we have a way to handle the backward incompatible policy things \ 09:19:01 <gmann> 2. Initial implementation for multi-policy support: 09:19:08 <gmann> this is in-progress 09:19:34 <gmann> and other things going good in patrole is documentation. 09:19:53 <masayukig> nice 09:20:15 <gmann> from documentation i remember frickler pointed out the devstack documentation for stable branch 09:20:27 <gmann> i think patch is merged , let me check 09:20:54 <gmann> #link https://review.openstack.org/#/c/579738/ 09:21:29 <gmann> need to to check that we have stable branches doc also published 09:21:50 <frickler> they were published before. just without gate checks 09:22:09 <gmann> #link https://docs.openstack.org/devstack/ocata/ 09:22:23 <gmann> frickler: but i remember pike one was 404 earlier 09:22:34 <frickler> so we were kind of lucky that docs build didn't break 09:23:15 <frickler> anyway, things should be fine now 09:23:23 <gmann> yeah. 09:23:44 <gmann> anything else from subteam side to discuss 09:24:10 <masayukig> nope 09:25:06 <gmann> ok let's move next 09:25:11 <gmann> #topic Bug Triage 09:25:37 <gmann> link seems old 09:25:52 <gmann> Chandan not here today 09:26:14 <gmann> but bug is something need more attention. 09:26:16 <gmann> #link https://bugs.launchpad.net/tempest/+bugs?search=Search&field.status=New 09:26:42 <gmann> tempest has 19 New, devstack 0 patrole 3 09:26:58 <masayukig> yeah 09:27:04 <masayukig> 19 New... :( 09:27:08 <gmann> from next week, i will compile the list and we will burn that during office hour 09:27:23 <masayukig> ok 09:27:30 <masayukig> ++ 09:28:34 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1758054 09:28:34 <openstack> Launchpad bug 1758054 in StackViz "stackviz failing on py3 causing tempest-full-py3 POST_FAILURES" [Undecided,In progress] - Assigned to Tim Buckley (timothy-jas-buckley) 09:29:04 <gmann> may be we can mark fix for tempest ? 09:30:01 <masayukig> yeah, I think so 09:30:30 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1766582 09:30:30 <openstack> Launchpad bug 1766582 in tempest "tempest cleanup unauthorized error" [Undecided,New] 09:30:45 <masayukig> it may be still on stackviz side for 1758054, though 09:30:46 <gmann> this seems cleanup CLI issue which i am not surprise if that is broken 09:30:59 <gmann> masayukig: thanks there is patch up for that 09:31:19 <gmann> masayukig: #link https://review.openstack.org/#/c/555388/ 09:31:53 <gmann> i will try to run the cleanup CLI tomorrow and see what is the issue 09:32:25 <masayukig> gmann: thanks, I'll dig and/or take over it (hopefully :) 09:32:53 <gmann> ok 09:33:10 <tosky> they did not say how tempest.conf was configured (dynamic or static credentials) 09:34:23 <gmann> tosky: yea, it is with heat plugin seems. anyways i will run cleanup cli and ask more ifnor on bug 09:34:28 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1676207 09:34:28 <openstack> Launchpad bug 1676207 in tempest "test_create_router_set_gateway_with_fixed_ip is flaky" [High,New] 09:34:59 <masayukig> it may be still on stackviz side for 1758054, though 09:35:08 <masayukig> oops, ignore me 09:35:53 <gmann> i agree with frickler and slaweq on that bug to unskip the test and monitor the bug 09:36:41 <gmann> i can push patch to unskip this unless anyone else want to do 09:37:13 * masayukig has to go ... o/ 09:37:23 <gmann> masayukig: sure thanks 09:37:31 <gmann> thats all on bug for today 09:37:40 <gmann> let's move to open discussion 09:37:44 <gmann> #topic Open Discussion 09:38:13 <gmann> anything else to discuss before we move to viks_ issue 09:38:25 <mbindlish_> I have a query 09:38:31 <mbindlish_> There are multiple test cases which used skip for some condition but when they are actually skipped then there is no message/info for that 09:38:31 <mbindlish_> Example: tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration 09:38:33 <gmann> mbindlish_: yea 09:39:01 <mbindlish_> System has 1 node, should get skipped with message “Less than 2 compute nodes, skipping multinode tests.” But in actual no message received and during tox run all test cases outputs as pass, no data for skipped 09:39:32 <gmann> #link https://github.com/openstack/tempest/blob/634bdf206b75336f0c97f7358323d3d7678c7a6a/tempest/scenario/test_network_advanced_server_ops.py#L211 09:40:02 <mbindlish_> yes i have checked for this test 09:40:09 <gmann> mbindlish_: you can find that on log 09:40:17 <gmann> there is proper msg it dump 09:40:19 <mbindlish_> i checked tempest.log too 09:40:28 <gmann> is it? 09:40:32 <gmann> let me check it quickly 09:40:33 <mbindlish_> no it was not 09:41:19 <mbindlish_> Aslo, After Running tox 09:41:21 <mbindlish_> Ran: 1822 tests in 12.0876 sec. 09:41:21 <mbindlish_> - Passed: 1822 09:41:21 <mbindlish_> - Skipped: 0 09:41:21 <mbindlish_> - Expected Fail: 0 09:41:39 <mbindlish_> no count for skip 09:42:03 <frickler> mbindlish_: if you are looking at e.g. http://logs.openstack.org/18/473718/26/check/tempest-full/118e675/job-output.txt.gz#_2018-07-04_12_53_32_625152 it is skipped because it is marked as slow 09:43:27 <frickler> mbindlish_: do you see this in gate or in your local runs? 09:43:41 <mbindlish_> in my local run 09:44:02 <mbindlish_> @frickler, 09:44:33 <gmann> yea like this - http://logs.openstack.org/57/580057/1/check/tempest-full/975414e/job-output.txt.gz#_2018-07-04_07_49_02_480729 09:45:11 <frickler> mbindlish_: what command are you using to run tempest? 09:45:39 <mbindlish_> for complete tests: "sudo tox" 09:45:49 <mbindlish_> for particular test: "python -m testtools.run tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration" 09:46:55 <gmann> mbindlish_: why you do not run tempest cli or via tox env - #link https://docs.openstack.org/tempest/latest/run.html 09:47:18 <mbindlish_> using tox env: result was 09:47:19 <mbindlish_> Ran: 1822 tests in 12.0876 sec. 09:47:20 <mbindlish_> - Passed: 1822 09:47:20 <mbindlish_> - Skipped: 0 09:47:20 <mbindlish_> - Expected Fail: 0 09:48:20 <gmann> and when there are skip tests ? 09:49:03 <mbindlish_> for this test, when no. of nodes are less than or equal to 1..it will be skipped 09:49:12 <mbindlish_> my system has 1 node 09:49:20 <frickler> mbindlish_: a) not sure why you are using sudo. b) running e.g. "tox -e full" also excludes slow tests 09:49:38 <mbindlish_> @frickler, i can try by this 09:50:17 <frickler> mbindlish_: you would need "tox -e all" to also run slow tests like the one above 09:50:21 <gmann> mbindlish_: there are different tox env and they have their own regex to exclude few tests 09:50:54 <gmann> #link https://github.com/openstack/tempest/blob/master/tox.ini 09:51:16 <mbindlish_> @gmann, ok..i can try what frickler suggested 09:51:17 <gmann> yea tox -eall will run all the tests and give you skip tests info also 09:51:25 <mbindlish_> ok thanks 09:51:37 <gmann> or tempest run --regex compute 09:51:42 <gmann> or network or cinder etc 09:52:01 <mbindlish_> okk 09:52:05 <gmann> viks_: back to you 09:52:39 <gmann> can you paste the traceback on http://paste.openstack.org/ 09:53:13 <viks_> http://paste.openstack.org/show/725086/ 09:53:36 <viks_> with dynamic credentials my test results are : 09:53:40 <viks_> Ran: 463 tests in 412.0000 sec. 09:53:40 <viks_> - Passed: 137 09:53:40 <viks_> - Skipped: 285 09:53:40 <viks_> - Expected Fail: 0 09:53:40 <viks_> - Unexpected Success: 0 09:53:41 <viks_> - Failed: 41 09:53:41 <viks_> Sum of execute time for each test: 135.9822 sec. 09:55:18 <viks_> see the difference in results: http://paste.openstack.org/show/725088/ 09:56:40 <viks_> i'm using swift only setup 09:56:58 <gmann> viks_: do you have admin cred in your account.yaml 09:57:10 <gmann> which are needed for admin tests if you run 09:58:10 <viks_> isn't that generated automatically by : tempest account-generator -c etc/tempest.conf -i 3 --with-admin accounts.yaml ? 09:59:06 <gmann> it should be 09:59:11 <viks_> this is what my accounts.yaml looks like: http://paste.openstack.org/show/725091/ 09:59:13 <gmann> if you can paste your account.yaml 10:01:04 <viks_> refere this: > http://paste.openstack.org/show/725091/ 10:03:56 <gmann> let's close the office hour and continue the discussion 10:03:59 <gmann> #endmeeeting 10:04:23 <gmann> viks_: you should get admin tests skipped if no admin cred for account.yaml 10:04:28 <gmann> #endmeeting