09:00:11 <gmann> #startmeeting qa
09:00:12 <openstack> Meeting started Thu Jul  5 09:00:11 2018 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:16 <openstack> The meeting name has been set to 'qa'
09:00:26 <gmann> PING LIST- gmann, andreaf, masayukig, chandankumar, felipemonteiro
09:00:33 <gmann> who all here today?
09:00:36 <masayukig> \o/
09:01:01 * tosky always reading the logs
09:01:08 <masayukig> But I have to go in 30mins or so
09:01:49 <gmann> sure.
09:01:59 * masayukig is tackling with tempest errors for the internal cloud. That means, I'm working :)
09:02:01 <gmann> tosky: ACK :)
09:02:15 <gmann> :)
09:02:19 <gmann> let's start
09:02:26 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours
09:02:26 <viks_> gmann: Some of the errors are like:
09:02:26 <viks_> tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
09:02:26 <viks_> Details: Credentials are: {'username': None, 'project_name': None, 'project_domain_id': None, 'user_domain_id': None, 'tenant_id': None, 'user_domain_name': None, 'domain_name': None, 'tenant_name': None, 'user_id': None, 'project_id': None, 'domain_id': None, 'project_domain_name': None} Password is not defined.
09:02:26 <viks_> Some others are like:
09:02:26 <viks_> tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
09:02:26 <viks_> Details: 503
09:02:32 <gmann> agenda for today ^^
09:03:03 <gmann> viks_: let's discuss in open discussion of this meeting
09:03:32 <gmann> #topic Announcement and Action Item (Optional)
09:04:02 <masayukig> viks_: I recommend you to use http://paste.openstack.org/ or similar sites to put your log
09:04:10 <gmann> no announcement and no AI from last office hour  - #link http://eavesdrop.openstack.org/meetings/qa/2018/qa.2018-06-14-08.59.html
09:04:35 <masayukig> gmann: k
09:04:35 <gmann> #topic Rocky Priority Items progress
09:04:48 <gmann> #link https://etherpad.openstack.org/p/qa-rocky-priorities-tracking
09:04:50 <viks_> masayukig: ok
09:05:33 <gmann> i do not see much progress on either of the item.
09:06:28 <gmann> except felipe doing lot of doc work in patrole which is good direction for stable release
09:06:46 <masayukig> ++
09:07:02 <gmann> around 2 month left for Rocky cycle and i should be tracking the item more actively
09:07:47 <gmann> i will spend some time in next week and see what all item are good progress and what all can be asked for volunteer
09:07:50 <viks_> gmann: meeting? i have no idea about it
09:08:21 <gmann> #action gmann to track each planned item and send ML for more volunteer etc
09:08:46 <gmann> viks_: yeah, we have office hour of discussion for QA on every thursday
09:08:47 <masayukig> gmann: yeah, I raise my hand for the volunteer :)
09:09:18 <gmann> viks_: you can get more info on this wiki and discuss anything you want - #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
09:09:38 <gmann> viks_: but you do not need to wait for office hour, you can always ping any of us any time
09:09:58 <gmann> masayukig: thanks. i will compile the list and send during next week
09:10:14 <masayukig> gmann: great, thanks
09:10:35 <gmann> #topic OpenStack Events Updates and Planning
09:10:48 <gmann> Next: Denver PTG
09:11:08 <gmann> Draft Schedule -http://lists.openstack.org/pipermail/openstack-dev/2018-June/131881.html
09:11:30 <gmann> QA is help hour for monday and next 3 days for discussion
09:11:36 <gmann> this sounds good to me
09:11:58 <masayukig> gmann: yeah, I think it's enough
09:12:33 <gmann> yea and 1 day help hour is enough as we do not get much audience looking for help.
09:13:00 <gmann> and on thursday we can run code sprint kind of and  friday we can attend other interested sessions
09:13:32 <masayukig> heh, yeah. (We need to recruit someone to QA :-p
09:14:09 <gmann> yeah :)
09:14:39 <gmann> let's move next
09:14:48 <gmann> #topic Sub Teams highlights (Sub Teams means individual projects under QA program)
09:14:52 <gmann> Tempest
09:15:20 <gmann> 1 bug mentioned there
09:15:23 <gmann> #link https://bugs.launchpad.net/nova/+bug/1776684
09:15:23 <openstack> Launchpad bug 1776684 in tempest "MultipleCreateTestJSON.test_multiple_create intermittently fails for cells v1 due to server name check change" [High,Fix released] - Assigned to Matt Riedemann (mriedem)
09:16:04 <gmann> that is fixed by reverting the extra checks. this might have left from last week agenda
09:16:23 <gmann> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
09:16:34 <gmann> ^^ ongoing reivew
09:16:57 <gmann> i think we are doing good pace of review but let's improve it more as not much reviewer active for Tempest
09:17:32 <masayukig> sure, will do
09:17:50 <gmann> thanks. you already doing lot of review anyways :)
09:18:02 <gmann> Patrole
09:18:03 <masayukig> heh, not that much
09:18:11 <gmann> i can update on patrole side
09:18:22 <gmann> 1. Feature flag for supporting backwards-incompatible policies for n-1/n-2 testing
09:18:38 <gmann> this is done and we have a way to handle the backward incompatible policy things \
09:19:01 <gmann> 2. Initial implementation for multi-policy support:
09:19:08 <gmann> this is in-progress
09:19:34 <gmann> and other things going good in patrole is documentation.
09:19:53 <masayukig> nice
09:20:15 <gmann> from documentation i remember frickler pointed out the devstack documentation for stable branch
09:20:27 <gmann> i think patch is merged , let me check
09:20:54 <gmann> #link https://review.openstack.org/#/c/579738/
09:21:29 <gmann> need to to check that we have stable branches doc also published
09:21:50 <frickler> they were published before. just without gate checks
09:22:09 <gmann> #link https://docs.openstack.org/devstack/ocata/
09:22:23 <gmann> frickler: but i remember pike one was 404 earlier
09:22:34 <frickler> so we were kind of lucky that docs build didn't break
09:23:15 <frickler> anyway, things should be fine now
09:23:23 <gmann> yeah.
09:23:44 <gmann> anything else from subteam side to discuss
09:24:10 <masayukig> nope
09:25:06 <gmann> ok let's move next
09:25:11 <gmann> #topic Bug Triage
09:25:37 <gmann> link seems old
09:25:52 <gmann> Chandan not here today
09:26:14 <gmann> but bug is something need more attention.
09:26:16 <gmann> #link https://bugs.launchpad.net/tempest/+bugs?search=Search&field.status=New
09:26:42 <gmann> tempest has 19 New, devstack 0 patrole 3
09:26:58 <masayukig> yeah
09:27:04 <masayukig> 19 New... :(
09:27:08 <gmann> from next week, i will compile the list and we will burn that during office hour
09:27:23 <masayukig> ok
09:27:30 <masayukig> ++
09:28:34 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1758054
09:28:34 <openstack> Launchpad bug 1758054 in StackViz "stackviz failing on py3 causing tempest-full-py3 POST_FAILURES" [Undecided,In progress] - Assigned to Tim Buckley (timothy-jas-buckley)
09:29:04 <gmann> may be we can mark fix for tempest ?
09:30:01 <masayukig> yeah, I think so
09:30:30 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1766582
09:30:30 <openstack> Launchpad bug 1766582 in tempest "tempest cleanup unauthorized error" [Undecided,New]
09:30:45 <masayukig> it may be still on stackviz side for 1758054, though
09:30:46 <gmann> this seems cleanup CLI issue which i am not surprise if that is broken
09:30:59 <gmann> masayukig: thanks there is patch up for that
09:31:19 <gmann> masayukig: #link https://review.openstack.org/#/c/555388/
09:31:53 <gmann> i will try to run the cleanup CLI tomorrow and see what is the issue
09:32:25 <masayukig> gmann: thanks, I'll dig and/or take over it (hopefully :)
09:32:53 <gmann> ok
09:33:10 <tosky> they did not say how tempest.conf was configured (dynamic or static credentials)
09:34:23 <gmann> tosky: yea, it is with heat plugin seems. anyways i will run cleanup cli and ask more ifnor on bug
09:34:28 <gmann> #link https://bugs.launchpad.net/tempest/+bug/1676207
09:34:28 <openstack> Launchpad bug 1676207 in tempest "test_create_router_set_gateway_with_fixed_ip is flaky" [High,New]
09:34:59 <masayukig> it may be still on stackviz side for 1758054, though
09:35:08 <masayukig> oops, ignore me
09:35:53 <gmann> i agree with frickler and slaweq on that bug to unskip the test and monitor the bug
09:36:41 <gmann> i can push patch to unskip this unless anyone else want to do
09:37:13 * masayukig has to go ... o/
09:37:23 <gmann> masayukig: sure thanks
09:37:31 <gmann> thats all on bug for today
09:37:40 <gmann> let's move to open discussion
09:37:44 <gmann> #topic Open Discussion
09:38:13 <gmann> anything else to discuss before we move to viks_ issue
09:38:25 <mbindlish_> I have a query
09:38:31 <mbindlish_> There are multiple test cases which used skip for some condition but when they are actually skipped then there is no message/info for that
09:38:31 <mbindlish_> Example:  tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration
09:38:33 <gmann> mbindlish_: yea
09:39:01 <mbindlish_> System has 1 node,  should get skipped with message “Less than 2 compute nodes, skipping multinode tests.” But in actual no message received and during tox run all test cases outputs as pass, no data for skipped
09:39:32 <gmann> #link https://github.com/openstack/tempest/blob/634bdf206b75336f0c97f7358323d3d7678c7a6a/tempest/scenario/test_network_advanced_server_ops.py#L211
09:40:02 <mbindlish_> yes i have checked for this test
09:40:09 <gmann> mbindlish_: you can find that on log
09:40:17 <gmann> there is proper msg it dump
09:40:19 <mbindlish_> i checked tempest.log too
09:40:28 <gmann> is it?
09:40:32 <gmann> let me check it quickly
09:40:33 <mbindlish_> no it was not
09:41:19 <mbindlish_> Aslo, After Running tox
09:41:21 <mbindlish_> Ran: 1822 tests in 12.0876 sec.
09:41:21 <mbindlish_> - Passed: 1822
09:41:21 <mbindlish_> - Skipped: 0
09:41:21 <mbindlish_> - Expected Fail: 0
09:41:39 <mbindlish_> no count for skip
09:42:03 <frickler> mbindlish_: if you are looking at e.g. http://logs.openstack.org/18/473718/26/check/tempest-full/118e675/job-output.txt.gz#_2018-07-04_12_53_32_625152 it is skipped because it is marked as slow
09:43:27 <frickler> mbindlish_: do you see this in gate or in your local runs?
09:43:41 <mbindlish_> in my local run
09:44:02 <mbindlish_> @frickler,
09:44:33 <gmann> yea like this  - http://logs.openstack.org/57/580057/1/check/tempest-full/975414e/job-output.txt.gz#_2018-07-04_07_49_02_480729
09:45:11 <frickler> mbindlish_: what command are you using to run tempest?
09:45:39 <mbindlish_> for complete tests: "sudo tox"
09:45:49 <mbindlish_> for particular test: "python -m testtools.run tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration"
09:46:55 <gmann> mbindlish_:  why you do not run tempest cli or via tox env - #link https://docs.openstack.org/tempest/latest/run.html
09:47:18 <mbindlish_> using tox env: result was
09:47:19 <mbindlish_> Ran: 1822 tests in 12.0876 sec.
09:47:20 <mbindlish_> - Passed: 1822
09:47:20 <mbindlish_> - Skipped: 0
09:47:20 <mbindlish_> - Expected Fail: 0
09:48:20 <gmann> and when there are skip tests ?
09:49:03 <mbindlish_> for this test, when no. of nodes are less than or equal to 1..it will be skipped
09:49:12 <mbindlish_> my system has 1 node
09:49:20 <frickler> mbindlish_: a) not sure why you are using sudo. b) running e.g. "tox -e full" also excludes slow tests
09:49:38 <mbindlish_> @frickler, i can try by this
09:50:17 <frickler> mbindlish_: you would need "tox -e all" to also run slow tests like the one above
09:50:21 <gmann> mbindlish_:  there are different tox env and they have their own regex to exclude few tests
09:50:54 <gmann> #link https://github.com/openstack/tempest/blob/master/tox.ini
09:51:16 <mbindlish_> @gmann, ok..i can try what frickler suggested
09:51:17 <gmann> yea tox -eall will run all the tests and give you skip tests info also
09:51:25 <mbindlish_> ok thanks
09:51:37 <gmann> or tempest run --regex compute
09:51:42 <gmann> or network or cinder etc
09:52:01 <mbindlish_> okk
09:52:05 <gmann> viks_: back to you
09:52:39 <gmann> can you paste the traceback on  http://paste.openstack.org/
09:53:13 <viks_> http://paste.openstack.org/show/725086/
09:53:36 <viks_> with dynamic credentials my test results are :
09:53:40 <viks_> Ran: 463 tests in 412.0000 sec.
09:53:40 <viks_> - Passed: 137
09:53:40 <viks_> - Skipped: 285
09:53:40 <viks_> - Expected Fail: 0
09:53:40 <viks_> - Unexpected Success: 0
09:53:41 <viks_> - Failed: 41
09:53:41 <viks_> Sum of execute time for each test: 135.9822 sec.
09:55:18 <viks_> see the difference in results: http://paste.openstack.org/show/725088/
09:56:40 <viks_> i'm using swift only setup
09:56:58 <gmann> viks_: do you have admin cred in your account.yaml
09:57:10 <gmann> which are needed for admin tests if you run
09:58:10 <viks_> isn't that generated automatically by : tempest account-generator -c etc/tempest.conf -i 3 --with-admin accounts.yaml ?
09:59:06 <gmann> it should be
09:59:11 <viks_> this is what my accounts.yaml looks like: http://paste.openstack.org/show/725091/
09:59:13 <gmann> if you can paste your account.yaml
10:01:04 <viks_> refere this: >  http://paste.openstack.org/show/725091/
10:03:56 <gmann> let's close the office hour and continue the discussion
10:03:59 <gmann> #endmeeeting
10:04:23 <gmann> viks_: you should get admin tests skipped if no admin cred for account.yaml
10:04:28 <gmann> #endmeeting