| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm 260586,4,3468b2a | 00:00 |
|---|---|---|
| *** rbak_ has quit IRC | 00:00 | |
| *** dimtruck is now known as zz_dimtruck | 00:04 | |
| *** tochi has quit IRC | 00:05 | |
| *** tochi_ has joined #openstack-qa | 00:05 | |
| *** sabeen3 has quit IRC | 00:09 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full 260588,4,2cc49f3 | 00:13 |
| *** jaypipes has quit IRC | 00:19 | |
| *** dims has joined #openstack-qa | 00:22 | |
| *** salv-orlando has quit IRC | 00:25 | |
| *** salv-orlando has joined #openstack-qa | 00:25 | |
| *** jasonsb has joined #openstack-qa | 00:27 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm-multinode 262074,9,55e482c | 00:27 |
| *** mriedem has quit IRC | 00:31 | |
| *** openstackgerrit has quit IRC | 00:32 | |
| *** openstackgerrit has joined #openstack-qa | 00:32 | |
| *** mriedem has joined #openstack-qa | 00:36 | |
| *** oomichi has joined #openstack-qa | 00:38 | |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-full-kilo 264354,5,e7b22b3 | 00:40 |
| *** tosky has quit IRC | 00:47 | |
| *** Swami_ has quit IRC | 00:49 | |
| *** zz_dimtruck is now known as dimtruck | 00:49 | |
| openstackrecheck | Console logs not available after 13:23s for gate-glance-python27 258891,3,5cf1857 | 00:54 |
| *** zhangjn has joined #openstack-qa | 00:54 | |
| *** zhangjn has quit IRC | 00:54 | |
| *** zhangjn has joined #openstack-qa | 00:55 | |
| *** woodster_ has quit IRC | 00:56 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048 https://review.openstack.org/265028 | 00:57 |
| openstack | bug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/1532048 | 00:57 |
| tonyb | Hmm it looks like greande is broken in liberty due to django-compressor recent release | 00:57 |
| *** pvaneck has quit IRC | 00:58 | |
| oomichi | tonyb: mriedem is fixing it on https://review.openstack.org/#/c/265026/ | 01:03 |
| mriedem | https://review.openstack.org/#/c/265025/ now | 01:04 |
| mriedem | oops | 01:04 |
| mriedem | yeah https://review.openstack.org/#/c/265025/ | 01:05 |
| mriedem | jlk: beat me by 12 minutes | 01:05 |
| oomichi | mtreinish: masayukig: can you take a look at https://review.openstack.org/#/c/263986/ ? | 01:05 |
| mtreinish | oomichi: no tests? | 01:06 |
| oomichi | mtreinish: yep, we will do it on tempest-lib side later. | 01:06 |
| *** zhangjn has quit IRC | 01:06 | |
| tonyb | Huh there you go. I shoudl have read scroll back! | 01:06 |
| oomichi | mtreinish: just migrate from tempest to -lib | 01:06 |
| mtreinish | ok | 01:07 |
| *** jacky-zhang has joined #openstack-qa | 01:07 | |
| *** hoangcx has joined #openstack-qa | 01:07 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full 264397,2,c1e0299 | 01:07 |
| oomichi | mtreinish: the release of tempest-lib(0.13.0) failed due to pypi site error(http 503) during the uploading. | 01:08 |
| oomichi | mtreinish: I will release a new version after merging it. | 01:08 |
| mtreinish | oomichi: bring it up on infra | 01:08 |
| mtreinish | they can resubmit the job for us | 01:08 |
| oomichi | mtreinish: I see. maybe the version number(13) and the timing(Friday) was wrong ;) | 01:09 |
| *** Swami has joined #openstack-qa | 01:11 | |
| gmann | mtreinish: 1 quick question | 01:11 |
| mtreinish | oomichi: so we have a 0.13.0 tag in the repo, but there isn't anything on pypi | 01:11 |
| gmann | mtreinish: if tempest is installed in venv then it cannot detect plugin right? | 01:11 |
| mtreinish | we definitely need to resubmit that job | 01:11 |
| *** hemna is now known as hemnafk | 01:12 | |
| mtreinish | gmann: either the plugin must be installed installed in the venv too, or system site-packages is enabled for the venv | 01:12 |
| gmann | mtreinish: tox all-plugin env make site-package true so that is only way to run plugin tests in case of tempest in venv | 01:12 |
| gmann | mtreinish: yea | 01:12 |
| mtreinish | gmann: yes, if you install a plugin in the system site-packages and use tox -e all-plugin it'll see the plugins | 01:12 |
| gmann | mtreinish: so having tempest-lib requirement in project (where plugin is there) requirement.txt does not make sense right? | 01:13 |
| mtreinish | gmann: I'm not sure I follow the question | 01:13 |
| gmann | mtreinish: i mean project having plugin, thought of adding tempest-lib in their requirement.txt | 01:13 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048 https://review.openstack.org/265028 | 01:14 |
| openstack | bug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/1532048 | 01:14 |
| gmann | mtreinish: that was because we thought that is needed if tempest is installed in venv and they want to run plugin test in that case | 01:14 |
| gmann | so they need to installed tempest-lib by themself as tempest will install the same in venv | 01:14 |
| mtreinish | gmann: if they actually require tempest-lib for run time of the project it should be in requirements.txt you can't rely on tempest-lib being installed | 01:14 |
| mtreinish | gmann: there isn't a secondary dependency in this case | 01:15 |
| *** zhangjn has joined #openstack-qa | 01:15 | |
| mtreinish | although realistically I can't see the need for it in requirements.txt. It lkikely should be in test-requirements.txt | 01:15 |
| mtreinish | for the project with a plugin | 01:15 |
| gmann | mtreinish: they just need in plugin tests which will be run from wither without venv or with tox all-plugin where site-package enabled | 01:16 |
| mtreinish | or use extras to show the dependency | 01:16 |
| gmann | mtreinish: yea in test_reuirement.txt make sense if they run tests from their repo | 01:16 |
| mtreinish | gmann: it's not just about that, it's also import. You want requirements to list everything thats pip installable in use by the code in the project | 01:17 |
| mtreinish | you shouldn't try to guess what the runtime environment provides and not add things to the list | 01:17 |
| *** mtanino has quit IRC | 01:18 | |
| gmann | mtreinish: but they do not need lib in code just in plugin tests | 01:18 |
| mtreinish | yes | 01:18 |
| gmann | mtreinish: if they do in code then yes, they should not assume and install all dependency by then self | 01:18 |
| gmann | mtreinish: another thing- is there any way we can detect plugin also in venv ? | 01:19 |
| mtreinish | gmann: if they have anything that is importing from tempest-lib in their repo it has to be in a requirements file | 01:19 |
| gmann | mtreinish: i mean plugin namespace we can detect in venv also | 01:19 |
| mtreinish | gmann: https://review.openstack.org/#/c/251542/ | 01:19 |
| gmann | mtreinish: even if that import is in tests only? | 01:19 |
| mtreinish | gmann: yes, put it in test requirements or add an extras 'tempest' target | 01:20 |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-npm-run-lint 259231,10,5f437eb | 01:20 |
| gmann | mtreinish: yea | 01:21 |
| gmann | mtreinish: with this can we detect plugin if running CLI in venv ? -https://review.openstack.org/#/c/251542/ | 01:21 |
| *** openstackrecheck has quit IRC | 01:22 | |
| gmann | mtreinish: i ran tempest in venv and it does not detect as plugin namespace was on top level | 01:22 |
| mtreinish | gmann: yes, if tempest is installed in the venv and you call that it'll only show which plugins are available | 01:22 |
| *** openstackrecheck has joined #openstack-qa | 01:22 | |
| *** ccneill has quit IRC | 01:22 | |
| mtreinish | it's making the same stevedore call as the plugin interface to get the list | 01:23 |
| gmann | mtreinish: ahh, i see | 01:23 |
| gmann | mtreinish: and so tempest run CLI will also do the same. | 01:24 |
| mtreinish | right | 01:24 |
| gmann | mtreinish: but still in doubt that stevedore call will detect namespace 'tempest.test_plugins' which is at top level | 01:26 |
| mtreinish | gmann: give it a try | 01:26 |
| gmann | mtreinish: because when i run tempest in venv it try to detect the same through test discovery and does not find any plugin | 01:26 |
| gmann | mtreinish: ok | 01:26 |
| mtreinish | gmann: the cli will show you which plugins are available to tempest as it's installed | 01:27 |
| mtreinish | if you have it venv isolated it won't show you system plugins (unless system site-packages are enabled for the venv) | 01:27 |
| gmann | mtreinish: yea, that was my doubt. So in this case too we have to make site-packages enabled | 01:28 |
| mtreinish | gmann: or install the plugins into the venv | 01:29 |
| mtreinish | that's not going to change | 01:29 |
| gmann | mtreinish: yea | 01:29 |
| gmann | mtreinish: how about enabling site-package for CLI during init | 01:30 |
| gmann | not sure if we already does that | 01:30 |
| mtreinish | gmann: that's not what the utility is for | 01:30 |
| mtreinish | it's to show you which plugins are available to tempest as installed | 01:30 |
| mtreinish | if you venv isolate tempest and no plugins are in the venv it won't show you anything | 01:31 |
| mtreinish | because tempest won't see any plugins | 01:31 |
| gmann | mtreinish: yea, and its upto users how they configure env and wantto run tempest | 01:31 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/tempest: Use floating_ips_client from tempest-lib https://review.openstack.org/263109 | 01:31 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/tempest: Update global-requirement for tempest-lib https://review.openstack.org/263108 | 01:31 |
| gmann | mtreinish: make sense now | 01:32 |
| gmann | mtreinish: Thanks | 01:33 |
| mtreinish | sure, np | 01:33 |
| *** Swami has quit IRC | 01:47 | |
| *** zhangjn has quit IRC | 01:48 | |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-docs 264942,2,e6b7601 | 01:52 |
| *** bochi-michael has joined #openstack-qa | 01:52 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 147516,59,1e9e13f | 02:05 |
| *** dpaterson has joined #openstack-qa | 02:10 | |
| dpaterson | Anyone else seeing devstack failures like this? Three voting jobs failing at same point: http://pastebin.com/4aEfhp0d | 02:11 |
| oomichi | dpaterson: https://review.openstack.org/#/c/265025/ is fixing it | 02:18 |
| dpaterson | oomichi; Thanks! | 02:19 |
| openstackrecheck | Console logs not available after 13:25s for gate-grenade-dsvm 265017,1,46e81df | 02:19 |
| *** zhangjn has joined #openstack-qa | 02:23 | |
| lifeless | jlvillal: not input, 'terminal' | 02:23 |
| lifeless | jlvillal: e.g. password prompts | 02:23 |
| lifeless | jlvillal: reading from stdin doesn't stop things with a signal | 02:23 |
| jlvillal | lifeless: Okay. I'm not quite sure it is happening though. | 02:24 |
| jlvillal | lifeless: When I set things up for a non-grenade type job, no issues | 02:24 |
| jlvillal | But when I switched to a grenade type job, I now get this hang occurring | 02:24 |
| lifeless | jlvillal: http://www.gnu.org/software/libc/manual/html_node/Job-Control-Signals.html | 02:24 |
| lifeless | jlvillal: hmm, perhaps it may be a simple stdin read - anyhow, its the combination: read + being in background | 02:25 |
| jlvillal | So I am working on setting up gdb in the VM. Unfortunately the apt-dbgsyms package doesn't seem to be up to date with the version of apt I have installed. | 02:25 |
| lifeless | if you straced it, I imagine you'd have caught whatever prompt its sending first | 02:26 |
| lifeless | in the trace | 02:26 |
| jlvillal | When I did the strace all I got was that it was stopped by the signal. Nothing else. | 02:27 |
| jlvillal | But yes, my theory is something is asking for some input and then it gets hung. | 02:28 |
| openstackrecheck | Console logs not available after 13:25s for gate-swift-pep8 265015,1,20a51f8 | 02:32 |
| *** dims has quit IRC | 02:36 | |
| lifeless | do your proxies need passwords? | 02:39 |
| *** shuquan has joined #openstack-qa | 02:39 | |
| *** sdague has quit IRC | 02:39 | |
| *** zhangjn_ has joined #openstack-qa | 02:39 | |
| *** sdague has joined #openstack-qa | 02:40 | |
| openstackgerrit | FelixLi proposed openstack/tempest: Splitting clients into client groups in manager.py https://review.openstack.org/254160 | 02:40 |
| *** zhangjn has quit IRC | 02:42 | |
| jlvillal | lifeless: No | 02:43 |
| jlvillal | lifeless: My next step will be to put the big long list of packages into my ansible setup script. Hopefully then the apt-get done by stack.sh won't hang. | 02:43 |
| *** jasonsb has quit IRC | 02:44 | |
| jlvillal | As I don't want to waste too much time trying to figure this out as it has side-tracked me for too long already | 02:44 |
| *** jasonsb has joined #openstack-qa | 02:44 | |
| jlvillal | lifeless: Thanks for your help! | 02:45 |
| * jlvillal is going to to head home... | 02:46 | |
| *** Deng has joined #openstack-qa | 02:47 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full 260587,4,b9ee937 | 02:47 |
| *** zhangjn has joined #openstack-qa | 02:53 | |
| *** shuquan has quit IRC | 02:53 | |
| Deng | Could anyone please help me to check a tempest case execution issue? | 02:55 |
| Deng | when api/computer/admin/test_quotas.py executed, no matter admin user or _member_ user, failure "Pollicy doesn't allow computer: create to be performaed." | 02:56 |
| Deng | thanks so much | 02:56 |
| *** zhangjn_ has quit IRC | 02:56 | |
| *** shuquan has joined #openstack-qa | 03:01 | |
| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm 256180,1,d7b4265 | 03:01 |
| *** shuquan has quit IRC | 03:01 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Separate base microversion client and compute client https://review.openstack.org/265051 | 03:03 |
| *** shuquan has joined #openstack-qa | 03:04 | |
| *** yamahata has quit IRC | 03:04 | |
| oomichi | lifeless: hi, can you take a look at https://review.openstack.org/#/c/263106/ ? | 03:04 |
| tonyb | lifeless: Did you have anything to do with: 264398 it has you as the author and the commit message reads like it's from you but you don't show up as the gerrit owner/uploader | 03:04 |
| oomichi | Deng: does that happen on the gate or on your local env? | 03:04 |
| tonyb | lifeless: https://review.openstack.org/#/c/264398 for clicky goodness | 03:04 |
| tonyb | lifeless: Oh NM I see it's been cherry-picked from your openstack-infra/infra-manual change | 03:07 |
| *** shuquan has quit IRC | 03:08 | |
| *** eil397 has joined #openstack-qa | 03:09 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: WIP: Attempt to work around hanging issue with apt-get https://review.openstack.org/265053 | 03:11 |
| *** shuquan has joined #openstack-qa | 03:13 | |
| openstackrecheck | Console logs not available after 13:24s for gate-grenade-dsvm 224910,8,caa00b4 | 03:15 |
| *** coolsvap|away is now known as coolsvap | 03:15 | |
| gmann | mtreinish: andreaf sdague - Could you review this (non-admin job fix)- https://review.openstack.org/#/c/263976/ | 03:16 |
| *** zhangjn_ has joined #openstack-qa | 03:17 | |
| *** zhangjn has quit IRC | 03:17 | |
| *** zhangjn_ has quit IRC | 03:17 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: WIP: Attempt to work around hanging issue with apt-get https://review.openstack.org/265053 | 03:18 |
| *** zhangjn has joined #openstack-qa | 03:18 | |
| *** hop has joined #openstack-qa | 03:18 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: WIP: Attempt to work around hanging issue with apt-get https://review.openstack.org/265053 | 03:23 |
| lifeless | oomichi: I will Monday; EOD now, and weekend | 03:27 |
| *** dpaterson has quit IRC | 03:28 | |
| oomichi | lifeless: ok, have a good weekend :) | 03:28 |
| *** shuquan has quit IRC | 03:29 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-neutron-full-kilo 263108,2,a6fc89e | 03:30 |
| *** tochi_ has quit IRC | 03:30 | |
| hop | oomichi: hi, if you have time please review again https://review.openstack.org/#/c/261264/ | 03:31 |
| *** shuquan has joined #openstack-qa | 03:32 | |
| oomichi | hop: nova side patch (https://review.openstack.org/#/c/128940/) should be merged first because the microversion number will be changed | 03:32 |
| oomichi | hop: I'd like to see concrete http status codes(204, 200) on nova side before temepst-lib patch | 03:34 |
| hop | oomichi: ok, I got it | 03:37 |
| hop | oomichi: thanks | 03:41 |
| openstackgerrit | Merged openstack-infra/devstack-gate: create GRE bridge interfaces only once https://review.openstack.org/239363 | 03:49 |
| openstackgerrit | Merged openstack/tempest: Safe ram/disk/vcpus values for test-made flavor https://review.openstack.org/261475 | 03:49 |
| openstackgerrit | Merged openstack/tempest: Make volume volume_client to use **kwargs-part1 https://review.openstack.org/256271 | 03:55 |
| openstackgerrit | Merged openstack/tempest: Make volume volume_client to use **kwargs-part2 https://review.openstack.org/259249 | 03:56 |
| *** zhangjn has quit IRC | 03:56 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/264793 failed because of: gate-grenade-dsvm-partial-ncpu: https://bugs.launchpad.net/bugs/1532048, gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1532048, gate-tempest-dsvm-full-ceph: https://bugs.launchpad.net/bugs/1464259 | 03:57 |
| openstack | Launchpad bug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] | 03:57 |
| openstack | Launchpad bug 1464259 in OpenStack Compute (nova) liberty "Volumes tests fails often with rbd backend" [High,In progress] - Assigned to Matt Riedemann (mriedem) | 03:57 |
| *** shuquan has quit IRC | 03:58 | |
| *** jacky-zhang has quit IRC | 03:58 | |
| gmann | oomichi: please look - https://review.openstack.org/#/c/252277/10 | 04:00 |
| *** tochi has joined #openstack-qa | 04:00 | |
| *** nicholasgoracke has joined #openstack-qa | 04:03 | |
| *** edmondsw has quit IRC | 04:05 | |
| *** salv-orl_ has joined #openstack-qa | 04:09 | |
| *** salv-orlando has quit IRC | 04:12 | |
| *** eil397 has quit IRC | 04:15 | |
| *** yamamoto_ has joined #openstack-qa | 04:19 | |
| openstackgerrit | Merged openstack/tempest: Refactoring of setting 'name' variable in tempest/common/compute.py https://review.openstack.org/261504 | 04:20 |
| *** mriedem has quit IRC | 04:20 | |
| openstackgerrit | Merged openstack/tempest: Fixing the deprecated library function. https://review.openstack.org/262720 | 04:20 |
| *** yamamot__ has joined #openstack-qa | 04:28 | |
| *** yamamoto_ has quit IRC | 04:29 | |
| *** salv-orl_ has quit IRC | 04:30 | |
| *** salv-orlando has joined #openstack-qa | 04:30 | |
| openstackgerrit | Merged openstack/tempest: remove log_console_of_servers from get_remote_client https://review.openstack.org/264313 | 04:32 |
| *** nicholasgoracke has quit IRC | 04:33 | |
| *** jacky-zhang has joined #openstack-qa | 04:34 | |
| *** markvoelker has quit IRC | 04:38 | |
| *** tochi_ has joined #openstack-qa | 04:45 | |
| *** ramineni has joined #openstack-qa | 04:46 | |
| ramineni | sdague: hi | 04:47 |
| *** nicholasgoracke has joined #openstack-qa | 04:52 | |
| *** shausy has joined #openstack-qa | 05:03 | |
| *** ramineni1 has joined #openstack-qa | 05:03 | |
| *** ramineni2 has joined #openstack-qa | 05:05 | |
| *** nicholasgoracke has quit IRC | 05:06 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: Fix SIGITIN hang with apt-get https://review.openstack.org/265053 | 05:06 |
| *** ramineni has quit IRC | 05:07 | |
| *** yamamoto_ has joined #openstack-qa | 05:07 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: Fix SIGITIN hang with apt-get https://review.openstack.org/265053 | 05:07 |
| *** ramineni1 has quit IRC | 05:07 | |
| *** yamamot__ has quit IRC | 05:07 | |
| *** ramineni has joined #openstack-qa | 05:08 | |
| *** ramineni2 has quit IRC | 05:09 | |
| *** hop has quit IRC | 05:10 | |
| *** nicholasgoracke has joined #openstack-qa | 05:12 | |
| *** ramineni has quit IRC | 05:12 | |
| *** salv-orlando has quit IRC | 05:12 | |
| *** salv-orlando has joined #openstack-qa | 05:12 | |
| oomichi | gmann: LGTM | 05:14 |
| gmann | oomichi: Thanks | 05:14 |
| *** yamamoto_ has quit IRC | 05:19 | |
| openstackgerrit | gongysh proposed openstack-dev/devstack: Expose right identity API version and auth type https://review.openstack.org/243002 | 05:20 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/264119 failed because of: gate-grenade-dsvm-multinode: https://bugs.launchpad.net/bugs/1355573 | 05:26 |
| openstack | Launchpad bug 1355573 in grenade "tempest volume scenarios periodically fail /w SSHTimeout" [Undecided,New] | 05:26 |
| *** amotoki has joined #openstack-qa | 05:27 | |
| *** jamielennox is now known as jamielennox|away | 05:27 | |
| *** jacky-zhang has quit IRC | 05:29 | |
| *** mragupat has joined #openstack-qa | 05:40 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-neutron-full 244440,8,8bdb0b8 | 05:41 |
| *** mragupat has quit IRC | 05:42 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: WIP: Just a test https://review.openstack.org/265080 | 05:42 |
| *** mragupat has joined #openstack-qa | 05:42 | |
| *** neeti has joined #openstack-qa | 05:46 | |
| *** dave-mccowan has quit IRC | 05:49 | |
| *** zhangjn has joined #openstack-qa | 05:52 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Add tests for Nova microversion v2.2 https://review.openstack.org/244996 | 05:56 |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Add tests for Nova microversion v2.2 https://review.openstack.org/244996 | 05:58 |
| *** zhangjn_ has joined #openstack-qa | 05:58 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Separate base microversion client and compute client https://review.openstack.org/265051 | 05:59 |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Remove leftover sample data https://review.openstack.org/265083 | 06:00 |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Switch to a scatter plot for test runtime https://review.openstack.org/265084 | 06:00 |
| *** zhangjn_ has quit IRC | 06:02 | |
| *** zhangjn has quit IRC | 06:02 | |
| *** Poornima has joined #openstack-qa | 06:03 | |
| *** zhangjn has joined #openstack-qa | 06:03 | |
| openstackgerrit | Merged openstack/tempest-lib: Migrated quotas_client and security_groups_client https://review.openstack.org/263986 | 06:05 |
| *** patchbot has quit IRC | 06:09 | |
| *** notmyname has quit IRC | 06:09 | |
| *** patchbot has joined #openstack-qa | 06:09 | |
| *** mgagne has quit IRC | 06:09 | |
| *** andreaf has quit IRC | 06:09 | |
| *** notmyname has joined #openstack-qa | 06:10 | |
| *** dtroyer has quit IRC | 06:11 | |
| jlvillal | FYI: lifeless, mtreinish https://review.openstack.org/265053 fixed the issue I was seeing with apt-get hanging. Thanks for all the help. | 06:12 |
| *** vgridnev has joined #openstack-qa | 06:13 | |
| *** markvoelker has joined #openstack-qa | 06:13 | |
| *** mgagne has joined #openstack-qa | 06:16 | |
| *** mgagne is now known as Guest41804 | 06:16 | |
| *** andreaf has joined #openstack-qa | 06:17 | |
| *** dtroyer has joined #openstack-qa | 06:17 | |
| lifeless | jlvillal: glad its solved | 06:17 |
| *** markvoelker has quit IRC | 06:18 | |
| *** nicholasgoracke has quit IRC | 06:19 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Separate base microversion client and compute client https://review.openstack.org/265051 | 06:19 |
| *** yamamoto_ has joined #openstack-qa | 06:20 | |
| *** vgridnev has quit IRC | 06:20 | |
| *** SridharG has joined #openstack-qa | 06:23 | |
| *** shausy has quit IRC | 06:25 | |
| *** shausy has joined #openstack-qa | 06:26 | |
| *** dimtruck is now known as zz_dimtruck | 06:28 | |
| *** Swami has joined #openstack-qa | 06:29 | |
| *** Swami has quit IRC | 06:29 | |
| *** yamamoto_ has quit IRC | 06:31 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full 264018,3,2476d0e | 06:38 |
| *** mragupat has quit IRC | 06:45 | |
| *** shuquan has joined #openstack-qa | 06:47 | |
| *** anshul has joined #openstack-qa | 06:49 | |
| openstackrecheck | Console logs not available after 13:21s for gate-tempest-dsvm-full 265079,1,423ad0c | 06:51 |
| *** coolsvap is now known as coolsvap|away | 06:52 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tempest: Updated from global requirements https://review.openstack.org/262923 | 06:56 |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 264015,3,ca117ec | 07:05 |
| *** coolsvap|away is now known as coolsvap | 07:13 | |
| *** anshul has quit IRC | 07:17 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 264116,2,130f4fe | 07:18 |
| *** _nadya_ has joined #openstack-qa | 07:18 | |
| *** shuquan has quit IRC | 07:22 | |
| openstackgerrit | Merged openstack-dev/devstack: Remove uses of --os-url and --os-token https://review.openstack.org/261535 | 07:23 |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 265080,1,dca71f4 | 07:31 |
| *** anshul has joined #openstack-qa | 07:32 | |
| *** _nadya_ has quit IRC | 07:43 | |
| openstackgerrit | Ryota MIBU proposed openstack/tempest: add client_manager_class param to BaseTestCase https://review.openstack.org/255161 | 07:43 |
| openstackrecheck | Console logs not available after 13:21s for gate-grenade-dsvm 265053,5,905e73c | 07:45 |
| *** yamamoto_ has joined #openstack-qa | 07:45 | |
| *** amotoki has quit IRC | 07:46 | |
| *** amotoki has joined #openstack-qa | 07:46 | |
| *** ihrachys has joined #openstack-qa | 07:48 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add command deleting *.pyc before executing ostestr https://review.openstack.org/265125 | 07:54 |
| *** shuquan has joined #openstack-qa | 07:55 | |
| *** ihrachys_ has joined #openstack-qa | 07:56 | |
| *** oomichi has quit IRC | 07:56 | |
| *** ihrachys has quit IRC | 07:57 | |
| *** Thelo has joined #openstack-qa | 07:57 | |
| Thelo | HI, all does anyone know how to set up the git repository and branch for the library specified by LIBS_FROM_GIT in local.conf (devstack) | 07:58 |
| Thelo | ? | 07:58 |
| openstackrecheck | Console logs not available after 13:24s for gate-horizon-npm-run-lint 214896,54,7cb04e4 | 07:58 |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/261536 failed gate-grenade-dsvm-multinode in the gate queue with an unrecognized error | 07:58 |
| *** salv-orlando has quit IRC | 08:00 | |
| *** salv-orlando has joined #openstack-qa | 08:00 | |
| *** coolsvap is now known as coolsvap|away | 08:00 | |
| *** zhangjn_ has joined #openstack-qa | 08:04 | |
| *** shuquan has quit IRC | 08:04 | |
| *** coolsvap|away is now known as coolsvap | 08:05 | |
| *** zhangjn has quit IRC | 08:07 | |
| *** ihrachys_ has quit IRC | 08:08 | |
| openstackrecheck | Console logs not available after 13:24s for gate-nova-tox-functional 264765,2,d9aef9a | 08:12 |
| *** oomichi has joined #openstack-qa | 08:14 | |
| *** markvoelker has joined #openstack-qa | 08:15 | |
| *** yamahata has joined #openstack-qa | 08:15 | |
| *** yamamoto_ has quit IRC | 08:15 | |
| *** shuquan has joined #openstack-qa | 08:17 | |
| andreaf | gmann: +1 | 08:18 |
| *** markvoelker has quit IRC | 08:19 | |
| openstackgerrit | Merged openstack-dev/grenade: force smaller ssh connect timeout https://review.openstack.org/263877 | 08:19 |
| openstackrecheck | Console logs not available after 13:24s for gate-horizon-dsvm-integration 234986,30,e476879 | 08:26 |
| *** yamamoto has joined #openstack-qa | 08:27 | |
| *** nicholasgoracke has joined #openstack-qa | 08:31 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** nicholasgoracke has quit IRC | 08:35 | |
| *** ramineni has joined #openstack-qa | 08:38 | |
| *** salv-orlando has quit IRC | 08:38 | |
| *** salv-orlando has joined #openstack-qa | 08:39 | |
| openstackrecheck | Console logs not available after 13:31s for gate-horizon-npm-run-lint 234875,23,276926e | 08:39 |
| *** zhangjn_ has quit IRC | 08:42 | |
| *** ramineni has quit IRC | 08:43 | |
| *** zhangjn has joined #openstack-qa | 08:43 | |
| *** ramineni has joined #openstack-qa | 08:45 | |
| *** ramineni1 has joined #openstack-qa | 08:47 | |
| *** ramineni has quit IRC | 08:49 | |
| *** ramineni has joined #openstack-qa | 08:51 | |
| *** safchain has joined #openstack-qa | 08:51 | |
| *** ramineni1 has quit IRC | 08:51 | |
| openstackrecheck | Console logs not available after 13:23s for gate-horizon-npm-run-test 234873,23,d9ff14e | 08:52 |
| *** shuquan has quit IRC | 08:53 | |
| *** gfidente has joined #openstack-qa | 08:55 | |
| *** gfidente has quit IRC | 08:55 | |
| *** gfidente has joined #openstack-qa | 08:55 | |
| *** ramineni has quit IRC | 08:55 | |
| *** ramineni has joined #openstack-qa | 08:56 | |
| *** shardy has joined #openstack-qa | 08:57 | |
| openstackgerrit | Andrea Frittoli proposed openstack/qa-specs: add bp:client-manager-refactor https://review.openstack.org/92804 | 08:58 |
| *** bkopilov has quit IRC | 08:58 | |
| *** ramineni1 has joined #openstack-qa | 08:59 | |
| *** jlanoux has joined #openstack-qa | 08:59 | |
| andreaf | gmann, masayukig, oomichi, mtreinish, sdague: can I get reviews on the spec please: https://review.openstack.org/#/c/92804/7 - I addressed mtreinish comments there | 08:59 |
| *** ifarkas has joined #openstack-qa | 09:00 | |
| *** shuquan has joined #openstack-qa | 09:00 | |
| *** ramineni has quit IRC | 09:01 | |
| *** ramineni has joined #openstack-qa | 09:01 | |
| *** zhangjn has quit IRC | 09:02 | |
| oomichi | andreaf: seeing it | 09:02 |
| *** ramineni1 has quit IRC | 09:03 | |
| *** ramineni has quit IRC | 09:06 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full 259677,21,d8caf31 | 09:06 |
| *** Poornima has quit IRC | 09:12 | |
| *** shausy has quit IRC | 09:12 | |
| *** shausy has joined #openstack-qa | 09:13 | |
| *** dstanek has quit IRC | 09:16 | |
| *** dstanek has joined #openstack-qa | 09:18 | |
| *** rossella_s has joined #openstack-qa | 09:19 | |
| openstackrecheck | Console logs not available after 13:21s for gate-tempest-dsvm-neutron-dvr 244996,32,a6f8a42 | 09:19 |
| *** ihrachys has joined #openstack-qa | 09:20 | |
| *** ihrachys has quit IRC | 09:21 | |
| *** ihrachys has joined #openstack-qa | 09:24 | |
| *** Poornima has joined #openstack-qa | 09:24 | |
| *** shuquan has quit IRC | 09:30 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add min ver to extra libs for graph https://review.openstack.org/265153 | 09:30 |
| openstackrecheck | Console logs not available after 13:21s for gate-nova-python34 228578,34,aa09ae0 | 09:33 |
| masayukig | andreaf: Is a recheck needed for https://review.openstack.org/#/c/92804/? | 09:34 |
| *** ihrachys has quit IRC | 09:35 | |
| *** mpaolino has joined #openstack-qa | 09:38 | |
| *** ihrachys has joined #openstack-qa | 09:38 | |
| *** jordanP has joined #openstack-qa | 09:39 | |
| *** ihrachys_ has joined #openstack-qa | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| *** yamahata has quit IRC | 09:41 | |
| *** ihrachys has quit IRC | 09:43 | |
| openstackrecheck | Console logs not available after 13:21s for gate-horizon-npm-run-test 234874,25,4ba44be | 09:46 |
| openstackgerrit | Andrea Frittoli proposed openstack/qa-specs: add bp:client-manager-refactor https://review.openstack.org/92804 | 09:47 |
| andreaf | masayukig: done ^^^ | 09:47 |
| *** ihrachys_ has quit IRC | 09:51 | |
| *** [1]Thelo has joined #openstack-qa | 09:54 | |
| *** ihrachys has joined #openstack-qa | 09:55 | |
| *** Thelo has quit IRC | 09:56 | |
| *** [1]Thelo is now known as Thelo | 09:56 | |
| *** bkopilov has joined #openstack-qa | 09:59 | |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-docs 264942,2,00ad90e | 09:59 |
| *** hop has joined #openstack-qa | 10:04 | |
| mhorban | andreaf: Hi | 10:09 |
| andreaf | mhorban hi | 10:09 |
| *** salv-orl_ has joined #openstack-qa | 10:09 | |
| mhorban | andreaf: Could you please review https://review.openstack.org/#/c/245815/ | 10:09 |
| andreaf | mhorban: sure | 10:10 |
| mhorban | It is blocker for adding new job that test nova services running under Apache | 10:10 |
| mhorban | andreaf: ^ | 10:11 |
| openstackgerrit | Merged openstack-dev/devstack: Use keystone[ldap] for ldap packages https://review.openstack.org/208153 | 10:11 |
| *** cdent has joined #openstack-qa | 10:11 | |
| *** salv-orlando has quit IRC | 10:12 | |
| *** ihrachys has quit IRC | 10:13 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full-kilo 265051,3,264471c | 10:13 |
| *** Deng has quit IRC | 10:13 | |
| *** markvoelker has joined #openstack-qa | 10:15 | |
| *** markvoelker has quit IRC | 10:20 | |
| andreaf | mhorban: sorry I didn't catch that before, but I think the regex could be a bit "tighter" | 10:21 |
| andreaf | mhorban: something like (?:^|\/)v\d+(?:\.\d+)?(?:$|\/) | 10:21 |
| *** mpaolino has quit IRC | 10:23 | |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-full 261253,5,94f2adb | 10:26 |
| *** hoangcx has quit IRC | 10:26 | |
| *** jlanoux has quit IRC | 10:40 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full-kilo 262923,11,b236f75 | 10:40 |
| *** ihrachys has joined #openstack-qa | 10:42 | |
| *** tosky has joined #openstack-qa | 10:47 | |
| *** chem has joined #openstack-qa | 10:52 | |
| openstackrecheck | Console logs not available after 13:28s for gate-nova-tox-functional 202617,32,dc2ac0c | 10:53 |
| *** ihrachys has quit IRC | 10:54 | |
| *** SridharG has quit IRC | 10:57 | |
| openstackrecheck | Console logs not available after 13:25s for gate-tempest-dsvm-full-kilo 262465,2,ed7a5cd | 11:07 |
| *** dims has joined #openstack-qa | 11:09 | |
| *** lucas-dinner is now known as lucasagomes | 11:12 | |
| *** zhangjn has joined #openstack-qa | 11:16 | |
| *** zhangjn has quit IRC | 11:16 | |
| *** Poornima has quit IRC | 11:16 | |
| *** zhangjn has joined #openstack-qa | 11:17 | |
| *** zhangjn has quit IRC | 11:17 | |
| *** zhangjn has joined #openstack-qa | 11:17 | |
| yfried | gmann: https://review.openstack.org/#/c/259746 | 11:18 |
| yfried | afazekas: andreaf: gmann: can you please review? do you know what's wrong with the gate? | 11:19 |
| *** jlanoux has joined #openstack-qa | 11:20 | |
| openstackrecheck | Console logs not available after 13:27s for gate-grenade-dsvm 264259,5,bdd1f15 | 11:21 |
| *** cdent has quit IRC | 11:27 | |
| *** jlanoux_ has joined #openstack-qa | 11:27 | |
| andreaf | yfried: the issue with kilo jobs should have been fixed now | 11:28 |
| andreaf | yfried: if that's what you mean | 11:28 |
| *** jlanoux has quit IRC | 11:29 | |
| dmellado | andreaf: I've seen also more patches | 11:35 |
| dmellado | with issues with the kilo gate | 11:35 |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full 261253,5,668f10c | 11:35 |
| dmellado | could the issue have shown up again maybe? | 11:35 |
| dmellado | yfried: ^^ | 11:35 |
| yfried | dmellado: andreaf: not only kilo. experimental gate was really bad as well | 11:36 |
| dmellado | yfried: for the experimental gates, regarding the test accounts there are some race condition issues which can lead them to fail | 11:37 |
| dmellado | but regarding the other ones I'm not sure about | 11:37 |
| *** shardy has quit IRC | 11:37 | |
| dmellado | yfried: did you try to check the issue on the gates? if not, I'll have a look later, as soon as I got some slot | 11:42 |
| *** ihrachys has joined #openstack-qa | 11:43 | |
| yfried | dmellado: what do you mean "check"? | 11:43 |
| dmellado | to debug | 11:43 |
| *** rfolco has joined #openstack-qa | 11:45 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 264119,1,c45c0ae | 11:48 |
| *** amotoki_ has joined #openstack-qa | 11:52 | |
| *** amotoki has quit IRC | 11:54 | |
| *** cdent has joined #openstack-qa | 11:56 | |
| *** ihrachys_ has joined #openstack-qa | 11:57 | |
| *** ihrachys has quit IRC | 11:57 | |
| *** zhangjn_ has joined #openstack-qa | 11:58 | |
| *** zhangjn_ has quit IRC | 11:58 | |
| *** zhangjn has quit IRC | 11:59 | |
| *** zhangjn has joined #openstack-qa | 11:59 | |
| *** markvoelker has joined #openstack-qa | 12:01 | |
| *** SridharG has joined #openstack-qa | 12:01 | |
| andreaf | oomichi, gmann: I'm looking through the etherpads for a list of what is missing in terms of service clients work (split, rename etc) - do you have a pointer for me? | 12:02 |
| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm-partial-ncpu 265090,1,1307eb1 | 12:02 |
| *** tochi has quit IRC | 12:03 | |
| *** tochi_ has quit IRC | 12:03 | |
| *** zhangjn has quit IRC | 12:03 | |
| *** markvoelker has quit IRC | 12:06 | |
| dmellado | andreaf: I didn't have time to work on the endpoints-client | 12:09 |
| dmellado | do you want to take that one over? | 12:09 |
| dmellado | andreaf: https://etherpad.openstack.org/p/mitaka-tempest-service-clients | 12:10 |
| andreaf | dmellado: not myself, but I wanted to send my colleagues a list of what's left to be done to see if anyone can help | 12:10 |
| dmellado | feel free to fetch that one, I'll deassign myself from it ;) | 12:10 |
| *** ihrachys has joined #openstack-qa | 12:11 | |
| *** SridharG has quit IRC | 12:12 | |
| *** ihrachys_ has quit IRC | 12:12 | |
| jlanoux_ | Hi andreaf. Would you be able to have a look at https://review.openstack.org/#/c/259515/ please? | 12:15 |
| andreaf | dmellado: ok thanks | 12:15 |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm-neutron 247240,9,a5e72f0 | 12:15 |
| *** SridharG has joined #openstack-qa | 12:15 | |
| *** SridharG has quit IRC | 12:16 | |
| *** ihrachys has quit IRC | 12:16 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/257744 failed gate-grenade-dsvm in the gate queue with an unrecognized error | 12:16 |
| andreaf | jlanoux_ : sure | 12:19 |
| jlanoux_ | andreaf: thanks | 12:19 |
| *** SridharG has joined #openstack-qa | 12:28 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 265154,1,2f5f8c9 | 12:29 |
| *** ihrachys has joined #openstack-qa | 12:31 | |
| *** ihrachys has quit IRC | 12:31 | |
| *** coolsvap is now known as coolsvap|away | 12:34 | |
| *** TuanNguyen has joined #openstack-qa | 12:34 | |
| *** jffische has joined #openstack-qa | 12:39 | |
| *** dims_ has joined #openstack-qa | 12:40 | |
| *** dims has quit IRC | 12:41 | |
| *** weshay_xchat has joined #openstack-qa | 12:41 | |
| *** weshay_xchat is now known as weshay | 12:42 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full-kilo 261283,2,0223d05 | 12:43 |
| *** jffische has quit IRC | 12:45 | |
| *** markvoelker has joined #openstack-qa | 12:45 | |
| *** neeti has quit IRC | 12:53 | |
| openstackrecheck | Console logs not available after 13:23s for gate-glance-pep8 234340,4,dc3aebf | 12:57 |
| jordanP | jlanoux_, your patch doesn't pass the gate. It's hard to review because so much things can change until you have a working patch | 12:58 |
| jordanP | also, the create_server, create_test_servers seem are super hard to follow | 12:58 |
| jordanP | we shouldn't have 150 LoC methods | 12:59 |
| *** alaski is now known as lascii | 12:59 | |
| jlanoux_ | jordanP: I know. I just wanted some comment about the strategy because it will impact a lot of files and I didn't want to spend hours implementing that to hear that it's not good. | 13:00 |
| jlanoux_ | test_create_server is working and the gate shows it | 13:00 |
| jordanP | why should it impact a lot of files ? | 13:00 |
| jlanoux_ | I would need to modify all test creating a server | 13:01 |
| jordanP | to replace active by pingable ? | 13:01 |
| jordanP | only servers that need to be SSHed into right ? | 13:01 |
| jlanoux_ | no, because of the addition of remote in the tuple | 13:01 |
| jlanoux_ | yes for your second q's | 13:02 |
| jordanP | jlanoux_, why would a method called "create_server" return a SSHClient ? | 13:02 |
| jordanP | what if a just want a server but couldn't care less about it beeing SSHable ? | 13:03 |
| jlanoux_ | jordanP: in the case of an non-sshable server, remote will be empty | 13:04 |
| jlanoux_ | jordanP: the goal of the wrapper is to enable a wait_for PINGABLE or SSHABLE | 13:04 |
| jlanoux_ | a ssh_client is needed for that | 13:04 |
| sdague | jlanoux_: so, honestly, this just makes the compute api side diverge even further from the scenario test side | 13:04 |
| sdague | those need to be converged first | 13:05 |
| sdague | I was just banging my head into that on the scenario side the other day | 13:05 |
| jlanoux_ | sdague: My plan is to refactor the scenario tests to match the API | 13:05 |
| jordanP | jlanoux_, what about adding these waiter but "outside" the create_server method ? | 13:06 |
| sdague | so I think this deep coupling of concerns is honestly a bad pattern | 13:06 |
| sdague | create_server should return a server | 13:07 |
| jordanP | agreed | 13:07 |
| sdague | you should have a utility that builds a connection object from a server | 13:07 |
| jlanoux_ | sdague: jordanP coupling is needed because if one wants an sshable server, additionnal resources needs to be created and passed on | 13:07 |
| sdague | only some times | 13:08 |
| jordanP | the fact that "remote" will most of the time be "None" because most of the time we don"t want a SSHable server is weird | 13:08 |
| sdague | yeh, that's a bad pattern | 13:08 |
| jordanP | jlanoux_, coupling can be done in another wrapper method | 13:09 |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-pep8 151081,24,bfd6600 | 13:11 |
| jlanoux_ | jordanP: sdague what do you suggest to do then? | 13:15 |
| sdague | I suggest we get a handle on the current code first | 13:15 |
| *** dims_ has quit IRC | 13:16 | |
| jordanP | yeah... so sdague the main problem is that how we create a test_server in scenarios and api tests are different right ? | 13:17 |
| jlanoux_ | not anymore | 13:17 |
| sdague | jordanP: right, mostly because there are now dozens of convenience functions all over the place | 13:17 |
| jlanoux_ | sdague: what would you like to see? | 13:19 |
| jordanP | in scenarios/manager.py we have a method called create_server() which is a wrapper around create_test_server() defined in common/compute.py | 13:21 |
| jordanP | this looks sane | 13:21 |
| jlanoux_ | jordanP: Oo are you kidding? | 13:21 |
| jordanP | hum ? | 13:22 |
| sdague | so, why another wrapper layer? | 13:22 |
| jordanP | I am stating my comprehension of the current code | 13:22 |
| sdague | every time we have wrapper layers, and you need to add a new feature, where do you put it? | 13:23 |
| sdague | that's the same problem with the current class hiearchy | 13:23 |
| jlanoux_ | jordanP: sorry, misread what you said | 13:23 |
| sdague | jlanoux_: so honestly, without spending a week trying to unwind all of this, I don't know exactly how to fix it. | 13:24 |
| sdague | but I know as long as there are functions that take server_or_ip as a param, I'm going to -1 any new complexity coming in | 13:25 |
| jlanoux_ | sdague: ok | 13:25 |
| sdague | because a really fundamental thing is that you know what your inputs and outputs are, and they are always the same | 13:26 |
| jordanP | then let's fix that server_or_ip first | 13:26 |
| *** dims has joined #openstack-qa | 13:26 | |
| sdague | which is also why that pattern of returning an extra param that's often none is really weird | 13:27 |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-neutron-full 231460,12,5f8829b | 13:27 |
| sdague | longer term, RemoteClient today only knows about an IP | 13:28 |
| sdague | I think that's the wrong model. I think it needs to know about the server as well | 13:28 |
| sdague | and make RemoteClient responsible for creating the floating ip if it needs it | 13:29 |
| sdague | because if it knew the server, it could also do analysis on the console log when things go wrong to figure out if the guest even has correct ip addresses assigned to it | 13:31 |
| *** edmondsw has joined #openstack-qa | 13:35 | |
| jlanoux_ | sdague: jordanP ok - I'll do something about the server_or_ip | 13:43 |
| jordanP | jlanoux_, https://review.openstack.org/#/c/264323 might be a start | 13:45 |
| *** bochi-michael has quit IRC | 13:46 | |
| jordanP | if you can come up with something sane, I think sdague and I could review it quickly to get it merged | 13:46 |
| jordanP | I am interested in cleaning up tempest code these days | 13:46 |
| *** sileht has quit IRC | 13:46 | |
| jlanoux_ | jordanP: ok | 13:47 |
| *** sileht has joined #openstack-qa | 13:48 | |
| *** dkranz has joined #openstack-qa | 13:48 | |
| sdague | jordanP: yeh, that patch isn't going to work at all because the whole floating IP access resource framework isn't plumbed into the scenario side | 13:48 |
| openstackrecheck | Console logs not available after 13:29s for gate-swift-python27 248377,6,b9515c7 | 13:48 |
| sdague | which made me a bit sad | 13:49 |
| sdague | but, honestly, the more I think about this, I feel like the abstraction layer is really really out of whack here. RemoteClient should manage building the access path to the server itself, which it can do if it has the server info. It can also update the accessIP field once it's built such a path so that it's not creating lots of extra floating ips | 13:51 |
| sdague | right now in the scenario side we double and tripple allocate floating ips | 13:51 |
| sdague | because of get_server_or_ip | 13:51 |
| *** e0ne has joined #openstack-qa | 13:52 | |
| *** nicholasgoracke has joined #openstack-qa | 13:58 | |
| jlanoux_ | sdague: May I continue to work and your change or would you prefer to do it? | 13:58 |
| sdague | jlanoux_: feel free to take it over | 14:00 |
| sdague | if you can get the server_or_ip stuff all sorted out, I think we can tackle RemoteClient in a cleaner way | 14:00 |
| sdague | and get everything you want out of this as well | 14:00 |
| jlanoux_ | sdague: ok - will do | 14:00 |
| sdague | thanks | 14:00 |
| jlanoux_ | np - thanks for the comment both | 14:01 |
| *** dtantsur is now known as dtantsur|brb | 14:02 | |
| *** nicholasgoracke has quit IRC | 14:03 | |
| *** ccarmack has joined #openstack-qa | 14:04 | |
| jordanP | jlanoux_, do you agree that every tests in tempest/api/compute/ should use tempest/api/base.py::create_test_server() and must not call client.create_server() directly ? | 14:08 |
| *** salv-orl_ has quit IRC | 14:08 | |
| *** salv-orlando has joined #openstack-qa | 14:08 | |
| jlanoux_ | jordanP: Yes. Or test can call common/compute.py directly. | 14:09 |
| jordanP | that would be a first step. The next step would be to kill tempest/api/base.py::create_test_server() because it's just a silly wrapper around l common/compute.py | 14:09 |
| jlanoux_ | jordanP: I agree 100%. We kept the wrapper for convenience. | 14:10 |
| jordanP | ok we are on the same page, this would remove some level of nesting and bring some consistency | 14:10 |
| openstackrecheck | Console logs not available after 13:21s for gate-nova-python27 258617,8,581e7f5 | 14:10 |
| jordanP | so I am going to work on that right now (step 1), I'llkeep the silly wrapper for now | 14:10 |
| jlanoux_ | jordanP: Yes - And actually, it's exactly the same than for the wrapper in scenario/manager.py | 14:11 |
| *** dave-mccowan has joined #openstack-qa | 14:13 | |
| *** jordanP is now known as jordanP|away | 14:14 | |
| *** anshul has quit IRC | 14:14 | |
| *** lucasagomes is now known as lucas-hungry | 14:14 | |
| *** shausy has quit IRC | 14:17 | |
| *** nicholasgoracke has joined #openstack-qa | 14:19 | |
| *** rossella_s has quit IRC | 14:22 | |
| *** rossella_s has joined #openstack-qa | 14:22 | |
| *** cdent has quit IRC | 14:25 | |
| *** shardy has joined #openstack-qa | 14:31 | |
| *** dtantsur|brb is now known as dtantsur | 14:34 | |
| openstackrecheck | Console logs not available after 13:26s for gate-tempest-dsvm-neutron-full 229854,11,857447c | 14:34 |
| *** mriedem has joined #openstack-qa | 14:42 | |
| *** jecarey has joined #openstack-qa | 14:47 | |
| *** jaypipes has joined #openstack-qa | 14:49 | |
| *** ChanServ sets mode: +o jaypipes | 14:49 | |
| *** jordanP|away is now known as jordanP | 14:50 | |
| *** amotoki_ has quit IRC | 14:51 | |
| *** rbak__ has joined #openstack-qa | 14:53 | |
| *** jffische has joined #openstack-qa | 14:54 | |
| *** vgridnev has joined #openstack-qa | 15:01 | |
| openstackrecheck | Console logs not available after 13:24s for gate-grenade-dsvm 245916,11,359060f | 15:07 |
| jroll | mtreinish: OS_TEST_TIMEOUT is for the full testr run, right? (just to doublecheck) | 15:07 |
| jroll | trying to figure out the best way to switch over to tempest plugin, wondering if we run both at once if it will time out given we'll be running BaremetalBasicOps twice for a short time | 15:07 |
| *** jaypipes is now known as leakypipes | 15:08 | |
| *** zz_dimtruck is now known as dimtruck | 15:09 | |
| *** nicholasgoracke has quit IRC | 15:13 | |
| sdague | jroll: OS_TEST_TIMEOUT is per test | 15:16 |
| sdague | jroll: https://github.com/openstack/tempest/blob/30d3c9b01c4cdb6fffafbf6466e8918859b41dbc/tempest/test.py#L405-L411 | 15:17 |
| sdague | it's a test fixture | 15:17 |
| jroll | sdague: oh cool, thanks | 15:19 |
| *** lucas-hungry is now known as lucasagomes | 15:20 | |
| andreaf | sdague: the reason I originally proposed to get the ping and ssh checks in the create_server helper is that that's something that should ideally be done for most servers we create | 15:20 |
| andreaf | sdague: and I wanted to provide a simple way for all tests to do that in the same manner - so that it's also easier to troubleshoot, as the code is common | 15:21 |
| andreaf | sdague: i.e. just create_server(ssh_check=true) | 15:21 |
| sdague | andreaf: honestly, I think that's a bad approach | 15:22 |
| sdague | it makes the tests more magical and implicit | 15:22 |
| andreaf | sdague: I see your point about the separation of concerns, however not everything required for ssh-ing into a server can be done after the server booted | 15:22 |
| sdague | which means when things go wrong... which they do all the time | 15:22 |
| sdague | figuring out where they went wrong is so much harder | 15:22 |
| sdague | and by making those things implicit people stop thinking about what their test is doing | 15:23 |
| *** nicholasgoracke has joined #openstack-qa | 15:23 | |
| *** dansmith is now known as superdan | 15:23 | |
| sdague | so we end up with things like allocating 3 floating ips for every server when checking because it was an implicit thing | 15:23 |
| sdague | or people writing tests A, B, C, D and B does A, C does B and A, D does C, B, A | 15:24 |
| andreaf | sdague: true, that's a risk - but on the other side you end up like we are now, with every test implementing it's own way of doing an ssh check - which is not necessarily a bad thing - only it's very hard to maintain and troubleshoot | 15:26 |
| sdague | so you have dedicated class to do that thing | 15:26 |
| sdague | but you still make people explicitly call it when they want it | 15:27 |
| andreaf | sdague: yes, that's what we build in common/compute.py - dedicated helper for that | 15:27 |
| sdague | so that has overrun multiple concerns | 15:28 |
| andreaf | sdague: I think I may be missing something in your point, I guess seeing a code sample could help | 15:28 |
| *** nwkarsten has joined #openstack-qa | 15:28 | |
| openstackgerrit | Federico Ressi proposed openstack-dev/devstack: Don't reinstall package just after installing. https://review.openstack.org/262074 | 15:29 |
| sdague | ok, so I think burying the validation in the server is not a good model | 15:29 |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: Fix SIGITIN hang with apt-get https://review.openstack.org/265053 | 15:29 |
| openstackgerrit | Federico Ressi proposed openstack-dev/devstack: Don't die when yum fails. https://review.openstack.org/263061 | 15:30 |
| sdague | especially when we've got such a high failure rate on that today, and we don't know why. | 15:30 |
| *** mtanino has joined #openstack-qa | 15:31 | |
| andreaf | sdague: heh, my idea was that if we can exercise it more, than we may reduce that failure rate - and to do that I wanted to make ssh check easy to use for test writers | 15:32 |
| sdague | andreaf: well, I don't see anyone else really looking at the debug here. So I feel like that's the wrong approach | 15:33 |
| sdague | so here are the concerns that exist: | 15:33 |
| sdague | 1) API calls are async, we need to wait some times | 15:35 |
| sdague | 2) we need cleanup so we aren't leaking the world | 15:35 |
| sdague | 3) we need to be able to be able to perform more advanced validation, which may need new resources | 15:36 |
| moravec | mtreinish: SpamapS: is there a puppet module to create the sql database used to store test results? | 15:37 |
| sdague | I think that 3) is really a fully externalized class, because I would expect it to extend over time. Which means that things like adding routes / floating ips / etc needs to end up there | 15:37 |
| sdague | a lot of these convenience functions were created for 1 & 2 | 15:37 |
| sdague | but even there we've gotten a bit muddled | 15:37 |
| SpamapS | moravec: create it? I don' tknow. | 15:38 |
| mtreinish | moravec: there's: https://github.com/openstack-infra/puppet-subunit2sql/blob/master/manifests/server.pp which will handle the subunit2sql db init | 15:38 |
| SpamapS | moravec: the infra puppet will run the migration command automatically | 15:38 |
| mtreinish | but it assumes a mysql server already exists | 15:38 |
| SpamapS | ah yeah init is there too | 15:38 |
| *** mragupat has joined #openstack-qa | 15:38 | |
| mtreinish | there are other puppet modules you could use to setup a mysql server if you need that too | 15:39 |
| sdague | I think it's probably a good rule of thumb that if a function returns a tuple, it's doing too many things | 15:39 |
| mtreinish | moravec: we used trove for the infra db so we just dumped the creds and ip into hiera and passed it to that module to setup the db | 15:40 |
| sdague | it's also massively unclear to me why any of the server_create functions would be plumbed for multiple create | 15:40 |
| sdague | that adds a bunch of complexity | 15:40 |
| sdague | that's barely used | 15:41 |
| andreaf | sdague: yes it does but that's what the API allows for, so you need to test that as well | 15:41 |
| sdague | andreaf: right, but we're talking about convenience methods | 15:41 |
| sdague | convenience methods should only be used when they aren't the primary purpose of the test | 15:42 |
| moravec | mtreinish: spamaps:ok, thanks for the hints. I'm thinking about how to add subunit2sql to puppet-openstackci for anyone running tests in their own env beyond the gate. Probably need to consider setting up a new db or pointing to existing. | 15:42 |
| andreaf | true | 15:42 |
| *** e0ne has quit IRC | 15:42 | |
| *** cdent has joined #openstack-qa | 15:42 | |
| sdague | multiple create is used 3 places in possitive tests in tempest, the fact that it's accounted for in a wrapper layer is really odd | 15:43 |
| sdague | and a big chunk of debt | 15:43 |
| sdague | and it also means that you never know what you are getting out of that function | 15:44 |
| andreaf | sdague: I guess that's easy to drop if we wanted to | 15:44 |
| sdague | because it takes kwargs for everything | 15:44 |
| *** yamahata has joined #openstack-qa | 15:44 | |
| *** pglass has joined #openstack-qa | 15:44 | |
| andreaf | sdague: the multiple create was a good way to discover scale issues though, we found a few nasty issue back at Diablo / Essex times with that | 15:45 |
| *** timello_ has quit IRC | 15:45 | |
| sdague | andreaf: that's fine, it doesn't need to be in this wrapper | 15:45 |
| *** maurosr has quit IRC | 15:46 | |
| mtreinish | moravec: sure, there is an upstream puppet module for dealing with mysql you probably should just call to that to setup a blank db and then point puppet-subunit2sql at that | 15:46 |
| sdague | same issue with all that volume_backed code | 15:46 |
| sdague | that is used for 1 test class? | 15:46 |
| *** maurosr has joined #openstack-qa | 15:46 | |
| sdague | stuff should not be in this common wrapper unless it's used in at least a dozen test classes | 15:46 |
| sdague | special cases are fine | 15:47 |
| *** timello_ has joined #openstack-qa | 15:47 | |
| sdague | adding complexity to all code because of one off tests is how we get into a giant mess | 15:47 |
| *** sabeen has joined #openstack-qa | 15:47 | |
| *** jlanoux_ has quit IRC | 15:49 | |
| *** jlanoux has joined #openstack-qa | 15:49 | |
| *** bnemec has quit IRC | 15:50 | |
| *** sabeen2 has joined #openstack-qa | 15:52 | |
| *** zhangjn has joined #openstack-qa | 15:52 | |
| andreaf | sdague: sure that's a good point - I would say it's a trade off between having multiple ways of doing things and having simpler code | 15:52 |
| sdague | basically, create_test_server needs a scope. being a generic adapter to POST /servers is way too broad imho. And today it's even broader than that as it also embeds calls to cinder | 15:52 |
| andreaf | sdague: and perhaps we are off balance towards the complex code | 15:52 |
| sdague | yeh, test code needs to be simple and boring | 15:52 |
| sdague | because you have to be more sure that the test code is correct than the code you are testing with it | 15:53 |
| sdague | otherwise you never know if things are working for real | 15:53 |
| andreaf | sdague: heh, right - we have unit test coverage for a lot of the more complex things in tempest now | 15:53 |
| sdague | right, but even that, if it's not simple enough, can hide lots of behavior issues | 15:54 |
| *** zhangjn has quit IRC | 15:54 | |
| sdague | you don't just fix test complexity by adding more tests to test the tests | 15:54 |
| mtreinish | timothyb89: on: https://review.openstack.org/#/c/265084/ any ideas why there's no interactivity? | 15:54 |
| *** sabeen has quit IRC | 15:54 | |
| mtreinish | timothyb89: oh I just realized it's the avg that breaks the interactivity, when I turn that off in the ui it shows me the pop up | 15:54 |
| mtreinish | must be something in nvd3 | 15:55 |
| *** safchain has quit IRC | 15:55 | |
| *** zhangjn has joined #openstack-qa | 15:55 | |
| openstackgerrit | Merged openstack-dev/devstack: Move horizon config and start to appropriate phase in stack.sh https://review.openstack.org/257744 | 15:56 |
| *** lekha is now known as lekha|away | 15:57 | |
| andreaf | sdague: again, I think it's a balance - however simple the test code, we have a lot of different configurations to support, we need the test to work against multiple branches - so we need certain degrees of complexity | 15:58 |
| *** nwkarste_ has joined #openstack-qa | 15:58 | |
| sdague | andreaf: sure, however this is way out of whack right now | 15:58 |
| sdague | and this whole idea of parameters to functions not having well defined types, and wrapper functions being entirely kwargs based really need to get addressed | 15:59 |
| sdague | it also makes the code base largely impenetrable for new people | 16:00 |
| sdague | so directly inhibits contributors showing up | 16:00 |
| *** bnemec has joined #openstack-qa | 16:00 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-postgres-full 188436,8,4e93421 | 16:01 |
| openstackgerrit | Scott DAngelo proposed openstack-dev/grenade: Add copying new api-paste.ini to Cinder upgrade https://review.openstack.org/264995 | 16:01 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/264167 failed gate-tempest-dsvm-neutron-full in the gate queue with an unrecognized error | 16:01 |
| jordanP | I agree that we really need to make things simpler around server creation | 16:02 |
| jordanP | there's a lot of magic and special cases in the code currently | 16:02 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/197569 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1355573 | 16:02 |
| openstack | Launchpad bug 1355573 in grenade "tempest volume scenarios periodically fail /w SSHTimeout" [Undecided,New] | 16:02 |
| openstackgerrit | Stephen Lowrie proposed openstack/tempest: Add tempest-list-plugins https://review.openstack.org/251542 | 16:02 |
| *** nwkarsten has quit IRC | 16:03 | |
| slowrie | mtreinish: I updated it to use the TempestTestPluginManager | 16:03 |
| andreaf | sdague: I wouldn't say there's an "idea" of functions having to not well defined types - I guess that's the "server or ip" thing you're referring to - I'm not sure where that comes from - but I think that's an isolated case that we should just fix - I hope so at least | 16:03 |
| mtreinish | slowrie: ok cool, it was just a question. Calling stevedore directly was probably fine too :) | 16:04 |
| andreaf | sdague, jordanP: anyways, I agree we need to simplify things where complexity is not needed | 16:04 |
| slowrie | mtreinish: Yeah, but it's already instantiating stevedore with a couple more options | 16:04 |
| slowrie | mtreinish: Don't lose anything by doing it that way | 16:05 |
| *** zhangjn has quit IRC | 16:05 | |
| *** regXboi has joined #openstack-qa | 16:06 | |
| andreaf | sdague: well, the ssh validation is one of the topics for the mid-cycle, hopefully we can highlight and fix the most pressing issues there - if not earlier | 16:06 |
| mtreinish | slowrie: nice: http://paste.openstack.org/show/483326/ | 16:07 |
| andreaf | sdague: is there a chance I could help doing the things you mentioned yesterday you could not do in tempest because of rest client indirection? | 16:07 |
| *** hop has quit IRC | 16:08 | |
| *** salv-orl_ has joined #openstack-qa | 16:08 | |
| sdague | well my current blocking point is the server_or_ip which is plumbed through a large number of layers | 16:08 |
| sdague | and I really don't have the time to unwind it | 16:09 |
| *** zigo has quit IRC | 16:09 | |
| openstackgerrit | Guillaume Chenuet proposed openstack/tempest: Fixing test_create_port_in_allowed_allocation_pool when 3 (or more) controllers https://review.openstack.org/243159 | 16:09 |
| andreaf | sdague: so we need to move towards that being a server or an IP - and if I got that right you propose that to be a server? | 16:09 |
| jlanoux | andreaf: I'm on it | 16:10 |
| *** zigo has joined #openstack-qa | 16:10 | |
| *** nwkarste_ has quit IRC | 16:10 | |
| andreaf | sdague: there's information about how the server was created that is not anymore in the server object returned by the API, so passing a server to RemoteClient means that RemoteClient needs to do some guessing work in trying to connect to the server | 16:11 |
| *** salv-orlando has quit IRC | 16:11 | |
| sdague | andreaf: yes, it needs to be a server | 16:11 |
| sdague | andreaf: what information is that? | 16:11 |
| sdague | and can it be directly sent to RemoteClient | 16:11 |
| andreaf | well you get a list of IP addresses for the server, but you don't know which one is the one that is supposed to be used according to the network setup and configuration | 16:12 |
| sdague | that's why the api for accessIP exists in Nova | 16:13 |
| sdague | accessIP is user settable | 16:13 |
| sdague | so either that should be updated on the server, when we know what it should be | 16:13 |
| sdague | or the semantic information about what IP we should care about should be passed onto RemoteClient | 16:14 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/264793 failed because of: gate-tempest-dsvm-large-ops: https://bugs.launchpad.net/bugs/1491949 | 16:14 |
| openstack | Launchpad bug 1491949 in OpenStack Compute (nova) "gate-tempest-dsvm-large-ops fails to deallocate instance network due to rpc timeout" [Medium,Confirmed] | 16:14 |
| sdague | honestly, I'm not sure which is better, but they are both options | 16:14 |
| *** hemnafk is now known as hemna | 16:16 | |
| openstackgerrit | Brant Knudson proposed openstack-dev/devstack: Keystone support deploy in uwsgi https://review.openstack.org/257571 | 16:18 |
| andreaf | sdague: remote client is about executing commands on a VM, I'm not sure it should get into the business of understanding networking options, creating SGs or so - I don't see any advantage in that - why does Remote client need to get anything more than in IP, an ssh key and username? | 16:19 |
| sdague | because of what happens when things go wrong | 16:19 |
| sdague | Can't ping 10.1.0.5 | 16:19 |
| sdague | what instance is that? | 16:19 |
| sdague | is that an ip address that instance should even have? | 16:20 |
| sdague | is it an ip address that instance got over dhcp? | 16:20 |
| sdague | one of our big issues right now is some times the instance never got an address configured | 16:20 |
| sdague | which we could actually figure out and tell people, if we know the ip address and the instance info at the same time | 16:21 |
| sdague | and look at the console | 16:21 |
| *** mriedem is now known as mriedem_meeting | 16:21 | |
| *** aopincar has joined #openstack-qa | 16:23 | |
| andreaf | I'm not sure that's the business of the remote client - but I see your point and I agree - failure needs the context | 16:23 |
| andreaf | the idea of having one way of doing validation was also about that, being able to write that code in a debuggable way | 16:24 |
| andreaf | whether that's in remote client or not is probably not the main point | 16:25 |
| *** dims has quit IRC | 16:25 | |
| *** dims has joined #openstack-qa | 16:26 | |
| andreaf | an easy way would be to get the server unplugged, then pass it to an handler which plugs-it into the network, sets up SG and rules etc, but that's not how cloud is used typically, and it would hide a race conditions that happen if you boot the server and plug it to the NW right away | 16:26 |
| openstackrecheck | Console logs not available after 13:26s for gate-tempest-dsvm-nova-v20-api 265272,1,352bd5a | 16:27 |
| andreaf | mtreinish: I re-spin this spec based on your comments https://review.openstack.org/#/c/92804/ please let me know what you think | 16:28 |
| *** devkulkarni has joined #openstack-qa | 16:29 | |
| *** salv-orl_ has quit IRC | 16:29 | |
| *** salv-orlando has joined #openstack-qa | 16:30 | |
| mtreinish | andreaf: what about: https://review.openstack.org/173334 ? | 16:32 |
| *** aopincar has quit IRC | 16:33 | |
| *** ccneill has joined #openstack-qa | 16:33 | |
| andreaf | mtreinish: heh ok I've drawn a bit into fixing the client manager to be able to proceed with cred manager migration | 16:33 |
| sdague | andreaf: so where do you put the code about explaining what went wrong? | 16:34 |
| sdague | because the most important thing about a test system is that when it fails, it tells you why, to the highest level of fidelity it can | 16:34 |
| mtreinish | andreaf: you've got too many specs :) | 16:34 |
| *** castulo has joined #openstack-qa | 16:34 | |
| andreaf | mtreinish: indeed - I'm not sure how testresources / fixtures would help there - is that clear to you? | 16:35 |
| andreaf | sdague: right, the most important thing is what you mentioned it to get good and reliable details about the failure - if you feel strongly about that being in RemoteClient so be it - I only want to avoid doing guess work to find out what IP to use | 16:37 |
| mtreinish | andreaf: I think the suggestion was to use it instead of a decorator. I'm not sure it helps or makes it worse, although it might make external consumption easier, I'm not sure | 16:38 |
| mtreinish | andreaf: did you ask afazekas for more details on his suggestion? | 16:39 |
| sdague | andreaf: right, which means we need to be explicit about it | 16:39 |
| sdague | being explicit is fine, when RemoteClient is instantiated we can tell it both the server and the ip it should validate on | 16:40 |
| sdague | but, the reality is, a lot of this is muddled behind the fact that there is some cleanup work that needs to happen. | 16:41 |
| andreaf | sdague: ok - sounds like a good option - but that works only as long as the IP is actually assigned, which does not cover one of the cases you mentioned earlier | 16:41 |
| andreaf | sdague: maybe the server and some metadata | 16:41 |
| openstackrecheck | Console logs not available after 13:33s for gate-grenade-dsvm-multinode 147520,68,0248ff7 | 16:41 |
| andreaf | sdague: but I got the idea | 16:41 |
| sdague | right, there are details to work out. | 16:41 |
| sdague | so you could also pass it "strategy" or something | 16:42 |
| sdague | oh, I need to access via a floating ip, which I don't have | 16:42 |
| sdague | so do that when needed | 16:42 |
| openstackgerrit | Jordan Pittier proposed openstack/tempest: API/compute: create all servers through the common wrappers https://review.openstack.org/265343 | 16:42 |
| andreaf | afazekas: are you around? about https://review.openstack.org/#/c/173334/ | 16:44 |
| *** nicholasgoracke has quit IRC | 16:44 | |
| andreaf | sdague: do you have a review or some details about what you're trying to do - where you got stuck? | 16:45 |
| openstackgerrit | Jordan Pittier proposed openstack/tempest: Add a volume extend scenario https://review.openstack.org/249100 | 16:45 |
| sdague | andreaf: I don't have anything out there, the server_or_ip that was getting passed through 3 or 4 function layers meant that in scenarios there was never the server object when yuo needed it | 16:46 |
| andreaf | timothyb89: do you have a moment to talk about filtering for o-h? | 16:46 |
| sdague | I spent about 3 hours unwinding, then gave up | 16:46 |
| andreaf | sdague: ok, so we need the server object (or the ID at least), and the servers client, so we can get the console log when needed on failure | 16:47 |
| andreaf | sdague: what about have a Server object around the dict returned by the API, which includes the client itself, and extra metadata? | 16:48 |
| openstackrecheck | Console logs not available after 13:28s for gate-tempest-dsvm-full-ceph 265266,1,693b215 | 16:54 |
| *** mragupat has quit IRC | 16:59 | |
| *** mragupat has joined #openstack-qa | 17:00 | |
| sdague | andreaf: a server object is probably a good idea, I think the real question is what additional info is needed | 17:01 |
| *** erlon has joined #openstack-qa | 17:03 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/tempest: Remove ironic tests from tree https://review.openstack.org/265358 | 17:06 |
| *** devkulkarni has quit IRC | 17:06 | |
| *** mriedem_meeting is now known as mriedem | 17:07 | |
| *** devkulkarni has joined #openstack-qa | 17:07 | |
| openstackrecheck | Console logs not available after 13:23s for gate-grenade-dsvm 260587,5,a64334c | 17:08 |
| andreaf | sdague: ok, that goes in the direction of something like testresources - but we should probably looks for a small change first | 17:12 |
| *** jlanoux has quit IRC | 17:16 | |
| openstackrecheck | Console logs not available after 13:31s for gate-tempest-dsvm-large-ops 258180,9,a7cc9d1 | 17:22 |
| *** TuanNguyen has quit IRC | 17:22 | |
| *** mriedem is now known as mriedem_lunch | 17:23 | |
| *** TuanNguyen has joined #openstack-qa | 17:23 | |
| *** jdandrea_ has joined #openstack-qa | 17:24 | |
| *** e0ne has joined #openstack-qa | 17:25 | |
| openstackgerrit | Scott DAngelo proposed openstack-dev/grenade: Add copying new api-paste.ini to Cinder upgrade https://review.openstack.org/264995 | 17:26 |
| *** jdandrea has quit IRC | 17:27 | |
| *** superdan has quit IRC | 17:27 | |
| andreaf | mtreinish, sdague: I worked for a couple of weeks on this spec and related series - so I hope I can save at least part of it, or get ideas on how to morph it in something you may like better https://review.openstack.org/#/c/92804/ https://review.openstack.org/#/q/topic:bp/client-manager-refactor | 17:27 |
| *** dansmith has joined #openstack-qa | 17:27 | |
| *** TuanNguyen has quit IRC | 17:27 | |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| *** dansmith is now known as superdan | 17:31 | |
| *** nicholasgoracke has joined #openstack-qa | 17:31 | |
| *** lezbar has quit IRC | 17:34 | |
| *** salv-orlando has quit IRC | 17:35 | |
| openstackrecheck | Console logs not available after 13:27s for gate-tempest-dsvm-neutron-full 265279,1,2486cf8 | 17:35 |
| *** salv-orlando has joined #openstack-qa | 17:36 | |
| *** ifarkas has quit IRC | 17:36 | |
| *** TuanNguyen has joined #openstack-qa | 17:39 | |
| *** Swami has joined #openstack-qa | 17:44 | |
| *** jordanP has quit IRC | 17:45 | |
| *** sabeen2 has quit IRC | 17:47 | |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-neutron-large-ops 264819,1,245cae9 | 17:49 |
| openstackgerrit | Jim Rollenhagen proposed openstack/tempest: Remove ironic tests from tree https://review.openstack.org/265358 | 17:51 |
| *** ccneill has quit IRC | 17:53 | |
| *** ccneill has joined #openstack-qa | 17:54 | |
| *** nicholasgoracke has quit IRC | 17:56 | |
| openstackgerrit | Dan Nguyen proposed openstack/tempest: Adds auth_version property to dashboard config https://review.openstack.org/264354 | 18:00 |
| *** e0ne has quit IRC | 18:01 | |
| *** yamahata_ has joined #openstack-qa | 18:01 | |
| *** dims_ has joined #openstack-qa | 18:02 | |
| openstackrecheck | Console logs not available after 13:27s for gate-grenade-dsvm 260588,5,6725d91 | 18:02 |
| *** mragupat_ has joined #openstack-qa | 18:02 | |
| *** VanL_ has quit IRC | 18:04 | |
| *** yamahata has quit IRC | 18:04 | |
| *** dims has quit IRC | 18:05 | |
| *** VanL has joined #openstack-qa | 18:05 | |
| *** yamahata_ has quit IRC | 18:06 | |
| *** mragupat has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-dev/devstack: lib/cinder:configure tgt even if the package was purged https://review.openstack.org/264917 | 18:07 |
| *** shardy has quit IRC | 18:07 | |
| timothyb89 | andreaf: apologies, looks like I might have missed you, but I am available the rest of today and through next week | 18:09 |
| andreaf | timothyb89: I'm still here, only for a few minutes, it should be enough though :) | 18:10 |
| timothyb89 | ah, great! | 18:10 |
| *** SridharG has quit IRC | 18:10 | |
| *** TuanNguyen has quit IRC | 18:11 | |
| andreaf | timothyb89: the filtering I had in mind for o-h would basically apply to the top two graphs of pages 2, 3 and 4 | 18:12 |
| andreaf | timothyb89: when browsing from page 1 to 2 a filter is applied based on the metadata key which is used for grouping on page 1 (project by default) | 18:13 |
| andreaf | timothyb89: moving to page 3 and 4 that filter is dropped - as by design | 18:13 |
| andreaf | timothyb89: but I think it would be useful to | 18:13 |
| andreaf | 1) display whatever filter is currently applied | 18:14 |
| andreaf | 2) being able to switch filters on-off / add new ones based on metadata keys - and maybe other fields | 18:15 |
| andreaf | an example - let's say that I'm looking at periodic jobs after grouping based on pipeline on page1 | 18:15 |
| andreaf | I select a specific job that fails 10% of the time | 18:15 |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm 261785,2,d7e3fae | 18:16 |
| andreaf | when I'm on 3rd page the results I see are not specific to periodic anymore, unless the job name is unique for periodic pipeline | 18:16 |
| andreaf | so I may want to re-apply the pipeline filter | 18:16 |
| andreaf | or I may want to filter by some other fields to try and isolate the source of the failure | 18:17 |
| andreaf | e.g. for a job that fails sometime, does it fail only on gate or in periodic as well? or for a specific test, does it fails in specific setups only? | 18:18 |
| andreaf | timothyb89: does it make sense? ^^^^ | 18:18 |
| *** ccarmack1 has joined #openstack-qa | 18:18 | |
| *** dimtruck is now known as zz_dimtruck | 18:18 | |
| timothyb89 | andreaf: I believe so, losing information when moving to page 3 has been an issue pretty much since the start of the project | 18:19 |
| *** ccarmack has quit IRC | 18:19 | |
| *** jasonsb has quit IRC | 18:20 | |
| andreaf | timothyb89: it may be fine as default behaviour, but I think it would be useful to visually hint that, and to give the ability to turn filters on/off by pressing a control | 18:20 |
| *** TuanNguyen has joined #openstack-qa | 18:20 | |
| timothyb89 | andreaf: to clarify a bit, if I browse from http://status.openstack.org/openstack-health/#/g/build_queue/periodic-qa to http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-identity-v3-only-full-master , what results there would you like to be able to filter out? | 18:21 |
| andreaf | timothyb89: I think it would be ok to start with one filter, and we could switch to multiple ones later if needed - with multiple ones than it also becomes more complicated because then you can and or or them | 18:21 |
| andreaf | timothyb89: well, the periodic example maybe was not ideal, because the job name is periodic-*, so it only runs on build_queue periodic-qa | 18:23 |
| *** ccarmack has joined #openstack-qa | 18:23 | |
| timothyb89 | andreaf: ah, right, makes sense | 18:24 |
| andreaf | timothyb89: in that case losing the filter on build_queue does not make a difference | 18:24 |
| *** ccarmack1 has quit IRC | 18:24 | |
| andreaf | but if I go on the specific test, e.g. http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern, then the graph is not specific to periodic anymore | 18:24 |
| andreaf | I can see the graph on failures but I don't know whether this test is failing on gate as well or not | 18:25 |
| *** ccarmack1 has joined #openstack-qa | 18:26 | |
| timothyb89 | andreaf: ah, okay, that makes sense then, the tests-specific page definitely loses some information | 18:26 |
| andreaf | or perhaps it could be interesting to filter by job name | 18:27 |
| andreaf | or even add some group-by graph to show failures at test level or job level grouped by some metadata key | 18:27 |
| timothyb89 | andreaf: do you also have the issue on the 3rd (job) page? | 18:27 |
| *** ccarmack has quit IRC | 18:28 | |
| openstackrecheck | Console logs not available after 13:22s for gate-cinder-python27 265281,1,9c8317e | 18:29 |
| andreaf | timothyb89: in some cases yes | 18:30 |
| andreaf | for instance http://status.openstack.org/openstack-health/#/g/node_provider/hpcloud-b1 | 18:31 |
| *** yamahata_ has joined #openstack-qa | 18:31 | |
| andreaf | when I select a job name from there, the node_provider part is lost | 18:31 |
| timothyb89 | andreaf: oh, I see then, the job page you end up on no longer applies at all, okay | 18:32 |
| andreaf | timothyb89: it may be ok to lose the filter, it depends on what I'm looking for - my wish would be to be able to decide whether to keep the filter or not, and eventually being able to define new filters | 18:33 |
| timothyb89 | andreaf: I'm not sure if filtering would be best, or just creating an entirely new page that is (grouping key + job) instead of just (job) | 18:33 |
| timothyb89 | andreaf: and then adding some UI to the current job page to see what grouping key members it is contained in | 18:34 |
| *** ccarmack has joined #openstack-qa | 18:35 | |
| *** ccarmack1 has quit IRC | 18:35 | |
| *** mriedem_lunch is now known as mriedem | 18:35 | |
| andreaf | timothyb89: I think I see what you mean - yes perhaps showing the groups it belongs to is more useful | 18:35 |
| *** dwalleck has joined #openstack-qa | 18:36 | |
| andreaf | timothyb89: applying filters when searching for something is like trial and error, being able to see group information is more useful, e.g. in which groups did a certain failure happen | 18:36 |
| andreaf | timothyb89: something like kibana does, it shows which values of a certain metadata key are matched | 18:37 |
| andreaf | timothyb89: not sure how easy it is to get that out of our DB :) | 18:37 |
| timothyb89 | andreaf: hopefully, but a dedicated page for the filtered results would help as well I think | 18:37 |
| timothyb89 | andreaf: getting it out might be easy, doing it quickly on the other hand... :) | 18:38 |
| andreaf | timothyb89: cool - I hope this is enough ground for now to start, I really have to go now, family is claiming me for week-end :P | 18:38 |
| timothyb89 | andreaf: alright, thanks for the info, I will start investigating | 18:39 |
| timothyb89 | andreaf: have a nice weekend :) | 18:39 |
| andreaf | timothyb89: have a nice week-end, ping me on Monday if you want to chat further | 18:39 |
| timothyb89 | andreaf: will do, thanks! | 18:39 |
| *** nicholasgoracke has joined #openstack-qa | 18:40 | |
| *** zz_dimtruck is now known as dimtruck | 18:42 | |
| *** _nadya_ has joined #openstack-qa | 18:42 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-neutron-full 243159,13,324a509 | 18:43 |
| openstackgerrit | Ryan Hsu proposed openstack/tempest: Update image creation error msg to mention the correct image type https://review.openstack.org/253234 | 18:44 |
| *** ccneill has quit IRC | 18:45 | |
| *** ccneill has joined #openstack-qa | 18:49 | |
| *** ccarmack has quit IRC | 18:52 | |
| *** ccarmack has joined #openstack-qa | 18:52 | |
| *** sabeen has joined #openstack-qa | 18:55 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full 260586,5,4f53857 | 18:56 |
| *** sabeen3 has joined #openstack-qa | 18:58 | |
| *** sabeen has quit IRC | 19:00 | |
| *** cdent has quit IRC | 19:00 | |
| *** lucasagomes is now known as lucas-dinner | 19:08 | |
| mriedem | what in blue blazes?! http://logs.openstack.org/86/260186/8/check/gate-manila-tempest-dsvm-neutron/6aeaae6/logs/devstacklog.txt.gz#_2016-01-08_18_46_43_870 | 19:08 |
| *** _nadya_ has quit IRC | 19:09 | |
| *** dkranz has quit IRC | 19:09 | |
| openstackrecheck | Console logs not available after 13:26s for gate-grenade-dsvm 263317,3,eb5bf05 | 19:10 |
| *** jasonsb has joined #openstack-qa | 19:11 | |
| *** slowrie has quit IRC | 19:12 | |
| mriedem | i think https://review.openstack.org/#/c/257744/ broke the manila dsvm jobs | 19:12 |
| *** slowrie has joined #openstack-qa | 19:13 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300 https://review.openstack.org/265410 | 19:17 |
| openstack | bug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,New] https://launchpad.net/bugs/1532300 | 19:17 |
| mriedem | mtreinish: sdague: ^ | 19:17 |
| *** aopincar has joined #openstack-qa | 19:17 | |
| *** nicholasgoracke has quit IRC | 19:21 | |
| *** nicholasgoracke has joined #openstack-qa | 19:21 | |
| mtreinish | mriedem: manila | 19:23 |
| mtreinish | s tests depend on having horizon? | 19:23 |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full-ceph 263567,1,49d3962 | 19:23 |
| mriedem | mtreinish: they have a horizon plugin it looks like | 19:25 |
| mriedem | https://github.com/openstack/manila/blob/master/devstack/plugin.sh#L206 | 19:25 |
| mtreinish | ah, ok | 19:25 |
| mriedem | https://github.com/openstack/manila-ui | 19:25 |
| *** nicholasgoracke has quit IRC | 19:25 | |
| mriedem | i don't understand the devstack change and what it means for whatever the manila devstack plugin needs to do | 19:26 |
| anteaya | anyone who knows how tempest plugin test setups look when properly done want to take a peek at https://review.openstack.org/#/c/265311/1 please? | 19:27 |
| mtreinish | mriedem: my guess is that the manila devstack plugin had some hard coded assumption on the previous structure | 19:30 |
| mtreinish | at least thats my guess without looking at anything in manila | 19:30 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300 https://review.openstack.org/265410 | 19:32 |
| openstack | bug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,Confirmed] https://launchpad.net/bugs/1532300 | 19:32 |
| openstackgerrit | Scott DAngelo proposed openstack-dev/grenade: Add copying new api-paste.ini to Cinder upgrade https://review.openstack.org/264995 | 19:32 |
| *** castulo has quit IRC | 19:37 | |
| openstackrecheck | Console logs not available after 13:27s for gate-cinder-pep8 265351,1,51134b4 | 19:37 |
| openstackgerrit | Scott DAngelo proposed openstack-dev/grenade: Add copying new api-paste.ini to Cinder upgrade https://review.openstack.org/264995 | 19:37 |
| *** openstackrecheck has quit IRC | 19:37 | |
| *** openstackrecheck has joined #openstack-qa | 19:37 | |
| mtreinish | EmilienM: we should ask my mriedem, he looks at the bdm code a lot | 19:39 |
| mtreinish | he's not in puppet-openstack | 19:39 |
| EmilienM | yeah, I need some help, let me give a bit of context here before : | 19:39 |
| mtreinish | mriedem: http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/console.html#_2016-01-08_18_27_48_222 | 19:39 |
| EmilienM | nova api: http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/nova/nova-api.txt.gz#_2016-01-08_18_27_29_267 | 19:40 |
| mtreinish | EmilienM: one thing is don't you use os-loganalyze for the log files? The lack of level filtering is a bit annoying | 19:45 |
| mriedem | so boot from volume fails b/c the volume passed in isn't found in the cinder side | 19:45 |
| EmilienM | mtreinish: sure. I did not know about that thing, will look at it | 19:46 |
| mriedem | i see it in cinder api though http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/cinder/cinder-api.txt.gz#_2016-01-08_18_27_25_516 | 19:46 |
| EmilienM | mtreinish: how can I do that? | 19:46 |
| mtreinish | mriedem: thats what it looks like, but cinder knows its there it returns a bunch of gets on it | 19:46 |
| *** eil397 has joined #openstack-qa | 19:47 | |
| mtreinish | EmilienM: I'm not sure on the details of how it's setup, maybe clarkb or jhesketh can help with that | 19:47 |
| mriedem | mtreinish: yeah | 19:47 |
| mtreinish | EmilienM: but having the ability to filter the logs by level is really useful :) | 19:47 |
| EmilienM | mtreinish: adding it on my todo | 19:47 |
| clarkb | aroo? | 19:47 |
| mtreinish | clarkb: setting up osloganalyze for the puppet tempest job logs | 19:48 |
| mtreinish | mriedem: that's where I got lost | 19:49 |
| clarkb | the simplest thing would be to use the same log format that devstack uses | 19:49 |
| clarkb | then, I htink you have to whitelist the files if they have different names in osloganalyze itself | 19:50 |
| EmilienM | clarkb: ok so change log format | 19:50 |
| mriedem | mtreinish: yeah i'm lost too | 19:50 |
| mriedem | and we don't have the request id | 19:50 |
| mtreinish | mriedem: right, I tried to use the timestamp to correlate, but I didn't see any 404s from cinder | 19:51 |
| EmilienM | clarkb: is it documented somewhere? Or should I just dig in devstack and find the log format? | 19:52 |
| clarkb | EmilienM: that I don't know | 19:52 |
| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm 234874,25,18d8fe3 | 19:52 |
| clarkb | I know rbradfor has been looking at making them more consistent by default | 19:52 |
| clarkb | that may be another approach | 19:52 |
| mtreinish | clarkb: heh, devstack differs from the default in a very minor manner. It uses user name and tenant name instead of ids | 19:53 |
| EmilienM | clarkb: is it logging_context_format_string = %(asctime)s.%(msecs)03d %(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s] %(instance)s%(message)s ? | 19:53 |
| mtreinish | other than that it's pretty much the defaults from oslo log | 19:53 |
| mtreinish | EmilienM: yeah, htat looks right | 19:53 |
| mriedem | mtreinish: hmm | 19:53 |
| EmilienM | mtreinish: so if I enable that everywhere in our jobs, what should I do more? | 19:53 |
| mriedem | there is a delete request on that volume right around when nova failed http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/cinder/cinder-api.txt.gz#_2016-01-08_18_27_29_285 | 19:53 |
| mriedem | that could just be tempest cleaning up | 19:54 |
| mtreinish | mriedem: yeah that's ~2 min after nova returns a 400, I assume it's just tempest doing clean up | 19:54 |
| *** [1]Thelo has joined #openstack-qa | 19:55 | |
| mriedem | 2 min? | 19:55 |
| mriedem | i'm seeing ~400ms | 19:55 |
| mriedem | nova fails at 2016-01-08 18:27:29.267 | 19:55 |
| mriedem | cinder delete is at 2016-01-08 18:27:29.285 | 19:55 |
| mriedem | not even 400 ms | 19:56 |
| mtreinish | oh, you're right, the numbers are jumping around on me | 19:56 |
| mriedem | in n-cpu we trace the calls to the cinder api (the request and response) | 19:57 |
| mriedem | i wonder why we don't have those logs in n-api | 19:57 |
| *** Thelo has quit IRC | 19:57 | |
| *** [1]Thelo is now known as Thelo | 19:57 | |
| EmilienM | the weird things is it happens only on trusty | 19:59 |
| mriedem | the integrated gate jobs are all on trusty | 19:59 |
| *** _nadya_ has joined #openstack-qa | 19:59 | |
| mtreinish | mriedem: these are the puppet jobs so its trusty or centos (iirc) | 19:59 |
| EmilienM | yeah | 19:59 |
| *** dkranz has joined #openstack-qa | 20:04 | |
| EmilienM | mtreinish: do you think that's enough https://review.openstack.org/265426 ? | 20:04 |
| mtreinish | EmilienM: maybe, I'm not sure that is the issue though | 20:05 |
| openstackrecheck | Console logs not available after 13:28s for gate-tempest-dsvm-full 256273,49,0bad3b8 | 20:05 |
| clarkb | you will alos need to whitelist and filenames in osloganalyze that don't match the devstack file names | 20:06 |
| EmilienM | ok looking now | 20:07 |
| EmilienM | mtreinish, mriedem: fyi, I could reproduce the bug in centos7 too. So we might have a bug in liberty (and not only in packaging). | 20:08 |
| mriedem | hrm, i have no idea really, this doesn't make sense | 20:10 |
| mriedem | w/o logging the request id though that nova is making to cinder via cinderclient, it's hard to tell what's going on | 20:10 |
| mriedem | i'm trying to sort out why n-cpu in normal gate jobs has the req/res traced from cinderclient | 20:10 |
| mriedem | but n-api doesn't here | 20:10 |
| EmilienM | clarkb: our log files format are already good I think http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/nova/ | 20:11 |
| EmilienM | .txt.gz | 20:12 |
| clarkb | its the name itself | 20:12 |
| clarkb | screen-n-cpu.txt.gz etc | 20:12 |
| EmilienM | oh.. | 20:12 |
| EmilienM | in there: https://github.com/openstack-infra/os-loganalyze/blob/082d0b942a4c402caddcdd6d21bb1ed5524b3615/os_loganalyze/filter.py | 20:12 |
| clarkb | ya there is the SUPPORTS_SEV rege | 20:13 |
| EmilienM | ok thx | 20:13 |
| EmilienM | anteaya: ^ | 20:14 |
| anteaya | EmilienM: thanks | 20:14 |
| mriedem | EmilienM: mtreinish: for example, see how in this job, n-cpu is tracking request/response from cinderclient | 20:15 |
| mriedem | http://logs.openstack.org/99/263399/1/check/gate-tempest-dsvm-full/0bc2f8e/logs/screen-n-cpu.txt.gz#_2016-01-04_22_19_04_778 | 20:15 |
| mriedem | i don't see that same thing in the puppet jobs | 20:15 |
| *** ccarmack has quit IRC | 20:16 | |
| mriedem | same in n-api http://logs.openstack.org/99/263399/1/check/gate-tempest-dsvm-full/0bc2f8e/logs/screen-n-api.txt.gz#_2016-01-04_22_18_47_936 | 20:16 |
| mtreinish | mriedem: is there a config option to turn that on? | 20:17 |
| EmilienM | mriedem: have you something wrong in config? | 20:17 |
| *** ccarmack has joined #openstack-qa | 20:18 | |
| mriedem | should be debug=true in keystone.conf from what i can tell | 20:18 |
| mriedem | keystoneclient defaults to logging the req/response https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/session.py#L222 | 20:18 |
| EmilienM | http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/etc/nova/nova.conf.txt.gz | 20:18 |
| EmilienM | nova.conf ^ | 20:18 |
| mriedem | that's nova.conf | 20:18 |
| mriedem | keystone debug has to be on | 20:19 |
| EmilienM | mriedem: http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/etc/keystone/keystone.conf.txt.gz | 20:19 |
| EmilienM | debug = True | 20:19 |
| EmilienM | it's on | 20:19 |
| mriedem | yeah | 20:19 |
| mriedem | hmm | 20:19 |
| openstackrecheck | Console logs not available after 13:24s for experimental-tempest-dsvm-multinode-full-dibtest 262715,1,7707a20 | 20:19 |
| mtreinish | SpamapS: so I see that the os-performance-tools has landed and is a thing now, what's the next step there? | 20:20 |
| mriedem | bknudson_: stevemar: lbragstad: can you help us figure out why keystoneclient isn't logging cinderclient requests and responses in the puppet jobs? keystone.conf has debug=True | 20:22 |
| mriedem | http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/etc/keystone/keystone.conf.txt.gz | 20:22 |
| bknudson_ | keystoneclient doesn't use debug=True | 20:22 |
| bknudson_ | from keystone.conf | 20:23 |
| mriedem | https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/session.py#L171 | 20:23 |
| mriedem | i was looking at that | 20:23 |
| bknudson_ | that logger is from line 39 | 20:23 |
| bknudson_ | so it'll be like keystoneclient.session | 20:24 |
| bknudson_ | so you could enable logging for it by setting the keystoneclient logger to debug | 20:24 |
| *** aopincar has quit IRC | 20:24 | |
| EmilienM | clarkb: I'm not going to add more REGEX but just rename our log files. Will be easier for everyone | 20:25 |
| clarkb | EmilienM: eh, I don't mind if you add to the regex | 20:25 |
| clarkb | shouldn't be too hard | 20:25 |
| clarkb | you could add a complete separate |'d cluase | 20:25 |
| EmilienM | clarkb: the thing is I like the types of logs: http://logs.openstack.org/05/261105/5/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-trusty/1d60061/logs/cinder/ | 20:25 |
| EmilienM | having them as a list with description is super useful | 20:25 |
| *** ccarmack has left #openstack-qa | 20:26 | |
| EmilienM | clarkb: yeah, the REGEX is not hard to build. It's just I want to have logs listed like it's done in devstack jobs, the the nice description | 20:26 |
| clarkb | gotcha | 20:26 |
| EmilienM | like screen-c-api.txt.gz: cinder-api | 20:27 |
| EmilienM | that's very cool ^ | 20:27 |
| *** dwalleck has quit IRC | 20:28 | |
| EmilienM | bknudson_: how can I enable that? do you have an example of config? | 20:29 |
| mriedem | bknudson_: any idea where that is configured in a normal dsvm gate job? http://logs.openstack.org/99/263399/1/check/gate-tempest-dsvm-full/0bc2f8e/logs/ | 20:29 |
| bknudson_ | EmilienM: it's not a config file, it's the application needs to set the logging configuration for the library | 20:29 |
| bknudson_ | what's the application in this case? | 20:30 |
| *** gfidente has quit IRC | 20:31 | |
| EmilienM | I would say nova | 20:31 |
| bknudson_ | each application sets up logging for itself, and keystoneclient will pick up whatever the application configures | 20:31 |
| bknudson_ | there's lots of nova processes | 20:31 |
| SpamapS | mtreinish: need to land the changes in devstack-gate and project-config. But I also have come to realize our trending data will be missing some things unfortunately | 20:31 |
| bknudson_ | typically the openstack processes use oslo.logging which has default settings for the logs... let me see if I can find the setting. | 20:32 |
| bknudson_ | see http://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/_options.py#n21 | 20:32 |
| *** salv-orlando has quit IRC | 20:32 | |
| openstackrecheck | Console logs not available after 13:26s for gate-tempest-dsvm-neutron-full 193135,70,af038ca | 20:32 |
| mtreinish | SpamapS: well, we've got to start somewhere | 20:33 |
| *** salv-orlando has joined #openstack-qa | 20:33 | |
| bknudson_ | if you don't want to use the default config you can provide a logging.conf file. | 20:33 |
| mriedem | nova's default log levels are here https://github.com/openstack/nova/blob/master/nova/config.py#L36 | 20:33 |
| bknudson_ | I wonder what it does for the loggers that aren't set there? seems like it would be debug if you enable debug in your .conf | 20:34 |
| *** Daviey has joined #openstack-qa | 20:39 | |
| mriedem | it's actually logging as keystoneauth.session | 20:40 |
| bknudson_ | sneaky | 20:40 |
| mriedem | oh sorry, no, that's only for tokens | 20:41 |
| bknudson_ | we've got the same code in keystoneauth -- http://git.openstack.org/cgit/openstack/keystoneauth/tree/keystoneauth1/session.py#n189 | 20:41 |
| mriedem | the cinder requests are going through keystoneclient.session since that's what nova uses to create the cinderclient in nova | 20:41 |
| mriedem | anyway, i have no clue how this works in the regular dsvm jobs | 20:41 |
| mriedem | maybe nova is using something that's setting default log levels for keystoneclient? | 20:42 |
| SpamapS | mtreinish: agreed. My worry is that we don't keep track of the number of tests run, so we have no idea if a jump is due to code or test changes. | 20:43 |
| bknudson_ | it's all global variables so something could mess with the log settings. | 20:43 |
| mtreinish | SpamapS: well that data is in subunit2sql is there a way we can correlate the 2 stores? | 20:43 |
| SpamapS | mtreinish: no | 20:44 |
| mtreinish | hmm, :/ | 20:45 |
| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm 264854,3,87edd01 | 20:46 |
| SpamapS | mtreinish: we could grab it from the stream and add it as a gauge actually | 20:46 |
| SpamapS | mtreinish: that would allow us to superimpose and visually see when it is added/removed tests | 20:47 |
| mtreinish | SpamapS: yeah, that works you actually could probably even use get_results() from here: https://github.com/openstack-infra/puppet-subunit2sql/blob/master/files/subunit-gearman-worker.py#L182 | 20:50 |
| mtreinish | that'll give you a total count iirc the data structure correctly | 20:51 |
| mtreinish | err I guess not, that's why we have: https://github.com/openstack-infra/subunit2sql/blob/master/subunit2sql/shell.py#L110-L119 | 20:52 |
| openstackrecheck | Console logs not available after 13:24s for gate-grenade-dsvm 264259,5,7ff027a | 20:59 |
| *** rfolco has quit IRC | 21:01 | |
| *** _nadya_ has quit IRC | 21:08 | |
| *** aopincar has joined #openstack-qa | 21:09 | |
| *** alyson_ has quit IRC | 21:12 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-neutron-dvr 202351,5,3811a8c | 21:13 |
| mtreinish | regXboi: if you get a sec: https://review.openstack.org/#/c/264984/ should address a couple of issues we hit in the prod server yesterday | 21:13 |
| *** aopincar has quit IRC | 21:13 | |
| mriedem | mtreinish: holy cow, cracked 50% http://status.openstack.org/elastic-recheck/data/uncategorized.html | 21:16 |
| mtreinish | \o/ nice | 21:17 |
| dims_ | yay :) | 21:17 |
| *** edmondsw has quit IRC | 21:17 | |
| regXboi | mtreinish - will be a little bit - fighting an ipv6 fire in the neutron channel right now | 21:17 |
| mtreinish | regXboi: sure, no worries | 21:18 |
| mtreinish | regXboi: I would just throw sc68cal on it, that has the right keywords to make it his problem :) | 21:18 |
| regXboi | mtreinish: he's involved | 21:19 |
| regXboi | but since I lit the fuse, I sorta need to hang around | 21:20 |
| *** castulo has joined #openstack-qa | 21:21 | |
| sc68cal | mtreinish: already on it. | 21:22 |
| *** nicholasgoracke has joined #openstack-qa | 21:22 | |
| openstackrecheck | Console logs not available after 13:26s for gate-tempest-dsvm-neutron-full 249100,6,86b06ed | 21:26 |
| *** nicholasgoracke has quit IRC | 21:27 | |
| *** castulo has quit IRC | 21:27 | |
| *** castulo has joined #openstack-qa | 21:28 | |
| timothyb89 | mtreinish: it looks like you're probably running into an nvd3 bug on https://review.openstack.org/#/c/265084/ | 21:29 |
| mtreinish | timothyb89: yeah that's what I was thinking | 21:30 |
| *** devkulkarni has left #openstack-qa | 21:38 | |
| regXboi | mtreinish: levers pulled | 21:39 |
| openstackrecheck | Console logs not available after 13:25s for gate-nova-pep8 249016,13,d8cb94c | 21:40 |
| mtreinish | regXboi: cool, thanks | 21:40 |
| regXboi | mtreinish: I wasn't going to hold up on the typo in the commit message :) | 21:40 |
| mtreinish | regXboi: heh, I'd never land any code if people held me to typo free commit messages | 21:45 |
| sc68cal | sdague: so - the connection timeout of 10 seconds didn't help. It appears that the connection actually succeeds - we get the dropbear ssh banner, but then it sits there for 5 minutes | 21:49 |
| sc68cal | http://logs.openstack.org/35/187235/12/experimental/gate-grenade-dsvm-neutron-multinode/984e651/logs/grenade.sh.txt.gz#_2016-01-08_20_13_40_040 | 21:49 |
| sc68cal | the only thing i can think of is I/O on the instance is messed up? like stuck in disk | 21:49 |
| openstackgerrit | Merged openstack/openstack-health: Improve pandas usage in test_runs_aggregator https://review.openstack.org/264984 | 21:52 |
| openstackgerrit | Merged openstack/openstack-health: Remove leftover sample data https://review.openstack.org/265083 | 21:53 |
| timothyb89 | mtreinish: so, it looks like the NaN values are actually what's causing the tooltips to go away | 21:53 |
| openstackrecheck | Console logs not available after 13:37s for gate-nova-pep8 259254,11,20b03c5 | 21:53 |
| timothyb89 | mtreinish: throwing a quick isNaN call into the processing loop to hide those entries works well, and cleans up all the console errors too | 21:55 |
| mtreinish | timothyb89: ok, cool | 21:56 |
| mtreinish | I'll respin it in a min | 21:56 |
| openstackrecheck | Console logs not available after 13:22s for gate-nova-pep8 249018,18,6b2cdc3 | 22:07 |
| *** mriedem has quit IRC | 22:07 | |
| *** castulo has quit IRC | 22:07 | |
| *** salv-orl_ has joined #openstack-qa | 22:09 | |
| *** nicholasgoracke has joined #openstack-qa | 22:11 | |
| *** salv-orlando has quit IRC | 22:12 | |
| *** salv-orl_ has quit IRC | 22:12 | |
| *** salv-orlando has joined #openstack-qa | 22:12 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Switch to a scatter plot for test runtime https://review.openstack.org/265084 | 22:16 |
| *** castulo has joined #openstack-qa | 22:18 | |
| *** castulo has quit IRC | 22:19 | |
| *** castulo has joined #openstack-qa | 22:19 | |
| openstackrecheck | Console logs not available after 13:22s for gate-nova-pep8 262586,11,f19e3ca | 22:20 |
| *** leakypipes has quit IRC | 22:20 | |
| *** rossella_s has quit IRC | 22:22 | |
| mtreinish | timothyb89: ^^^ that should do it | 22:22 |
| *** rossella_s has joined #openstack-qa | 22:22 | |
| timothyb89 | mtreinish: looks good, although you might have a couple of lint errors | 22:23 |
| timothyb89 | extra blank line on line 35? | 22:24 |
| mtreinish | timothyb89: I always have lint errors | 22:24 |
| timothyb89 | I know the feeling :) | 22:24 |
| *** castulo has quit IRC | 22:25 | |
| *** castulo has joined #openstack-qa | 22:25 | |
| *** dkranz has quit IRC | 22:29 | |
| *** castulo has quit IRC | 22:30 | |
| *** castulo has joined #openstack-qa | 22:30 | |
| *** nicholasgoracke has quit IRC | 22:30 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Switch to a scatter plot for test runtime https://review.openstack.org/265084 | 22:33 |
| mtreinish | timothyb89: ^^^ now it passes lint. Although I'm not sure whether it's an improvement or not | 22:33 |
| openstackrecheck | Console logs not available after 13:22s for gate-nova-pep8 262583,7,6269c06 | 22:34 |
| mtreinish | the patch in general that is | 22:34 |
| *** erlon has quit IRC | 22:36 | |
| timothyb89 | mtreinish: well, the NaN fix is a huge improvement if nothing else | 22:37 |
| timothyb89 | and we need a scatterplot to do that | 22:37 |
| timothyb89 | mtreinish: that should also help performance a ton, when we don't have thousands of exceptions being logged every render | 22:38 |
| *** castulo has quit IRC | 22:39 | |
| *** castulo has joined #openstack-qa | 22:40 | |
| *** castulo has quit IRC | 22:40 | |
| timothyb89 | mtreinish: I also looked into a way to combine scatter and line charts... unfortunately that would need a totally custom nvd3 chart :/ | 22:41 |
| Thelo | Hi all, When I run devstack and configuring it to get nova/cinder from a special commit (not merged yet), it takes a lot of time and timeout at the end. Is there anyway to improve that or increase the timeout? | 22:46 |
| openstackrecheck | Console logs not available after 13:24s for gate-nova-pep8 262584,7,c0e7ffb | 22:47 |
| *** regXboi has quit IRC | 22:51 | |
| *** mragupat_ has quit IRC | 22:55 | |
| openstackrecheck | Console logs not available after 13:23s for gate-grenade-dsvm-multinode 265415,1,0ef396c | 23:01 |
| mtreinish | timothyb89: yeah, I need to do some reading on d3 proper and just do it manually I think | 23:09 |
| mtreinish | if we could keep the mean as a line that would likely be for the best | 23:09 |
| mtreinish | nvd3 does have a scatter with line, but it's only a straight line | 23:09 |
| timothyb89 | mtreinish: I agree, although manual d3 can be a deep rabbit hole to enter :) | 23:10 |
| timothyb89 | yeah, that scatter+line seems pretty useless imo :/ | 23:10 |
| mtreinish | yeah, that's the feeling I got when I briefly look at things before. Well we can cross that bridge when the time comes | 23:11 |
| mtreinish | for right now I guess just moving to the scatter is a decent step | 23:11 |
| *** dimtruck is now known as zz_dimtruck | 23:12 | |
| openstackrecheck | Console logs not available after 13:22s for gate-nova-pep8 262585,10,68a2912 | 23:14 |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/262731 failed because of: gate-tempest-dsvm-keystone-eventlet-full: https://bugs.launchpad.net/bugs/1509004 | 23:16 |
| openstack | Launchpad bug 1509004 in neutron ""test_dualnet_dhcp6_stateless_from_os" failures seen in the gate" [Medium,Confirmed] | 23:16 |
| *** pglass has quit IRC | 23:17 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/devstack: Fix SIGITIN hang with apt-get https://review.openstack.org/265053 | 23:18 |
| *** vgridnev has quit IRC | 23:22 | |
| openstackgerrit | Tim Buckley proposed openstack/stackviz: Add axis to the timeline viewport. https://review.openstack.org/265473 | 23:23 |
| *** jasonsb has quit IRC | 23:44 | |
| *** mmedvede has joined #openstack-qa | 23:49 | |
| openstackrecheck | Console logs not available after 13:27s for gate-swift-pep8 265461,1,25b5ec4 | 23:55 |
| *** hemna is now known as hemnafk | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!