| *** armax has quit IRC | 00:08 | |
| *** markvoelker has joined #openstack-qa | 00:28 | |
| *** zz_dimtruck is now known as dimtruck | 00:34 | |
| *** hieulq_ has joined #openstack-qa | 00:39 | |
| *** dimtruck is now known as zz_dimtruck | 00:44 | |
| *** edmondsw has joined #openstack-qa | 00:54 | |
| openstackgerrit | zhufl proposed openstack/tempest: Move dscv and ca_certs to config section service_clients https://review.openstack.org/349749 | 00:56 |
|---|---|---|
| *** mriedem has quit IRC | 01:07 | |
| *** karthikp_ has quit IRC | 01:09 | |
| *** yangyapeng has joined #openstack-qa | 01:17 | |
| *** hichihara has joined #openstack-qa | 01:19 | |
| *** Deng has joined #openstack-qa | 01:21 | |
| *** hoangcx has joined #openstack-qa | 01:23 | |
| *** dkranz has quit IRC | 01:31 | |
| *** EinstCrazy has joined #openstack-qa | 01:41 | |
| *** fnaval has joined #openstack-qa | 01:48 | |
| *** gyx has joined #openstack-qa | 01:52 | |
| *** fnaval_ has joined #openstack-qa | 01:55 | |
| gyx | I have some patches to be reviewed | 01:57 |
| *** fnaval has quit IRC | 01:58 | |
| gyx | If anyone have time,can you help me to review them ? | 01:59 |
| gyx | :) | 01:59 |
| gyx | https://review.openstack.org/#/c/351667/ | 02:00 |
| *** yangyape_ has joined #openstack-qa | 02:00 | |
| gyx | https://review.openstack.org/#/c/351157/ | 02:00 |
| gyx | https://review.openstack.org/#/c/349411/ | 02:01 |
| *** zz_dimtruck is now known as dimtruck | 02:03 | |
| *** yangyapeng has quit IRC | 02:04 | |
| *** bswartz has quit IRC | 02:30 | |
| *** yangyapeng has joined #openstack-qa | 02:36 | |
| *** markvoelker has quit IRC | 02:36 | |
| *** yangyape_ has quit IRC | 02:39 | |
| *** gyx has quit IRC | 02:40 | |
| *** gouthamr has quit IRC | 02:55 | |
| masayukig | gmann: can we remove XML tests for swift from tempest? I think we can't do that but there's no document for it. | 03:02 |
| *** gyx has joined #openstack-qa | 03:06 | |
| openstackgerrit | guo yunxian proposed openstack/tempest: Add update object testcase https://review.openstack.org/352197 | 03:08 |
| *** armax has joined #openstack-qa | 03:08 | |
| *** armax has quit IRC | 03:13 | |
| *** bswartz has joined #openstack-qa | 03:17 | |
| gmann | masayukig: right , we should nto remove as long as swift support those | 03:25 |
| gmann | masayukig: and swift do support format=xml | 03:25 |
| masayukig | gmann : ok, I'm correct. I was reviewing https://review.openstack.org/#/c/351157/ | 03:26 |
| gmann | masayukig: but you should -1 :) | 03:28 |
| *** hoangcx has left #openstack-qa | 03:29 | |
| masayukig | gmann : done :) | 03:33 |
| *** shausy has joined #openstack-qa | 03:34 | |
| *** dave-mccowan has quit IRC | 03:47 | |
| *** jesusaur has joined #openstack-qa | 03:58 | |
| *** binbincong has joined #openstack-qa | 04:13 | |
| *** shausy has quit IRC | 04:15 | |
| *** shausy has joined #openstack-qa | 04:20 | |
| gyx | masayukig: gmann: Thank you for review my patch.As you said swift do support format=xml, then I will abondon this patch | 04:21 |
| *** Ravikiran_K has joined #openstack-qa | 04:29 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/tempest: Fix manager->get_auth_provider interface https://review.openstack.org/352214 | 04:34 |
| *** chandankumar has joined #openstack-qa | 04:36 | |
| *** markvoelker has joined #openstack-qa | 04:37 | |
| *** markvoelker has quit IRC | 04:42 | |
| *** shausy has quit IRC | 04:50 | |
| *** shausy has joined #openstack-qa | 04:50 | |
| *** sridharg has joined #openstack-qa | 04:52 | |
| *** gamado has joined #openstack-qa | 05:00 | |
| *** pcaruana has quit IRC | 05:03 | |
| *** dimtruck is now known as zz_dimtruck | 05:08 | |
| *** gvrangan has joined #openstack-qa | 05:13 | |
| openstackrecheck | Console logs not available after 13:25s for gate-cross-cinder-python27-db-ubuntu-xenial 350193,1,99a09b3 | 05:15 |
| openstackgerrit | Jianghua Wang proposed openstack-infra/devstack-gate: xenapi: Skip multi-host setup for nova network setup https://review.openstack.org/352222 | 05:16 |
| *** gcheresh has joined #openstack-qa | 05:17 | |
| *** hichihara has quit IRC | 05:27 | |
| *** pgadiya has joined #openstack-qa | 05:35 | |
| *** _nadya_ has joined #openstack-qa | 05:52 | |
| *** yamahata has joined #openstack-qa | 05:53 | |
| *** Lord_Phoenix has joined #openstack-qa | 05:56 | |
| openstackrecheck | Console logs not available after 13:24s for keystone-coverage-db-ubuntu-xenial 351988,2,fddd658 | 06:08 |
| openstackrecheck | openstack/requirements change: https://review.openstack.org/350866 failed gate-cross-keystone-python27-db-ubuntu-xenial in the gate queue with an unrecognized error | 06:12 |
| *** vsaienko1 has quit IRC | 06:23 | |
| *** _nadya_ has quit IRC | 06:23 | |
| *** edannon has joined #openstack-qa | 06:26 | |
| *** markvoelker has joined #openstack-qa | 06:38 | |
| *** pcaruana has joined #openstack-qa | 06:38 | |
| *** Poornima has joined #openstack-qa | 06:39 | |
| *** rcernin has joined #openstack-qa | 06:41 | |
| *** markvoelker has quit IRC | 06:42 | |
| *** tesseract- has joined #openstack-qa | 06:53 | |
| *** gyx has quit IRC | 06:55 | |
| gmann | masayukig: andreaf oomichi mkoderer mtreinish : can you review this one - https://review.openstack.org/#/c/352214/1 | 06:56 |
| gmann | masayukig: andreaf oomichi mkoderer mtreinish manager->get_auth_provider got broken while moving service_clients in lib and congress and other plugin failing due to that | 06:56 |
| *** _nadya_ has joined #openstack-qa | 07:02 | |
| *** ifarkas_afk is now known as ifarkas | 07:03 | |
| *** _nadya_ has quit IRC | 07:04 | |
| *** _nadya_ has joined #openstack-qa | 07:05 | |
| *** _nadya_ has quit IRC | 07:10 | |
| *** jpena|off is now known as jpena | 07:11 | |
| *** gyx has joined #openstack-qa | 07:12 | |
| *** yamamoto has quit IRC | 07:14 | |
| *** _nadya_ has joined #openstack-qa | 07:18 | |
| *** _nadya_ has quit IRC | 07:18 | |
| masayukig | gmann : +2 | 07:22 |
| *** jlanoux has joined #openstack-qa | 07:23 | |
| *** gyx has quit IRC | 07:23 | |
| *** _nadya_ has joined #openstack-qa | 07:31 | |
| *** gvrangan has quit IRC | 07:32 | |
| openstackgerrit | huan proposed openstack-dev/devstack: Install conntrack in XenServer Dom0 https://review.openstack.org/352256 | 07:33 |
| *** Serlex has joined #openstack-qa | 07:35 | |
| *** e0ne has joined #openstack-qa | 07:36 | |
| andreaf | gmann: +A | 07:38 |
| andreaf | gmann: nice catch, sorry about that | 07:38 |
| *** gyx has joined #openstack-qa | 07:38 | |
| andreaf | I checked the plugins we run in the puppet jobs but that wasm | 07:39 |
| andreaf | wasn't enough :( | 07:39 |
| *** gvrangan has joined #openstack-qa | 07:39 | |
| andreaf | gmann: if you can have a look https://review.openstack.org/#/c/347756/ after this I can start sending an email to the ML and plugins can start to switch to the stable clients modul | 07:41 |
| andreaf | module | 07:41 |
| *** yaume has joined #openstack-qa | 07:44 | |
| *** nradojevic has joined #openstack-qa | 07:45 | |
| *** yaume has quit IRC | 07:47 | |
| gmann | andreaf: np, those things are very hard to catch till we get gate failure. | 07:50 |
| *** e0ne has quit IRC | 07:51 | |
| gmann | andreaf: sure i will check that, yea it will be nice if plugin can start using that. I can ask my colleague to do for congress after you mail | 07:51 |
| *** rossella_s has quit IRC | 07:53 | |
| nradojevic | Hi guys, I've got a general question concerning the accounts file used for tempest. Can someone explain to me how to use this file? Do I have to create the users before I run the tests? I created some users in a new domain, added these ones in an accounts file, started a tempest run and now the majority of tests failed due to Invalid Credentials. The documentation for Tempest didn't help me so far. | 07:55 |
| lennyb | gmann: hi, pls review https://review.openstack.org/#/c/335447/ . It's for SRIOV Migration . Thanks | 07:55 |
| gcheresh | andreaf: hi, could you please review https://review.openstack.org/#/c/344193/2/ | 07:59 |
| *** dtantsur|afk is now known as dtantsur | 08:02 | |
| *** yfried has joined #openstack-qa | 08:02 | |
| *** ramishra_ has joined #openstack-qa | 08:02 | |
| *** ramishra has quit IRC | 08:05 | |
| *** abehl has joined #openstack-qa | 08:19 | |
| andreaf | nradojevic: hi | 08:37 |
| *** rossella_s has joined #openstack-qa | 08:37 | |
| andreaf | there are two ways you can get credentials for your tests | 08:37 |
| andreaf | nradojevic: you can either have them created on the fly for you - if you use the dynamic creds provider | 08:38 |
| andreaf | nradojevic: or you can pre-provision them, and store them in the accounts.yaml file, and use the pre-provisioned nradojevic creds provider | 08:38 |
| andreaf | nradojevic: there's a CLI tool you can use to generate creds and the file for you | 08:39 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted | 08:39 | |
| *** gfidente has joined #openstack-qa | 08:39 | |
| andreaf | nradojevic: try tempest account-generator | 08:39 |
| andreaf | gcheresh: ok, but after the restart :) | 08:40 |
| gcheresh | andreaf: thanks | 08:40 |
| gmann | andreaf: your bad luck, gerrit went down, i was almost reviewed your patch :) | 08:41 |
| openstackgerrit | huan proposed openstack-dev/devstack: Install conntrack in XenServer Dom0 https://review.openstack.org/352256 | 08:41 |
| gmann | andreaf: anyways ll check in 5 min after coffee break. | 08:41 |
| *** cdent has joined #openstack-qa | 08:42 | |
| nradojevic | andreaf: thanks! I'll give it a try :) | 08:44 |
| *** dtantsur is now known as dtantsur|sick | 08:44 | |
| *** grafuls has joined #openstack-qa | 08:49 | |
| *** tosky has joined #openstack-qa | 08:51 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/352214 failed gate-grenade-dsvm-neutron-multinode in the gate queue with an unrecognized error | 08:55 |
| *** yaume has joined #openstack-qa | 08:56 | |
| gmann | andreaf: +2 | 09:02 |
| andreaf | thanks | 09:02 |
| gmann | andreaf: apart from that, how about dumping all duplicate service clients together. I left comment there | 09:02 |
| *** binbincong has quit IRC | 09:03 | |
| andreaf | gmann: yeah we could do that | 09:04 |
| *** gvrangan has quit IRC | 09:10 | |
| *** e0ne has joined #openstack-qa | 09:11 | |
| *** yamamoto has joined #openstack-qa | 09:12 | |
| *** shausy has quit IRC | 09:13 | |
| *** shausy has joined #openstack-qa | 09:13 | |
| *** yamamot__ has joined #openstack-qa | 09:14 | |
| *** yamamoto has quit IRC | 09:17 | |
| *** mvk has joined #openstack-qa | 09:24 | |
| *** mvk_ has joined #openstack-qa | 09:27 | |
| *** binbincong has joined #openstack-qa | 09:27 | |
| *** yamamot__ has quit IRC | 09:35 | |
| *** yamamoto has joined #openstack-qa | 09:35 | |
| openstackgerrit | Andrea Frittoli proposed openstack/tempest: Try to register all service clients https://review.openstack.org/352292 | 09:36 |
| andreaf | gmann: something like this ^^^ ? | 09:36 |
| *** mvk_ has quit IRC | 09:38 | |
| openstackgerrit | Andrea Frittoli proposed openstack/tempest: Introduce test case with class cleanups https://review.openstack.org/351871 | 09:40 |
| *** binbincong has quit IRC | 09:40 | |
| *** yamamoto has quit IRC | 09:41 | |
| *** kbaikov has quit IRC | 09:41 | |
| openstackgerrit | Merged openstack/openstack-health: Add total count table to per test page https://review.openstack.org/351831 | 09:59 |
| *** binbincong has joined #openstack-qa | 10:04 | |
| *** yamamoto has joined #openstack-qa | 10:05 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** yamamoto has joined #openstack-qa | 10:06 | |
| *** kbaikov has joined #openstack-qa | 10:16 | |
| *** sshnaidm has quit IRC | 10:20 | |
| *** EinstCrazy has quit IRC | 10:24 | |
| *** yangyapeng has quit IRC | 10:26 | |
| *** hieulq_ has quit IRC | 10:27 | |
| *** yamamoto has quit IRC | 10:33 | |
| *** gszasz has joined #openstack-qa | 10:38 | |
| *** yamamoto has joined #openstack-qa | 10:39 | |
| gcheresh | andreaf: thanks | 10:39 |
| *** sshnaidm has joined #openstack-qa | 10:40 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:40 | |
| *** Ravikiran_K has quit IRC | 10:40 | |
| *** yamamoto has quit IRC | 10:51 | |
| *** yamamoto has joined #openstack-qa | 10:55 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-qa | 10:56 | |
| *** yamahata has quit IRC | 10:58 | |
| *** sdague has joined #openstack-qa | 11:00 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-qa | 11:03 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-qa | 11:05 | |
| *** pcaruana has quit IRC | 11:06 | |
| *** pcaruana has joined #openstack-qa | 11:07 | |
| andreaf | mtreinish: I had to re-spin https://review.openstack.org/#/c/347756, there was an issue after the rebase - could you have a look at it again? It works fine with the plugins I tested | 11:08 |
| *** yamamoto has quit IRC | 11:10 | |
| *** yamamoto has joined #openstack-qa | 11:10 | |
| *** lucasagomes is now known as lucas-hungry | 11:21 | |
| *** e0ne has quit IRC | 11:23 | |
| *** yamamoto has quit IRC | 11:26 | |
| *** yamamoto has joined #openstack-qa | 11:26 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:28 | |
| *** yamamoto has quit IRC | 11:31 | |
| *** rodrigods has quit IRC | 11:33 | |
| *** rodrigods has joined #openstack-qa | 11:33 | |
| *** yamamoto has joined #openstack-qa | 11:34 | |
| *** yamamoto has quit IRC | 11:34 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/344193 failed gate-grenade-dsvm-neutron-multinode in the gate queue with an unrecognized error | 11:35 |
| *** markvoelker has joined #openstack-qa | 11:45 | |
| *** yamamoto has joined #openstack-qa | 11:46 | |
| *** yamamoto has quit IRC | 11:46 | |
| *** binbincong has quit IRC | 11:55 | |
| *** Poornima has quit IRC | 11:55 | |
| *** raildo has joined #openstack-qa | 11:56 | |
| *** larainema has quit IRC | 11:57 | |
| *** jpena is now known as jpena|lunch | 11:58 | |
| *** larainema has joined #openstack-qa | 11:59 | |
| *** rossella_s has quit IRC | 12:03 | |
| *** yangyapeng has joined #openstack-qa | 12:04 | |
| sdague | sc68cal: let me know once you pop up for the morning, we should figure out how we're going to move forward with the defaults change in devstack | 12:06 |
| *** yamamoto has joined #openstack-qa | 12:12 | |
| *** yamamoto has quit IRC | 12:12 | |
| *** yamamoto has joined #openstack-qa | 12:13 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-qa | 12:14 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** gouthamr has joined #openstack-qa | 12:17 | |
| *** lucas-hungry is now known as lucasagomes | 12:18 | |
| *** moravec has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-qa | 12:20 | |
| *** e0ne has joined #openstack-qa | 12:20 | |
| openstackrecheck | Console logs not available after 13:28s for gate-nova-pep8-ubuntu-xenial 343046,5,56de713 | 12:20 |
| *** yamamoto has quit IRC | 12:25 | |
| *** weshay has joined #openstack-qa | 12:26 | |
| *** amotoki has joined #openstack-qa | 12:31 | |
| sc68cal | sdague: ok. I have a meeting from 9 to 10, maybe do a hangout after 11am? | 12:33 |
| *** hichihara has joined #openstack-qa | 12:34 | |
| sdague | sure | 12:35 |
| *** Ravikiran_K has joined #openstack-qa | 12:38 | |
| openstackgerrit | Igor Milovanovic proposed openstack-dev/devstack: Fixed language. https://review.openstack.org/352392 | 12:42 |
| *** ihrachys has joined #openstack-qa | 12:46 | |
| *** rlrossit has joined #openstack-qa | 12:48 | |
| sdague | dansmith: http://logs.openstack.org/76/351876/2/experimental/gate-tempest-dsvm-neutron-dvr/500e970/ is an ovh run with your reject patch (still fails) but maybe there is some further debug in there | 12:48 |
| openstackgerrit | guo yunxian proposed openstack/tempest: Add update object testcase https://review.openstack.org/352197 | 12:49 |
| *** Deng has quit IRC | 12:50 | |
| *** dave-mccowan has joined #openstack-qa | 12:52 | |
| *** rfolco has joined #openstack-qa | 12:54 | |
| *** amotoki has quit IRC | 12:56 | |
| *** jecarey_ has quit IRC | 12:59 | |
| *** gyx has quit IRC | 12:59 | |
| *** _nadya_ has quit IRC | 13:05 | |
| *** rossella_s has joined #openstack-qa | 13:05 | |
| *** _nadya_ has joined #openstack-qa | 13:09 | |
| *** amotoki has joined #openstack-qa | 13:13 | |
| *** dustins has joined #openstack-qa | 13:17 | |
| *** jpena|lunch is now known as jpena | 13:17 | |
| *** dkranz has joined #openstack-qa | 13:17 | |
| *** pgadiya has quit IRC | 13:20 | |
| openstackgerrit | Kyrylo Romanenko proposed openstack/tempest: Add function to parse raw CLI output into dictionary https://review.openstack.org/342165 | 13:22 |
| *** yamamoto has joined #openstack-qa | 13:24 | |
| *** amotoki has quit IRC | 13:27 | |
| *** porrua has joined #openstack-qa | 13:29 | |
| *** yamamoto has quit IRC | 13:29 | |
| dansmith | sdague: I will look in a bit | 13:29 |
| *** masayukig has quit IRC | 13:31 | |
| *** masayukig has joined #openstack-qa | 13:34 | |
| dansmith | kernel: [ 4507.225246] wtf: IN=eth0 OUT=br-ex MAC=fa:16:3e:2e:c0:b3:ce:31:15:5f:84:7a:08:00 SRC=10.100.0.18 DST=172.24.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=246 ID=29825 DF PROTO=ICMP TYPE=0 CODE=0 ID=24577 SEQ=0 | 13:39 |
| dansmith | sdague: my rule doesn't match because it's coming from fixed to public, so I'll tweak it | 13:41 |
| sdague | dansmith: ok, cool | 13:42 |
| dansmith | sdague: but that's definitely what is happening | 13:42 |
| *** moravec has joined #openstack-qa | 13:44 | |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: DNM: try me https://review.openstack.org/351876 | 13:44 |
| dansmith | sdague: this ^ prevents packets from ever leaving so they can't be bounced, which is more optimistic and cleaner.. if that doesn't work I'll do something else, but I think it should and that's the best option, IMHO | 13:45 |
| *** masayukig has quit IRC | 13:45 | |
| *** rbak has joined #openstack-qa | 13:46 | |
| sdague | ok, lets see if we can land it on an ovh node and figure out how it looks | 13:46 |
| *** masayukig has joined #openstack-qa | 13:46 | |
| sdague | it seems to me like we have the following options: | 13:46 |
| sdague | 1) merge the neutron patch as is & skip the tempest test until an ovh solution exists | 13:47 |
| sdague | 2) build a workaround in devstack for ovh (per your patch) | 13:47 |
| sdague | 3) put a workaround into devstack-gate ... given that it's a provider level issue | 13:48 |
| dansmith | yeah | 13:48 |
| sdague | dansmith: you have opinions? I'd like to get the neutron default moved forward soon so that we have time to react to any other fallout | 13:48 |
| dansmith | I've never seen a case where networking was configured as such, so 3 is probably sufficient, but this also doesn't hurt anything for normal people | 13:49 |
| *** mriedem has joined #openstack-qa | 13:52 | |
| *** rlrossit has quit IRC | 13:53 | |
| *** mfisher_ora has joined #openstack-qa | 13:55 | |
| *** rlrossit has joined #openstack-qa | 13:55 | |
| *** Liuqing has joined #openstack-qa | 14:00 | |
| sdague | mtreinish: you want to weigh in here on path forward? | 14:02 |
| *** shausy has quit IRC | 14:03 | |
| *** e0ne has quit IRC | 14:03 | |
| *** gvrangan has joined #openstack-qa | 14:04 | |
| mtreinish | sdague: on the neutron as default thing? | 14:04 |
| sdague | on how we get past this ovh vs. tempest issue | 14:05 |
| jroll | it looks like clouds.yaml is used in devstack (neutron --os-cloud ...), is clouds.yaml stored in CI logs? | 14:05 |
| * jroll waits patiently while ovh things are figured out | 14:05 | |
| mtreinish | sdague: I think a devstack-gate workaround seems like the best approach. I agree with dansmith having a network env where things bounce back is kinda weird and I don't think people will be likely to see it elsewhere | 14:07 |
| *** grafuls has quit IRC | 14:07 | |
| *** EinstCrazy has joined #openstack-qa | 14:07 | |
| *** abehl has quit IRC | 14:08 | |
| tosky | jroll: clous.yaml is stored in the CI logs IIRC with the "standard" result collector | 14:08 |
| tosky | clouds.yaml | 14:09 |
| jroll | tosky: I'm not seeing it here, unless I'm missing something http://logs.openstack.org/87/351987/5/check/gate-grenade-dsvm-ironic/e203b36/logs/ | 14:09 |
| tosky | jroll: etc/openstack | 14:09 |
| jroll | oh wow, thank you | 14:10 |
| jroll | was looking for /etc/clouds.yaml, /etc/openstack didn't even register in my brain | 14:10 |
| * jroll gets more coffee | 14:10 | |
| *** spzala has joined #openstack-qa | 14:10 | |
| *** ajmiller has joined #openstack-qa | 14:10 | |
| *** woodster_ has joined #openstack-qa | 14:11 | |
| jroll | next question, does anyone know why this command would fail (looks like because it's connecting to keystone v2) when clouds.yaml specifies v3? http://logs.openstack.org/87/351987/5/check/gate-grenade-dsvm-ironic/e203b36/logs/grenade.sh.txt.gz#_2016-08-08_13_17_47_714 clouds.yaml: http://logs.openstack.org/87/351987/5/check/gate-grenade-dsvm-ironic/e203b36/logs/etc/openstack/clouds.yaml.txt.gz | 14:12 |
| jroll | context, our grenade job is failing :/ | 14:12 |
| openstackgerrit | Vasyl Saienko proposed openstack-dev/devstack: DO NOT REVIEW https://review.openstack.org/352442 | 14:14 |
| *** zz_dimtruck is now known as dimtruck | 14:15 | |
| *** lutostag has joined #openstack-qa | 14:15 | |
| *** spzala has quit IRC | 14:15 | |
| vponomaryov1 | jroll: you are not alone: https://bugs.launchpad.net/manila/+bug/1610983 | 14:15 |
| openstack | Launchpad bug 1610983 in Manila "Grenade job fails on tempest tests" [Critical,In progress] - Assigned to Valeriy Ponomaryov (vponomaryov) | 14:15 |
| *** e0ne has joined #openstack-qa | 14:15 | |
| *** spzala has joined #openstack-qa | 14:15 | |
| jroll | oh, yay | 14:15 |
| *** spzala has quit IRC | 14:16 | |
| *** spzala has joined #openstack-qa | 14:16 | |
| *** chandankumar has quit IRC | 14:17 | |
| jroll | vponomaryov1: I was just filing a bug, thanks. any progress beyond what's in the bug? | 14:17 |
| vponomaryov1 | jroll: I know that it is some incompatibility between v2 and v3 auth in Keystone | 14:18 |
| jroll | vponomaryov1: right, but according to clouds.yaml, that should be using v3 | 14:18 |
| vponomaryov1 | jrol: but haven't find change that triggered that bug | 14:18 |
| *** mtanino has joined #openstack-qa | 14:18 | |
| jroll | I think that's a known incompatibility | 14:19 |
| jroll | yeah, same | 14:19 |
| sdague | mtreinish: so, how about if we skip the test for now in tempest to get things merged, then we can work on a fix on the devstack-gate side to work around things. | 14:24 |
| sdague | so 1 then 3 | 14:25 |
| *** EinstCrazy has quit IRC | 14:26 | |
| *** beekneemech is now known as bnemec | 14:27 | |
| *** EinstCrazy has joined #openstack-qa | 14:28 | |
| *** e0ne has quit IRC | 14:28 | |
| *** e0ne has joined #openstack-qa | 14:28 | |
| *** vponomaryov1 is now known as vponomaryov | 14:29 | |
| *** Lord_Phoenix has quit IRC | 14:30 | |
| *** jecarey has joined #openstack-qa | 14:31 | |
| mtreinish | sdague: tbh, I'm not sure we want to skip that test. It seems like a kinda big thing to make sure we don't regress on (even if it's a poorly written implementation) | 14:34 |
| *** jecarey has quit IRC | 14:35 | |
| *** pglass has joined #openstack-qa | 14:36 | |
| *** thiagop has joined #openstack-qa | 14:37 | |
| *** nradojevic1 has joined #openstack-qa | 14:39 | |
| sdague | mtreinish: well, I feel like if we delay the devstack default change much further, we have to take if off the table for newton, because we don't have enough time to handle fallout | 14:39 |
| sdague | and, through this whole process we've also discovered that it is unlikely that any neutron tests ever run in our system have created guests which actually can talk to external networks | 14:40 |
| *** nradojevic has quit IRC | 14:40 | |
| *** markvoelker has quit IRC | 14:41 | |
| *** nradojevic1 has quit IRC | 14:42 | |
| *** jecarey has joined #openstack-qa | 14:43 | |
| openstackgerrit | Sean Dague proposed openstack/tempest: skip test_connectivity_between_vms_on_different_networks https://review.openstack.org/352463 | 14:47 |
| sc68cal | sdague: that is correct, we've only done tests that do east west inside the cloud | 14:50 |
| sdague | sc68cal: that seems kind of bonkers to me :) | 14:50 |
| sc68cal | sdague: otherwise it's from tempest, into a vm | 14:51 |
| sc68cal | inside the cloud | 14:51 |
| sdague | sc68cal: right | 14:51 |
| sdague | it's just, still bonkers to me | 14:51 |
| sc68cal | I don't recall nova having tests that test outbound | 14:51 |
| sdague | because cloud guests that can't route out seem completely useless | 14:51 |
| sdague | it doesn't, because it works, and no one thought it needed to be tested as such | 14:52 |
| sc68cal | right but those east west tests test routing out, albiet only routing one hop, while still in the cloud | 14:52 |
| sdague | it's kind of such a fundamental unit of what one expects, I don't think it occurred to anyone that it needed testing | 14:52 |
| sdague | maybe it's just me, but I was legitimately surprised by this | 14:53 |
| *** markvoelker has joined #openstack-qa | 14:54 | |
| *** edannon has quit IRC | 14:56 | |
| *** jecarey_ has joined #openstack-qa | 14:57 | |
| *** zackf has joined #openstack-qa | 14:58 | |
| *** jecarey has quit IRC | 14:59 | |
| *** amotoki has joined #openstack-qa | 14:59 | |
| *** e0ne has quit IRC | 14:59 | |
| *** amotoki has quit IRC | 14:59 | |
| *** karthikp_ has joined #openstack-qa | 15:00 | |
| *** e0ne has joined #openstack-qa | 15:00 | |
| *** kragniz has quit IRC | 15:00 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/348512 failed because of: gate-grenade-dsvm-multinode: https://bugs.launchpad.net/bugs/1449136 | 15:00 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 15:00 |
| *** amotoki has joined #openstack-qa | 15:01 | |
| *** rlrossit has quit IRC | 15:02 | |
| *** mriedem1 has joined #openstack-qa | 15:04 | |
| *** rlrossit has joined #openstack-qa | 15:05 | |
| *** karthikp_ has quit IRC | 15:05 | |
| *** mriedem has quit IRC | 15:05 | |
| *** mriedem1 is now known as mriedem | 15:07 | |
| *** amotoki has quit IRC | 15:08 | |
| *** devkulkarni1 has joined #openstack-qa | 15:09 | |
| mriedem | mtreinish: did something slip through in tempest? http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/screen-n-api.txt.gz?level=TRACE#_2016-08-06_23_42_20_711 | 15:09 |
| mriedem | http://goo.gl/P4gVQe | 15:10 |
| mriedem | no recent changes in tempest | 15:10 |
| *** chandankumar has joined #openstack-qa | 15:13 | |
| mtreinish | mriedem: I havn't seen that before | 15:14 |
| mriedem | i guess 4 hits in 7 days | 15:14 |
| *** amotoki has joined #openstack-qa | 15:14 | |
| mriedem | oh this is one of those tests that re-use a single instance and it's rebuilt | 15:15 |
| *** gvrangan has quit IRC | 15:16 | |
| *** amotoki has quit IRC | 15:16 | |
| *** amotoki has joined #openstack-qa | 15:16 | |
| mtreinish | mriedem: ugh, I thought we dropped all of that | 15:19 |
| *** amotoki_ has joined #openstack-qa | 15:20 | |
| *** amotoki has quit IRC | 15:21 | |
| *** gcheresh has quit IRC | 15:21 | |
| *** amotoki_ has quit IRC | 15:22 | |
| *** fnaval_ has quit IRC | 15:22 | |
| *** fnaval has joined #openstack-qa | 15:22 | |
| mriedem | mtreinish: looks like there was a shelved offloading test right before it | 15:25 |
| mriedem | then seeing this http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/screen-n-cpu.txt.gz#_2016-08-06_23_42_09_098 | 15:25 |
| mriedem | http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/screen-n-cpu.txt.gz#_2016-08-06_23_42_18_623 | 15:26 |
| mriedem | During the sync_power process the instance has moved from host None to host ubuntu-xenial-rax-ord-3248933 | 15:26 |
| mriedem | that's 2 seconds before suspend blows up | 15:27 |
| *** cody-somerville has joined #openstack-qa | 15:30 | |
| sdague | mriedem: no, that's used quite a bit during the actions testing | 15:31 |
| *** yamahata has joined #openstack-qa | 15:31 | |
| mtreinish | mriedem: yeah, it's got the same server uuid too | 15:31 |
| mriedem | it's unclear to me if tempest asked to rebuild the server | 15:32 |
| mriedem | the test cases should run in order right? | 15:33 |
| mtreinish | mriedem: it should be in the tempest log if it did: http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/tempest.txt.gz#_2016-08-06_23_42_20_719 is the suspend request it blows up on | 15:33 |
| *** devkulkarni2 has joined #openstack-qa | 15:33 | |
| *** EinstCrazy has quit IRC | 15:34 | |
| sdague | mtreinish: ok, circling back to the neutron default plan - http://lists.openstack.org/pipermail/openstack-dev/2016-August/101128.html | 15:34 |
| *** csomerville has joined #openstack-qa | 15:34 | |
| *** karthikp_ has joined #openstack-qa | 15:34 | |
| *** EinstCrazy has joined #openstack-qa | 15:35 | |
| *** armax has joined #openstack-qa | 15:35 | |
| sdague | I feel pretty strongly that merge / skip / d-g change / skip revert is the only plan we can execute that gives us enough time to do neutron switch over this cycle | 15:35 |
| *** devkulkarni1 has quit IRC | 15:36 | |
| mriedem | mtreinish: yeah i can see that the hostId on the server in the tempest logs before it fails is "" | 15:36 |
| mriedem | so there isn't a host | 15:36 |
| sdague | given that freeze is in 13 business days | 15:36 |
| mtreinish | mriedem: it doesn't do a rebuild before that test. The teardown right before does a GET and then setup does another GET and then it's into the suspend test (probably to check the state and whether it needs to rebuild) | 15:36 |
| *** hichihara has quit IRC | 15:37 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/350645 failed gate-grenade-dsvm-neutron-multinode in the gate queue with an unrecognized error | 15:37 |
| mtreinish | sdague: if you get the devstack-gate patch up for review before the skip lands I won't have that big an issue with it | 15:37 |
| sdague | mtreinish: we don't even know what that patch would be at this point | 15:37 |
| *** cody-somerville has quit IRC | 15:37 | |
| sdague | it's going to take days to get the right onw | 15:37 |
| sdague | that's the point, timelines | 15:38 |
| sdague | we're waiting for the next 3 hours to get the next iteration of results off of dansmith's devstack patch | 15:41 |
| sdague | if it manages to land on an ovh node | 15:41 |
| sdague | it will be tomorrow if we don't get lucky | 15:41 |
| sdague | once we have some idea that it works there, we can then translate it to d-g, which will probably take 3 days before we get enough data to know it worked | 15:42 |
| mtreinish | sdague: that's kinda my concern. It feels like we're trying to force this through without having a clear plan/idea on doing the unskip part. | 15:42 |
| sdague | because you have to have neutron + ovh to see the issue | 15:42 |
| *** Liuqing has quit IRC | 15:43 | |
| sdague | mtreinish: yes, we are trying to force through the #1 priority for devstack for the whole cycle | 15:43 |
| mriedem | mtreinish: yeah idk, something is shelving it http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/screen-n-cpu.txt.gz#_2016-08-06_23_42_05_126 | 15:43 |
| mriedem | vm_state='shelved' | 15:43 |
| mtreinish | right I understand all the parts that are involved, and the timelines for this. I didn't say I'm blocking anything, just that'd I'd be more comfortable skipping the test if we had the d-g patch at least up before | 15:44 |
| *** sridharg has quit IRC | 15:44 | |
| sdague | mtreinish: that's fine, but I'm not comfortable merging the devstack default after wednesday | 15:44 |
| sdague | and that's where we hit a snag | 15:44 |
| sdague | because I'll guaruntee you that we can't have the d-g change by then | 15:45 |
| *** Ravikiran_K has quit IRC | 15:45 | |
| sdague | so... given that, how do you want to proceed | 15:45 |
| *** jlanoux has quit IRC | 15:45 | |
| mtreinish | sdague: fwiw, it does look like it occasionally is catching something (or just racing) http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks?duration=P6M&resolutionKey=day | 15:45 |
| mriedem | nothing in the api for a shelve request though, so wonder if something is doing it in a task on the compute manager... | 15:45 |
| mtreinish | especially recently | 15:45 |
| *** Serlex has quit IRC | 15:46 | |
| mriedem | umm, wtf http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/tempest.txt.gz#_2016-08-06_23_42_05_262 | 15:46 |
| mriedem | looks like tempest is re-using the same shared server between 2 different test classes | 15:46 |
| mtreinish | sdague: the 4 steps you outlined is the only way we get it landed by wed. and not have everyone grabbing torches and pitchforks for failing every neutron ovh run | 15:47 |
| sdague | right, which would basically just kill tons of unrelated freeze content | 15:47 |
| mtreinish | mriedem: it looks like those are still part of ServersNegativeTestJSON | 15:48 |
| mtreinish | mriedem: that's teh same class | 15:48 |
| sdague | mtreinish: so that graph doesn't tell me much, because I can't figure out what runs any of those were on or go any deeper into things | 15:48 |
| mriedem | oh right | 15:48 |
| mriedem | test_shelve_shelved_server | 15:48 |
| mtreinish | sdague: yeah, I have an inprogress patch adding a recent failures list to that page | 15:49 |
| mriedem | mtreinish: do the tests run in a random order then? | 15:49 |
| mtreinish | I still need to finish it :) | 15:49 |
| mtreinish | mriedem: sorta, there is no guarantee on the test ordering it's determined by the test runner | 15:49 |
| mriedem | ok | 15:49 |
| mtreinish | normally subunit.run is pretty consistent in how it orders things, but we can't rely on that | 15:50 |
| *** edmondsw has quit IRC | 15:50 | |
| mriedem | yup, ok | 15:50 |
| mriedem | so unshelve is a cast, and then the test will move on | 15:50 |
| sdague | mtreinish: that still wouldn't tell me what's needed to figure out if that failure rate means anything substantial | 15:50 |
| sdague | there are 43 failures listed | 15:51 |
| sdague | I know where 4 of them are, they are in my patch | 15:51 |
| mtreinish | sdague: this is only in the gate queue | 15:52 |
| mtreinish | sdague: and that's 43 over the full 6 months date range for teh graph | 15:52 |
| mtreinish | well gate and periodic | 15:52 |
| mtreinish | sdague: it would let you dig into the failures. After the table is there I plan to add the e-r integration to it as well so it will identify any e-r matches | 15:53 |
| sdague | right, well the issue is that without having a link through to all the fails listed, it's hard to figure out if they are real or if it's pip blowing up | 15:53 |
| sdague | or other wonky things in the run where a set of other circumstances happened | 15:53 |
| sdague | or the provider corelation | 15:53 |
| mtreinish | this is only on the actual test failing in the subunit, so a pip failure wouldn't be included there | 15:53 |
| mtreinish | it could be provider thing though | 15:54 |
| sdague | ok... so beyond arguing about openstack health :) | 15:54 |
| *** weshay is now known as weshay_mtg | 15:55 | |
| mtreinish | mriedem: https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_servers_negative.py#L451-L470 so do we need a wait loop there? | 15:55 |
| sdague | things we are agreed upon. The only way to get default merge by wed is the skip plan. | 15:55 |
| mriedem | mtreinish: that happens in server_check_teardown | 15:56 |
| mtreinish | sdague: heh, it's the only way I get feedback on things, is to force it down people's throat | 15:56 |
| mriedem | it waits for the shared server to be active and if that fails, it deletes the shared server so setUp can recreate it | 15:56 |
| sdague | which, means, I'd like to get us moved forward on that plan, so we can start to see the other fallout (which I'm sure there will be some), and work on docs cleanups and the d-g bits once we have the devstack data returned | 15:56 |
| *** hemnafk is now known as hemna | 15:56 | |
| mriedem | mtreinish: right here before we fail we have an active server with no task_state that doesn't have a host http://logs.openstack.org/91/327191/22/check/gate-tempest-dsvm-full-ubuntu-xenial/6a11005/logs/tempest.txt.gz#_2016-08-06_23_42_20_558 | 15:56 |
| mriedem | that's got to be a bug in the nova unshelve flow | 15:57 |
| mtreinish | sdague: is that just your roundabout way asking me to +2 the devstack patch? :) | 15:58 |
| sdague | and the tempest patch | 15:58 |
| sdague | mtreinish: I'm trying to figure out how we move forward with this whole transition, given the constraints we have | 15:58 |
| sdague | or realizing we have no path that meets the timeline, and abandoning all the patches and working on something else this week | 15:59 |
| *** amotoki has joined #openstack-qa | 15:59 | |
| *** karthikp_ has quit IRC | 15:59 | |
| mtreinish | sdague: ok, I'll review both patches nowish then | 16:00 |
| sdague | mtreinish: thanks | 16:00 |
| *** karthikp_ has joined #openstack-qa | 16:01 | |
| *** sshnaidm has quit IRC | 16:01 | |
| *** rcernin has quit IRC | 16:01 | |
| *** amotoki has quit IRC | 16:02 | |
| *** amotoki has joined #openstack-qa | 16:02 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova unshelve race bug 1611008 https://review.openstack.org/352495 | 16:03 |
| openstack | bug 1611008 in OpenStack Compute (nova) "ServersNegativeTestJSON.test_suspend_server_invalid_state fails with "NovaException: Unable to find host for Instance"" [Undecided,Confirmed] https://launchpad.net/bugs/1611008 | 16:03 |
| *** sabeen1 has joined #openstack-qa | 16:03 | |
| mriedem | mtreinish: ^ | 16:03 |
| mtreinish | mriedem: btw, what did you think of my patch to split the integrated uncategorized page out? You didn't leave any comments on the idea, just a bunch of nits :) | 16:04 |
| mriedem | mtreinish: seems good | 16:06 |
| mriedem | it was the weekend, what do you want from me?! | 16:06 |
| mtreinish | mriedem: hah, well not to be reviewing code. You could have waited until today | 16:07 |
| *** hichihara has joined #openstack-qa | 16:08 | |
| *** e0ne has quit IRC | 16:09 | |
| *** EinstCrazy has quit IRC | 16:09 | |
| *** pcaruana has quit IRC | 16:11 | |
| *** Ravikiran_K has joined #openstack-qa | 16:11 | |
| *** dalees has quit IRC | 16:12 | |
| *** dimtruck is now known as zz_dimtruck | 16:12 | |
| *** Apoorva has joined #openstack-qa | 16:12 | |
| *** sshnaidm has joined #openstack-qa | 16:13 | |
| *** hichihara has quit IRC | 16:14 | |
| *** amotoki has quit IRC | 16:15 | |
| *** _nadya_ has quit IRC | 16:18 | |
| *** rlrossit has quit IRC | 16:19 | |
| *** rlrossit has joined #openstack-qa | 16:20 | |
| mtreinish | sdague: I'm +2 on the devstack change, do you want me to +A it. Or is there a reason to wait? | 16:22 |
| *** ifarkas is now known as ifarkas_afk | 16:24 | |
| *** sabeen1 has quit IRC | 16:26 | |
| *** yfried has quit IRC | 16:26 | |
| *** gamado has quit IRC | 16:27 | |
| *** david-lyle_ has joined #openstack-qa | 16:29 | |
| *** david-lyle has quit IRC | 16:33 | |
| *** gyee has joined #openstack-qa | 16:33 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add option to ensure get_test_runs_by_test_id() is ordered by date https://review.openstack.org/352506 | 16:35 |
| *** rlrossit has quit IRC | 16:36 | |
| *** zz_dimtruck is now known as dimtruck | 16:36 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/269825 failed because of: gate-grenade-dsvm-neutron-multinode: unrecognized error, gate-tempest-dsvm-full-ubuntu-xenial: https://bugs.launchpad.net/bugs/1449136 | 16:39 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 16:39 |
| *** rlrossit has joined #openstack-qa | 16:39 | |
| *** rlrossit has quit IRC | 16:40 | |
| jroll | ^ that unrecognized error is being fixed right now by https://review.openstack.org/#/c/352454/ jfyi | 16:40 |
| *** Guest20583 has joined #openstack-qa | 16:40 | |
| jroll | do y'all want an e-r for that still? | 16:40 |
| *** Swami has joined #openstack-qa | 16:42 | |
| sdague | mtreinish: we should probably approve the tempest patch first to minimize chances of things failing | 16:42 |
| openstackrecheck | openstack/swift change: https://review.openstack.org/326903 failed gate-tempest-dsvm-full-ubuntu-xenial in the gate queue with an unrecognized error | 16:43 |
| *** jpena is now known as jpena|off | 16:44 | |
| *** Guest20583 has quit IRC | 16:45 | |
| *** dalees has joined #openstack-qa | 16:47 | |
| *** e0ne has joined #openstack-qa | 16:49 | |
| *** e0ne has quit IRC | 16:49 | |
| mtreinish | jroll: probably, it lets us track that it was fixed | 16:49 |
| mtreinish | jroll: and gets the categorization rate up, even if a little bit | 16:50 |
| jroll | mtreinish: okay, I'll try to get it up, got a meeting in 8 minutes | 16:51 |
| *** devkulkarni2 has quit IRC | 16:53 | |
| openstackgerrit | Sean Dague proposed openstack/tempest: skip test_connectivity_between_vms_on_different_networks https://review.openstack.org/352463 | 16:54 |
| sdague | mtreinish: had to hack in a depends on for the tempest test, because the occ failure | 16:54 |
| *** tesseract- has quit IRC | 16:57 | |
| *** e0ne has joined #openstack-qa | 17:09 | |
| *** gvrangan has joined #openstack-qa | 17:16 | |
| *** yamamoto has joined #openstack-qa | 17:21 | |
| *** pglass has quit IRC | 17:25 | |
| *** devkulkarni1 has joined #openstack-qa | 17:26 | |
| *** yamamoto has quit IRC | 17:27 | |
| *** karthikp_ has quit IRC | 17:27 | |
| *** edmondsw has joined #openstack-qa | 17:28 | |
| *** ihrachys has quit IRC | 17:37 | |
| *** Nakato has quit IRC | 17:41 | |
| *** gszasz has quit IRC | 17:41 | |
| *** david-lyle_ has quit IRC | 17:42 | |
| *** Nakato has joined #openstack-qa | 17:42 | |
| mriedem | mtreinish: figured it out | 17:45 |
| mriedem | the race | 17:45 |
| *** david-lyle has joined #openstack-qa | 17:45 | |
| *** e0ne has quit IRC | 17:47 | |
| mtreinish | mriedem: cool | 17:47 |
| mriedem | just needed a giant burrito in my gut to make my brain work | 17:48 |
| mriedem | mtreinish: there is a bug here too i think https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_servers_negative.py#L455 | 17:52 |
| mriedem | tempest doesn't wait for the server to change state to shelved before it tries to unshelve | 17:53 |
| mriedem | it just waits for the snapshot to show pu | 17:53 |
| mriedem | *up | 17:53 |
| mriedem | so it's really just getting lucky | 17:53 |
| *** karthikp_ has joined #openstack-qa | 17:53 | |
| *** sabeen has joined #openstack-qa | 17:54 | |
| openstackrecheck | Console logs not available after 13:27s for keystone-coverage-db-ubuntu-xenial 341401,7,3d56ca8 | 17:55 |
| *** mriedem has quit IRC | 17:55 | |
| *** tosky has quit IRC | 17:55 | |
| *** mriedem has joined #openstack-qa | 17:56 | |
| *** fnaval has quit IRC | 17:58 | |
| *** pvaneck has joined #openstack-qa | 17:58 | |
| *** mriedem1 has joined #openstack-qa | 17:58 | |
| *** karthikp_ has quit IRC | 17:59 | |
| *** karthikp_ has joined #openstack-qa | 17:59 | |
| *** mriedem has quit IRC | 18:01 | |
| *** pglass has joined #openstack-qa | 18:07 | |
| *** lucasagomes is now known as lucas-dinner | 18:10 | |
| *** weshay_mtg is now known as weshay | 18:11 | |
| *** jecarey_ has quit IRC | 18:11 | |
| *** devkulkarni1 has quit IRC | 18:11 | |
| *** jecarey has joined #openstack-qa | 18:12 | |
| *** jecarey has quit IRC | 18:12 | |
| *** jecarey has joined #openstack-qa | 18:12 | |
| *** gcheresh has joined #openstack-qa | 18:14 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/350545 failed gate-grenade-dsvm-neutron-multinode in the gate queue with an unrecognized error | 18:15 |
| *** fnaval has joined #openstack-qa | 18:16 | |
| *** gfidente has quit IRC | 18:19 | |
| *** ajmiller has quit IRC | 18:25 | |
| *** ajmiller has joined #openstack-qa | 18:29 | |
| *** greghaynes has quit IRC | 18:35 | |
| *** jecarey has quit IRC | 18:38 | |
| *** gvrangan has quit IRC | 18:39 | |
| *** jecarey_ has joined #openstack-qa | 18:43 | |
| *** devkulkarni1 has joined #openstack-qa | 18:44 | |
| *** chandankumar has quit IRC | 18:46 | |
| *** greghaynes has joined #openstack-qa | 18:47 | |
| *** mriedem1 is now known as mriedem | 18:47 | |
| *** mtreinish_ has joined #openstack-qa | 18:50 | |
| *** sc`_ has joined #openstack-qa | 18:50 | |
| *** raissa_ has joined #openstack-qa | 18:50 | |
| *** nibalize1 has joined #openstack-qa | 18:50 | |
| *** ianw_ has joined #openstack-qa | 18:50 | |
| *** sc68cal_ has joined #openstack-qa | 18:50 | |
| *** pcm_ has joined #openstack-qa | 18:50 | |
| *** raissa has quit IRC | 18:50 | |
| *** ianw has quit IRC | 18:50 | |
| *** mtreinish has quit IRC | 18:50 | |
| *** sc68cal has quit IRC | 18:50 | |
| *** pc_m has quit IRC | 18:50 | |
| *** jay-mehta has quit IRC | 18:50 | |
| *** kprabhuv has quit IRC | 18:50 | |
| *** sneti has quit IRC | 18:50 | |
| *** nibalizer has quit IRC | 18:50 | |
| *** sc` has quit IRC | 18:50 | |
| *** SlickNik has quit IRC | 18:50 | |
| *** sc`_ is now known as sc` | 18:50 | |
| *** mtreinish_ is now known as mtreinish | 18:50 | |
| *** SlickNik has joined #openstack-qa | 18:50 | |
| *** ianw_ is now known as ianw | 18:50 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add DB API function to get list of runs from ids https://review.openstack.org/352558 | 18:50 |
| *** jay-mehta has joined #openstack-qa | 18:51 | |
| *** Ravikiran_K has quit IRC | 18:51 | |
| *** sneti has joined #openstack-qa | 18:53 | |
| *** devkulkarni1 has quit IRC | 18:53 | |
| *** devkulkarni1 has joined #openstack-qa | 18:55 | |
| *** sc68cal_ is now known as sc68cal | 18:56 | |
| *** devkulkarni1 has quit IRC | 18:59 | |
| *** devkulkarni1 has joined #openstack-qa | 19:00 | |
| mriedem | mtreinish: should the scenario tests honor the run_validation flag? | 19:09 |
| mriedem | like https://github.com/openstack/tempest/blob/master/tempest/scenario/test_shelve_instance.py#L76 | 19:09 |
| mriedem | we don't have network_for_ssh configured | 19:10 |
| mriedem | so that blows up | 19:10 |
| mriedem | and we do'nt have floating ips | 19:10 |
| mtreinish | mriedem: normally we don't add that flag on scenario tests, it's not the only one which does ssh | 19:11 |
| *** karthikp_ has quit IRC | 19:11 | |
| *** ajmiller has quit IRC | 19:12 | |
| sdague | mtreinish: is oomichi out this week? | 19:13 |
| mtreinish | sdague: not that I know of, but his schedule's been kinda erratic the past few weeks | 19:13 |
| sdague | ok | 19:13 |
| sdague | mtreinish: ok, so question on nova_cert item in tempest.conf | 19:14 |
| *** ajmiller has joined #openstack-qa | 19:14 | |
| mriedem | mtreinish: shouldn't the test pass validatable=True when creating the server then? | 19:14 |
| mtreinish | mriedem: I think it is in the wrapper in the manager class | 19:14 |
| sdague | that defaults to True, we've deprecated it in Nova, and it's not in the base services for local devstack | 19:14 |
| sdague | should we change tempest default to False, and then set it to true if service enabled in devstack? | 19:14 |
| mriedem | mtreinish: well it wouldn't matter anway, since it wouldn't be used | 19:14 |
| mriedem | if CONF.validation.run_validation and validatable: | 19:15 |
| mriedem | and later: if CONF.validation.run_validation and validatable: | 19:15 |
| mriedem | it associates a floating IP if connect_method=='floating' | 19:15 |
| mriedem | if CONF.validation.connect_method == 'floating': | 19:15 |
| mtreinish | sdague: I think it would be better swap it to false in devstack, and deprecate the option in tempest for a mitaka eol removal | 19:16 |
| sdague | sc68cal: related, is neutron metering something we should have in the default service list for neutron? We don't for the local case, which is the reason for 3 tempest failures | 19:16 |
| mtreinish | sdague: swapping defaults is something that's tripped people up before in the past | 19:16 |
| sc68cal | no I don't think so | 19:16 |
| sdague | mtreinish: ok, that will mean that people will start getting tempest fails unless they also change that | 19:17 |
| sdague | sc68cal: ok, that service isn't parametrized in tempest yet, so I'll need to add that | 19:17 |
| sc68cal | does anyone actually care about neutron metering? | 19:17 |
| sc68cal | it's for ceilometer | 19:18 |
| sc68cal | so really it should be defined in their jobs and enabled | 19:18 |
| sc68cal | and in their tempest plugin | 19:18 |
| sc68cal | unless I am mistaken | 19:18 |
| sdague | well, there are tempest tests for it | 19:18 |
| mtreinish | sdague: only on new deployments, not on an existing one. It's a tradeoff between skipping tests silently when you upgrade tempest against an existing cloud vs having to explicitly set that to false when running against a new deployment | 19:18 |
| mtreinish | sdague: is that the neutron extension list test? | 19:18 |
| sdague | mtreinish: that, plus 2 actual metering tests | 19:19 |
| mtreinish | for the extension one I would just pull it out of the list of things we check | 19:19 |
| sdague | mtreinish: when we delete the nova-cert daemon, any upgraded environments will start failing | 19:20 |
| mtreinish | sdague: after a nova upgrade | 19:20 |
| sdague | yep | 19:20 |
| *** kprabhuv has joined #openstack-qa | 19:21 | |
| *** ajmiller has quit IRC | 19:21 | |
| sdague | ok, well I don't understand the logic of doing it the way you explain, but if that's the way you think it should be done, so be it. | 19:21 |
| mtreinish | sdague: I'm not committed to either way, but my gut instinct was to avoid changing defaults. | 19:22 |
| mtreinish | but if you think it's better to change the tempest default that's fine | 19:22 |
| mtreinish | just make sure you release note it | 19:23 |
| sdague | sure | 19:23 |
| *** ajmiller has joined #openstack-qa | 19:23 | |
| sdague | mtreinish: which reno section would it go under? upgrade? | 19:24 |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: WIP: Add failed runs list to test page https://review.openstack.org/352569 | 19:25 |
| mtreinish | sdague: yeah upgrade is probably the best fit | 19:25 |
| mtreinish | sdague: also ^^^ is what I have for getting the runs list on that page | 19:25 |
| mtreinish | I'm still fighting angular to get the table to render though | 19:25 |
| mriedem | mtreinish: btw, ssh_connect_method in the compute group and connect_method in the validation group in tempest.conf, not at all confusing :) | 19:26 |
| mtreinish | timothyb89: ^^^ if you have any thoughts of what I'm clearly missing there | 19:26 |
| mtreinish | mriedem: one should be deprecated and the validation group is the only one that exists anymore | 19:26 |
| mtreinish | mriedem: https://github.com/openstack/tempest/blob/master/tempest/config.py#L581 ssh_connect_method doesn't even exist anymore | 19:27 |
| mriedem | was just going to say that | 19:27 |
| mriedem | it's mentioned in the confg help for something else thogh | 19:27 |
| mtreinish | mriedem: free patch :) | 19:27 |
| mriedem | will update | 19:28 |
| mriedem | i'm more worried about how busted our scripts are for setting these | 19:28 |
| *** yaume has quit IRC | 19:28 | |
| timothyb89 | mtreinish: I'll take a look | 19:28 |
| *** gvrangan has joined #openstack-qa | 19:29 | |
| openstackgerrit | Sean Dague proposed openstack/tempest: change nova_cert config to default False and deprecate it https://review.openstack.org/352570 | 19:29 |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: WIP: Add failed runs list to test page https://review.openstack.org/352569 | 19:29 |
| *** krtaylor has quit IRC | 19:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/tempest: Fix reference to nonexistent ssh_connect_method in config help https://review.openstack.org/352571 | 19:30 |
| mriedem | sdague: good call on the nova-cert one :) i just did that internally last week | 19:30 |
| sdague | mriedem: that will fix 2 of the 5 tempest fails with a local devstack | 19:31 |
| mtreinish | timothyb89: thanks | 19:31 |
| mtreinish | sdague: btw, that spike in failures on that graph was caused by the tempest preprov creds jobs regressing: http://paste.openstack.org/show/552077/ (the table rendering doesn't work yet, but the api side is) | 19:35 |
| mtreinish | so completely irrelevant to our prior discussion :) | 19:35 |
| sdague | mtreinish: nice :) | 19:35 |
| mtreinish | well, the periodic and: 'gate-tempest-dsvm-networking-midonet-v2' which I guess is a midokura test job | 19:37 |
| *** ajmiller has quit IRC | 19:37 | |
| timothyb89 | mtreinish: oh, it looks like the failed_runs data is an object and not an array | 19:37 |
| mtreinish | timothyb89: oh, yeah it is. I can switch it to a list | 19:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova unshelve race bug 1611008 https://review.openstack.org/352495 | 19:39 |
| openstack | bug 1611008 in OpenStack Compute (nova) "ServersNegativeTestJSON.test_suspend_server_invalid_state fails with "NovaException: Unable to find host for Instance"" [High,In progress] https://launchpad.net/bugs/1611008 - Assigned to Matt Riedemann (mriedem) | 19:39 |
| *** edmondsw has quit IRC | 19:42 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: WIP: Add failed runs list to test page https://review.openstack.org/352569 | 19:43 |
| mtreinish | timothyb89: that fixed it, but now 2 columns are being displayed: http://i.imgur.com/ctbSo3i.png | 19:44 |
| *** ajmiller has joined #openstack-qa | 19:45 | |
| mriedem | mtreinish: i'm going to bug you some more. if we are using dynamic creds and isolated networks, why would we need to hard-code in config the network to use for ssh? | 19:45 |
| mriedem | https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L668 | 19:45 |
| mriedem | we get a KeyError there | 19:45 |
| mriedem | we're using https://github.com/openstack/tempest/blob/master/tempest/common/dynamic_creds.py#L296 | 19:45 |
| openstackrecheck | openstack/cinder change: https://review.openstack.org/318737 failed gate-tempest-dsvm-neutron-full-ubuntu-xenial in the gate queue with an unrecognized error | 19:46 |
| mtreinish | timothyb89: oh, I know what it is, they're null on the wire now | 19:46 |
| mtreinish | hmm | 19:46 |
| timothyb89 | that would do it | 19:47 |
| timothyb89 | unrelated, but there's a missing </th> on line 91 :) | 19:47 |
| *** karthikp_ has joined #openstack-qa | 19:47 | |
| *** ajmiller has quit IRC | 19:48 | |
| *** ajmiller has joined #openstack-qa | 19:48 | |
| *** karthikp_ has quit IRC | 19:48 | |
| *** karthikp_ has joined #openstack-qa | 19:48 | |
| mtreinish | oh, I know why. That loop only happens if elastic-search is enabled I had to back out the config to push it | 19:48 |
| mtreinish | but that's a general bug (which I would have missed otherwise) | 19:48 |
| sdague | mtreinish: hmmm... so metering is listed as a network-api extension | 19:51 |
| *** yamahata has quit IRC | 19:51 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add failed runs list to test page https://review.openstack.org/352569 | 19:51 |
| sdague | which means, I think we are fine if we remove it from the list | 19:51 |
| mtreinish | timothyb89: ^^^ that should fix it | 19:51 |
| sdague | except then we need the complete list, right? | 19:51 |
| *** gvrangan has quit IRC | 19:52 | |
| mtreinish | sdague: well we default to all which means the answer to is 'extension x available' is always true | 19:52 |
| mtreinish | on stable we hard code the list in devstack | 19:52 |
| mtreinish | sdague: yeah, it would have to be everything minus metering | 19:53 |
| sdague | [u'default-subnetpools', u'network-ip-availability', u'network_availability_zone', u'auto-allocated-topology', u'ext-gw-mode', u'binding', u'agent', u'subnet_allocation', u'l3_agent_scheduler', u'tag', u'external-net', u'revisions', u'flavors', u'net-mtu', u'availability_zone', u'quotas', u'l3-ha', u'provider', u'multi-provider', u'address-scope', u'extraroute', u'timestamp_core', u'l3-flavors', u'port-security', | 19:53 |
| sdague | u'extra_dhcp_opt', u'service-type', u'security-group', u'dhcp_agent_scheduler', u'router_availability_zone', u'rbac-policies', u'standard-attr-description', u'router', u'allowed-address-pairs', u'dvr'] | 19:53 |
| sdague | ug.... | 19:53 |
| mtreinish | sdague: like: https://github.com/openstack-dev/devstack/blob/stable/kilo/lib/tempest#L353-L386 | 19:54 |
| *** karthikp_ has quit IRC | 19:55 | |
| *** openstackrecheck has quit IRC | 19:57 | |
| *** openstackrecheck has joined #openstack-qa | 19:58 | |
| *** karthikp_ has joined #openstack-qa | 20:00 | |
| sdague | oh, the tempest verify lets me do the exclude list, this might work that way | 20:02 |
| *** Apoorva has quit IRC | 20:03 | |
| mtreinish | mriedem: you shouldn't, the command to look up a network and pass it nova should use the dynamic creds generated one first | 20:04 |
| mtreinish | mriedem: oh in that case it's because the neutron created networks aren't externally accessible by default (in most cases, provider networks probably are) | 20:05 |
| *** ccarmack has joined #openstack-qa | 20:05 | |
| mtreinish | but something doesn't look quite right in that line with the key error | 20:06 |
| mtreinish | sdague: so would this be a better interface for that link I sent you before?: http://i.imgur.com/c4zLp76.png | 20:07 |
| *** Apoorva has joined #openstack-qa | 20:08 | |
| sdague | yeh | 20:08 |
| openstackgerrit | Monty Taylor proposed openstack-dev/devstack: DNM - Debugging occ/neutronclient interaction https://review.openstack.org/352583 | 20:09 |
| mriedem | mtreinish: i just reported this to try and dump details https://bugs.launchpad.net/tempest/+bug/1611100 | 20:09 |
| openstack | Launchpad bug 1611100 in tempest "scenario get_server_ip relies on hard-coded network_for_ssh network name" [Undecided,New] | 20:09 |
| *** cdent has quit IRC | 20:09 | |
| sdague | the date field wrapping definitely makes it harder to read clusters | 20:10 |
| *** ajmiller_ has joined #openstack-qa | 20:10 | |
| sdague | the area graph by percentage is also pretty misleading. Because based on that graph I'd have guessed the giant dip would be more than 3 failures | 20:11 |
| mtreinish | andreaf: ^^^ on mriedem's bug, your memory on the network selection details might be a bit fresher in your memory than for me | 20:11 |
| *** yamahata has joined #openstack-qa | 20:12 | |
| *** devkulkarni1 has quit IRC | 20:12 | |
| mtreinish | sdague: Yeah, the area graph it's just a percentage. But, if you switch it to stacked or stream it shows the raw count instead | 20:12 |
| *** nibalize1 is now known as nibalizer | 20:13 | |
| *** devkulkarni1 has joined #openstack-qa | 20:13 | |
| mtreinish | sdague: I can try to fix the wrapping in: https://review.openstack.org/#/c/352569/ , I just suck at the js/html ui bits | 20:14 |
| sdague | mtreinish: right, I still find the stacked and stream to be confusing in different ways | 20:14 |
| sdague | this is not area data | 20:14 |
| sdague | this should be a bar graph with daily buckets, and error bars to indicate how sure we are of that percentage | 20:14 |
| *** ajmiller has quit IRC | 20:15 | |
| sdague | area data implies there is something connected between points, which, there really isn't here. | 20:16 |
| mtreinish | sdague: sure, which of http://nvd3-community.github.io/nvd3/ do you think would be the best fit? | 20:18 |
| mtreinish | we can expand beyond that set, but nvd3 is what its setup for now, so that'll be the fastest way to switch | 20:18 |
| *** gvrangan has joined #openstack-qa | 20:20 | |
| dansmith | sdague: armax abandoned that DNM patch.. I presume we didn't get a run out of it as a result? | 20:21 |
| sdague | dansmith: oh... really? | 20:21 |
| sdague | armax: ? | 20:21 |
| dansmith | yeah | 20:21 |
| sdague | abandoned doesn't kill jobs directly | 20:22 |
| sdague | i don't think | 20:22 |
| armax | dansmith, sdague I followed sc68cal’s feedback, I didn’t notice there was a pending job in execution | 20:22 |
| sdague | I just restored | 20:22 |
| sdague | armax: there was a check experimental on it | 20:22 |
| armax | sdague, dansmith: sorry folks, I thought we were going with a d-g patch instead | 20:23 |
| dansmith | we are | 20:23 |
| dansmith | but this was getting us a much faster run of the proposed change | 20:23 |
| dansmith | and it's DNM so nobody is going to merge it :) | 20:23 |
| armax | dansmith: my bad | 20:23 |
| dansmith | s'ok | 20:23 |
| andreaf | mtreinish, mriedem: well it seems that https://github.com/openstack/tempest/blob/c8aa34b05bac2167f92c3cbf7247177dfa844835/tempest/scenario/manager.py#L663 assumes network_for_ssh is set, and we're not going to use a dynamically generated network for ssh checks | 20:23 |
| sdague | hmmm... I don't see anything in the experimental queue though | 20:23 |
| andreaf | mtreinish, mriedem: it may be it's not a valid assumption, we could fallback to the tenant dynamic network if ssh_for_network is not set | 20:24 |
| sdague | but I also feel like I've had multiple instances of it dropping check experimental on the floor recently | 20:24 |
| openstackgerrit | Sean Dague proposed openstack-dev/devstack: disable metering extension in tempest if q-metering not running https://review.openstack.org/352588 | 20:24 |
| armax | sdague: how long does it take for an experimental run to execute? | 20:24 |
| sdague | well... right now, many hours | 20:25 |
| *** spzala has quit IRC | 20:25 | |
| mriedem | andreaf: i'm going to throw up a simple POC quick | 20:25 |
| *** spzala has joined #openstack-qa | 20:27 | |
| *** tqtran has joined #openstack-qa | 20:29 | |
| *** gcheresh has quit IRC | 20:29 | |
| *** nwkarsten has joined #openstack-qa | 20:31 | |
| *** edtubill has joined #openstack-qa | 20:31 | |
| sdague | mtreinish: "HistoricalBarChart" ? | 20:31 |
| sdague | mtreinish: or TimeSeries | 20:32 |
| mtreinish | sdague: ok, I can push a patch to swap it over to one of those | 20:32 |
| lifeless | andreaf: don't do class level setup/cleanup - its a really really bad idea | 20:36 |
| *** mordred has joined #openstack-qa | 20:36 | |
| lifeless | andreaf: use either testscenarios or testresources | 20:36 |
| *** vhosakot has joined #openstack-qa | 20:37 | |
| mtreinish | timothyb89: if you get a sec: https://review.openstack.org/#/c/351842/ should be a fast review | 20:37 |
| mtreinish | david-lyle: ^^^ | 20:38 |
| *** rossella_s has quit IRC | 20:38 | |
| openstackgerrit | Matt Riedemann proposed openstack/tempest: Check for dynamic creds / isolated network when getting IP for ssh https://review.openstack.org/352596 | 20:38 |
| mriedem | andreaf: ^ WIP | 20:38 |
| *** rossella_s has joined #openstack-qa | 20:38 | |
| *** moravec1 has joined #openstack-qa | 20:40 | |
| *** krtaylor has joined #openstack-qa | 20:43 | |
| *** moravec has quit IRC | 20:43 | |
| *** moravec1 has quit IRC | 20:44 | |
| *** krtaylor has quit IRC | 20:46 | |
| *** gyee has quit IRC | 20:46 | |
| ccarmack | mriedem: I'm going to grep to see what uses network_for_ssh | 20:47 |
| *** gyee has joined #openstack-qa | 20:47 | |
| *** hichihara has joined #openstack-qa | 20:47 | |
| mriedem | ccarmack: https://github.com/openstack/tempest/blob/e07579c6034ed8b2cd51ddefeff4b600691088bb/tempest/api/compute/base.py#L357 | 20:47 |
| mriedem | same thing in the compute API tests | 20:47 |
| mriedem | ccarmack: which is why i don't want to set run_validation=True yet for us | 20:48 |
| mriedem | run_validation is used in the API tests | 20:48 |
| mriedem | and i think it's only tested in the CI upstream with floating IPs | 20:48 |
| mtreinish | mriedem: when we get it turned on by default in tempest we're going to most likely have scenario respect it too | 20:48 |
| mriedem | respect your authority?! | 20:49 |
| *** vhosakot has quit IRC | 20:51 | |
| *** edmondsw has joined #openstack-qa | 20:51 | |
| *** jdandrea has quit IRC | 20:52 | |
| *** vhosakot has joined #openstack-qa | 20:52 | |
| ccarmack | mriedem: ok, so we don't set run_validation=true until this bug is fixed | 20:54 |
| ccarmack | if I understand this correctly.. | 20:54 |
| ccarmack | mriedem in the meantime I will test your patch | 20:54 |
| mriedem | ccarmack: among other things yeah | 20:55 |
| ccarmack | mriedem: does this same fix need to be in api/compute/base.py ? | 20:56 |
| mriedem | ccarmack: it would eventually yet | 20:57 |
| mriedem | *yeah | 20:57 |
| openstackgerrit | Merged openstack/openstack-health: De-vendorify d3 and nvd3 https://review.openstack.org/351842 | 20:57 |
| *** ihrachys has joined #openstack-qa | 20:58 | |
| *** gouthamr has quit IRC | 21:01 | |
| *** gvrangan has quit IRC | 21:03 | |
| *** raildo has quit IRC | 21:03 | |
| *** vhosakot_ has joined #openstack-qa | 21:05 | |
| *** vhosakot has quit IRC | 21:05 | |
| ccarmack | mriedem: fyi network_for_ssh defaults to public, so I have to explicitly set it to None | 21:06 |
| mriedem | ccarmack: hmm yeah | 21:06 |
| mriedem | help="Network used for SSH connections. Ignored if " | 21:06 |
| mriedem | "connect_method=floating or run_validation=false.", | 21:06 |
| mriedem | that's also not true | 21:06 |
| mriedem | it's used regardless of run_validation in the scenario tests | 21:07 |
| *** vhosakot_ has quit IRC | 21:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/tempest: Fix network_for_ssh config option help https://review.openstack.org/352604 | 21:08 |
| *** yamamoto has joined #openstack-qa | 21:09 | |
| *** _nadya_ has joined #openstack-qa | 21:09 | |
| *** weshay has quit IRC | 21:10 | |
| *** yamamoto has quit IRC | 21:10 | |
| *** _nadya_ has quit IRC | 21:10 | |
| *** yamamoto has joined #openstack-qa | 21:10 | |
| *** _nadya_ has joined #openstack-qa | 21:11 | |
| *** _nadya__ has joined #openstack-qa | 21:13 | |
| *** _nadya_ has quit IRC | 21:13 | |
| *** yamamoto has quit IRC | 21:15 | |
| ccarmack | mriedem: its embarassing, I don't know how to set a config to None in the file | 21:18 |
| *** thiagop has quit IRC | 21:20 | |
| mriedem | [validation] | 21:20 |
| mriedem | network_for_ssh = None | 21:20 |
| *** jecarey_ has quit IRC | 21:20 | |
| timothyb89 | mtreinish: left a quick comment on https://review.openstack.org/#/c/352569 but it seems good otherwise... anything left before it's done being WIP? | 21:21 |
| *** jecarey has joined #openstack-qa | 21:21 | |
| mtreinish | mriedem: that's not the string 'None'? | 21:21 |
| *** _nadya_ has joined #openstack-qa | 21:22 | |
| mriedem | mtreinish: no | 21:22 |
| *** _nadya__ has quit IRC | 21:22 | |
| ccarmack | I tried what mriedem posted ^ | 21:22 |
| mtreinish | timothyb89: I dropped the WIP flag on it in the last rev. sdague pointed out the date wrapping makes it more difficult to read, if you know how to fix that that'd be good too :) | 21:22 |
| ccarmack | and I took that as "None" | 21:23 |
| ccarmack | ..it | 21:23 |
| mtreinish | ccarmack: so I was right? :) | 21:23 |
| timothyb89 | mtreinish: oh, I was looking at the wrong rev in gerrit :) I'll poke at the css a bit | 21:23 |
| mtreinish | timothyb89: I'm probably still missing the closing tag | 21:23 |
| ccarmack | I'm not picking sides here (A) | 21:23 |
| *** mfisher_ora has quit IRC | 21:26 | |
| *** devkulkarni1 has quit IRC | 21:27 | |
| *** dkranz has quit IRC | 21:27 | |
| mriedem | ccarmack: well 'None' isn't going to work here https://review.openstack.org/#/c/352596/1/tempest/scenario/manager.py@669 | 21:29 |
| mriedem | 'None' != None | 21:29 |
| timothyb89 | mtreinish: left a comment with a fix for word wrap | 21:32 |
| mtreinish | timothyb89: cool, thanks | 21:32 |
| *** gvrangan has joined #openstack-qa | 21:33 | |
| *** _nadya_ has quit IRC | 21:37 | |
| *** porrua has quit IRC | 21:37 | |
| *** _nadya_ has joined #openstack-qa | 21:38 | |
| *** _nadya_ has quit IRC | 21:42 | |
| *** ihrachys has quit IRC | 21:43 | |
| *** dimtruck is now known as zz_dimtruck | 21:43 | |
| *** zz_dimtruck is now known as dimtruck | 21:43 | |
| andreaf | lifeless: well we already have a lot of test class level resource defined so that's going to be difficult to change | 21:43 |
| *** ihrachys has joined #openstack-qa | 21:43 | |
| *** spzala has quit IRC | 21:49 | |
| *** rfolco has quit IRC | 21:49 | |
| andreaf | lifeless: we used testscenarios before but it generates tests dynamically, which is great in some use cases, but in integration tests it's not ideal, it adds some complexity which makes things more complicated to troubleshoot | 21:49 |
| *** dustins has quit IRC | 21:50 | |
| *** moravec has joined #openstack-qa | 21:50 | |
| *** nwkarsten has quit IRC | 21:52 | |
| *** dimtruck is now known as zz_dimtruck | 21:52 | |
| andreaf | lifeless: testresources again I don't think it would solve my problem - there are a few resources which I want to be shared across tests in a class, such as credentials and network associated with them | 21:53 |
| andreaf | lifeless: and I need to make sure they are cleaned-up at the end of the test class | 21:54 |
| *** nwkarsten has joined #openstack-qa | 21:54 | |
| andreaf | lifeless: I thought I could use fixtures, and schedule the cleanup at class tearDown | 21:55 |
| *** ihrachys_ has joined #openstack-qa | 21:55 | |
| *** mriedem has quit IRC | 21:55 | |
| *** ihrachys has quit IRC | 21:58 | |
| *** nwkarsten has quit IRC | 21:58 | |
| *** Apoorva has quit IRC | 22:00 | |
| *** Apoorva has joined #openstack-qa | 22:01 | |
| *** Apoorva has quit IRC | 22:05 | |
| *** Apoorva has joined #openstack-qa | 22:06 | |
| *** edtubill has quit IRC | 22:06 | |
| *** ccarmack has left #openstack-qa | 22:07 | |
| *** ihrachys_ has quit IRC | 22:08 | |
| *** jecarey has quit IRC | 22:10 | |
| mtreinish | larainema: I'm looking at the failure on your mysql 5.7 fix: http://logs.openstack.org/52/347752/1/check/gate-subunit2sql-python27-db-ubuntu-xenial/88963b7/console.html.gz and it looks like the py27 tests that fail are all off by 1 sec. | 22:11 |
| mtreinish | that's kinda weird | 22:11 |
| *** hichihara has quit IRC | 22:11 | |
| larainema | mtreinish: I didn't find out why cause the failure:( | 22:12 |
| larainema | Any idea about it? | 22:12 |
| *** krtaylor has joined #openstack-qa | 22:13 | |
| larainema | It happened on my dev server also | 22:14 |
| *** pglass has quit IRC | 22:15 | |
| mtreinish | larainema: I'm not sure. We should be writting a consistent value to the db since we set utcnow to a variable before hand. I would say slow machine if we didn't set it explicitly | 22:18 |
| mtreinish | larainema: the only thing I'd try is to use a hardcoded datetime instead of utcnow() | 22:18 |
| mtreinish | larainema: the tests now run on xenial so besides just mysql 5.7 we get a new package for python2.7 | 22:19 |
| larainema | mtreinish: I considered it might caused by my dev machine very slow, But it also failed on the check test | 22:23 |
| mtreinish | larainema: yeah since we set the time to a variable I find it hard to believe it's just a slow machine | 22:24 |
| mtreinish | larainema: I'm thinking it's something more subtle as part of the migration to xenial | 22:24 |
| mtreinish | larainema: I'm not able to reproduce it on my arch linux desktop | 22:24 |
| mtreinish | I'm going to spin up a xenial vm and give it a shot in there | 22:25 |
| *** yamamoto has joined #openstack-qa | 22:26 | |
| larainema | mtreinish: will handle some family stuff, back in one or two hours, ping me any updates about your tests, thanks | 22:30 |
| openstackrecheck | Console logs not available after 13:32s for keystone-coverage-db-ubuntu-xenial 350815,2,fe518af | 22:32 |
| openstackgerrit | Jerry Zhao proposed openstack-infra/devstack-gate: WIP: restart ntpd to sync time https://review.openstack.org/352621 | 22:33 |
| *** sshnaidm is now known as sshnaidm|afk | 22:38 | |
| mtreinish | larainema: so all the tests that fail set utcnow() to a variable which is used in a db write and then check that against a db read() and it's different by 1 sec | 22:39 |
| mtreinish | no other tests do that from I could see, and everything else is working | 22:39 |
| mtreinish | fungi: ^^^ have you seen that on xenial at all | 22:39 |
| fungi | mtreinish: not anything that's come to my attention before now, no | 22:40 |
| fungi | mtreinish: some sort of off-by-one error maybe? | 22:40 |
| mtreinish | fungi: https://github.com/openstack-infra/subunit2sql/blob/master/subunit2sql/tests/db/test_api.py#L455-L475 and http://logs.openstack.org/52/347752/1/check/gate-subunit2sql-python27-db-ubuntu-xenial/88963b7/console.html.gz#_2016-07-27_15_57_49_627450 | 22:40 |
| mtreinish | fungi: yeah, that's as probable as anything at this point | 22:41 |
| fungi | got me stumped, for sure | 22:41 |
| fungi | if you can distill that a bit it might point to the culprit and we can file a (probably ubuntu) bug | 22:41 |
| mtreinish | fungi: my issue is I'm not sure if it's on the python side or mysql | 22:42 |
| fungi | right, that's what we'd need to know. file a bug against ubuntu's python package or ubuntu's mysql package | 22:43 |
| mtreinish | fungi: after the unit tests finish on my very small vm I'll try to write an interactive test case just with sqlalchemy, pymysql and xenial mysql 5.7 | 22:43 |
| mtreinish | it's definitely consistent on xenial (it worked fine for me on arch with mariadb) | 22:44 |
| mtreinish | and the 3.4 job passed in the gate (which I assume is still trusty) | 22:44 |
| fungi | fwiw the next ubuntu release will probably default to mariadb since devian's next release will | 22:45 |
| fungi | er, debian | 22:45 |
| fungi | mtreinish: yeah, 3.4 is trusty, 3.5 is xenial. have a working 3.5 job for it? | 22:45 |
| fungi | that might narrow it down if so | 22:46 |
| *** rbak has quit IRC | 22:46 | |
| mtreinish | fungi: py 3.5 failed on the same thing (but with less tests) http://logs.openstack.org/52/347752/1/check/gate-subunit2sql-python35-db-nv/03eae4b/console.html.gz | 22:47 |
| mtreinish | fungi: this is on the patch to fix the migrations for a mysql 5.7 world: https://review.openstack.org/#/c/347752/1 | 22:47 |
| mtreinish | fungi: ok, the number of tests failing on the recheck is different. There is definitely a race somewhere | 22:48 |
| mtreinish | mordred: ^^^ something mysql related, any input? :) | 22:49 |
| mordred | aroo? | 22:49 |
| mordred | oh - I had to read less scrollback than I originally expected for context | 22:50 |
| * mordred looks | 22:50 | |
| mtreinish | mordred: on xenial the subunit2sql tests fail with timestamp comparisons which are off by exactly 1 second | 22:50 |
| mordred | hrm | 22:51 |
| mordred | I was just looking at mysql docs on timestamp columns the other day | 22:52 |
| mordred | mtreinish: the most immediate thing that comes to mind is this: http://dev.mysql.com/doc/refman/5.7/en/fractional-seconds.html | 22:53 |
| mordred | but 5.6 also had fractional seconds | 22:53 |
| mtreinish | mordred: yeah, it's also not rounding because the microseconds are the same: http://logs.openstack.org/52/347752/1/check/gate-subunit2sql-python27-db-ubuntu-xenial/63d0a2f/console.html#_2016-08-08_22_37_05_720458 | 22:54 |
| mordred | oh wow - that really is in the seconds part though, not in the fractional | 22:54 |
| mordred | because that ... yeah | 22:54 |
| mordred | mtreinish: that is FASCINATING | 22:55 |
| mtreinish | mordred: so the only tests that fail are ones that use datetime.datetime.utcnow() to set a known date var. My guess is it has something to do with the microseconds, because I think that's the only time we pass sqlalchemy/pymsql/mysql a datetime object with microseconds set to something | 22:58 |
| openstackgerrit | Merged openstack/tempest: skip test_connectivity_between_vms_on_different_networks https://review.openstack.org/352463 | 22:59 |
| mordred | mtreinish: yea - I'm reading that test right now | 22:59 |
| mordred | mtreinish: I'm thinking maybe before you pass it in to SQLA, maybe do the replace(microseconds=0) there? | 23:00 |
| mordred | mtreinish: like: timestamp_a = datetime.datetime.utcnow().replace(microseconds=0) | 23:01 |
| mordred | and see if that ... | 23:01 |
| mordred | you know ... | 23:01 |
| mtreinish | yeah, I can test that | 23:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/subunit2sql: Trim microseconds before passing to sqlalchemy https://review.openstack.org/352628 | 23:03 |
| mordred | mtreinish: ^^ like that | 23:03 |
| mordred | perhaps | 23:03 |
| mtreinish | mordred: it seems to fix it for me locally | 23:06 |
| mtreinish | mordred: it'll get us around the current issue for gating, but that is still kinda weird behaviour to have to workaround | 23:07 |
| *** gouthamr has joined #openstack-qa | 23:08 | |
| *** gouthamr_ has joined #openstack-qa | 23:09 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add failed runs list to test page https://review.openstack.org/352569 | 23:10 |
| openstackrecheck | Console logs not available after 13:34s for keystone-coverage-db-ubuntu-xenial 352586,1,d14333e | 23:10 |
| mtreinish | timothyb89: ^^^ I think that should do it | 23:10 |
| mordred | mtreinish: I agree. that said - it's entirely possible that either sqlalchemy's mysql dialect isn't up to speed on 5.6/5.7 mysql datetime/timestamp columns and how to do the conversion ... or that they are but we aren't triggering the right behavior somehow | 23:10 |
| *** sabeen has quit IRC | 23:10 | |
| *** gouthamr has quit IRC | 23:13 | |
| mtreinish | mordred: fwiw, as a random data point it works fine for me on arch with mariadb 10.1.16 (which also supports fractional seconds, i'm assuming with the same syntax) | 23:20 |
| mordred | mtreinish: I was just looking in to custom type things for sqla, because hesketh has a sql reporter patch up for zuul and I was wanting to be a real dick and suggest that the uuid columns be stored as bigints instead of char(32) ... but to make that not suck everything would require a uuid custom type | 23:20 |
| *** sdague has quit IRC | 23:20 | |
| mordred | mtreinish: if I write one of those, I'll give you a copy | 23:20 |
| mordred | mtreinish: oh good. I love it when maria and mysql behave different | 23:21 |
| mtreinish | mordred: cool, we can retype all the uuid columns in subunit2sql to use that | 23:21 |
| mtreinish | right now we just treat it as a string | 23:21 |
| mordred | yah. I mean - strings are cool - but are 30 bytes more than are needed | 23:22 |
| mordred | otoh - it might or might not be a good choice for subunit2sql - depending on how frequently someone might want to do ad-hoc command line sql to look something up by uuid | 23:23 |
| *** nwkarsten has joined #openstack-qa | 23:24 | |
| mtreinish | mrodden: well besides the key value metadata stuff which might get populated with a uuid value from zuul, we only have 1 uuid column (which is just the default, it's something a user can set to any string) and it's there for a user facing id/backwards compat | 23:24 |
| mtreinish | s/mrodden/mordred | 23:25 |
| mtreinish | yeah, so I'm not sure we'd actually need it on subunit2sql anymore | 23:25 |
| mtreinish | before SpamapS's giant migration that took days to run, it would have probably made for an easier migration | 23:25 |
| mtreinish | mordred: btw, any movement on setting up a real mysql server for things. I have a couple of migration patches pending but am not pushing forward there, and am waiting on the cool new thing :) | 23:26 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/elastic-recheck: Add bug 1610960 https://review.openstack.org/352639 | 23:27 |
| openstack | bug 1610960 in neutron "Invalid input for external_gateway_info. Reason: '' is not a valid UUID." [Critical,Confirmed] https://launchpad.net/bugs/1610960 | 23:27 |
| jroll | mtreinish: super late but this is the grenade fallout thing ^ | 23:27 |
| mtreinish | jroll: heh | 23:27 |
| mordred | mtreinish: no - but thanks for remind me - I really do think we need to do it | 23:27 |
| * jroll got sidelined by failing engines | 23:28 | |
| mordred | mtreinish: I started looking in to it, then got squirrelled by looking in to how people are doing things in the modern era | 23:28 |
| mordred | then discovered that there have been no _substantive_ improvements in the space in the last 10 years that are relevant for us | 23:28 |
| mordred | then got drunk and forgot all about it | 23:28 |
| * SpamapS perks up | 23:29 | |
| mtreinish | mordred: haha, ok no worries | 23:30 |
| * mordred hands SpamapS a steak he found laying around | 23:30 | |
| mtreinish | jroll: logstash says that query has 302 hits on successes and 198 on failures in the past 7 days | 23:30 |
| mtreinish | jroll: is there a better query we can use? | 23:31 |
| * SpamapS is now distracted by the steak and cannot think about sql | 23:31 | |
| jroll | mtreinish: ouch, on success? | 23:31 |
| jroll | lemme see | 23:31 |
| * jroll clearly needs more logstash skills | 23:31 | |
| mtreinish | jroll: http://logs.openstack.org/39/320039/28/check/gate-grenade-dsvm-ubuntu-trusty/3b097ff/logs/grenade.sh.txt.gz#_2016-08-08_22_07_51_010 | 23:31 |
| mtreinish | jroll: that run didn't fail | 23:32 |
| jroll | yeah looking | 23:32 |
| jroll | so this is the line http://logs.openstack.org/26/352326/1/check/gate-grenade-dsvm-neutron-ubuntu-trusty/acf0516/logs/grenade.sh.txt.gz#_2016-08-08_12_03_12_873 | 23:32 |
| jroll | er, the one below | 23:32 |
| mordred | yah | 23:33 |
| mordred | that's the one I've been tracking | 23:33 |
| mordred | I've got a patch up to print out clouds.yaml and any OS_ env vars right before it does that | 23:33 |
| jroll | so I'm not sure how to improve that query | 23:33 |
| mordred | jroll: http://logs.openstack.org/26/352326/1/check/gate-grenade-dsvm-neutron-ubuntu-trusty/acf0516/logs/grenade.sh.txt.gz#_2016-08-08_12_03_14_135 | 23:34 |
| mordred | a couple of lines below ... I don't think there is usually a valid reason for that to be empty there? | 23:34 |
| *** zz_dimtruck is now known as dimtruck | 23:34 | |
| mordred | oh, I lied | 23:34 |
| mordred | ignore me | 23:34 |
| jroll | mordred: it's empty in mtreinish's example above | 23:34 |
| jroll | yeah | 23:34 |
| jroll | we need like... this query + that line + whatever decides if an l3 thing is enabled | 23:34 |
| mordred | yah | 23:34 |
| mtreinish | jroll: well we can use the tempest error message if there is nothing else | 23:35 |
| jroll | can e-r match on multiple lines? | 23:35 |
| mtreinish | jroll: no, unless it's part of a traceback | 23:35 |
| mtreinish | jroll: a line is an elasticsearch document (except for tracebacks where we explicitly say use the whole traceback) | 23:35 |
| jroll | right right | 23:35 |
| jroll | aha | 23:36 |
| mtreinish | jroll: if we use the tempest error message we just have to make sure it's not too generic so it matches all sorts of random failures | 23:36 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/elastic-recheck: Add bug 1610960 https://review.openstack.org/352639 | 23:36 |
| openstack | bug 1610960 in neutron "Invalid input for external_gateway_info. Reason: '' is not a valid UUID." [Critical,Confirmed] https://launchpad.net/bugs/1610960 | 23:36 |
| jroll | right, this should match fairly close | 23:36 |
| jroll | there's ways to trigger that without this failure, of course, but it seems relevant | 23:37 |
| *** nwkarsten has quit IRC | 23:37 | |
| *** markvoelker has quit IRC | 23:39 | |
| mtreinish | jroll: yeah that's probably good enough. So the underlying issue is devstack/grenade fails to set the public network and router ids in the tempest.conf: http://logs.openstack.org/67/347867/15/check/gate-grenade-dsvm-neutron-multinode/c253933/logs/new/tempest_conf.txt.gz | 23:40 |
| *** mtanino has quit IRC | 23:40 | |
| mtreinish | I think that query is about as good as we can get, we do have the multiline traceback available in the tempest log file, but it doesn't give us any better context | 23:41 |
| jroll | correct | 23:41 |
| *** fnaval has quit IRC | 23:42 | |
| mtreinish | jroll: +A | 23:43 |
| jroll | thanks | 23:43 |
| * jroll crosses "make an e-r patch" off his bucket list | 23:43 | |
| mordred | mtreinish: and it fails to set those because it can't discover them, right? | 23:44 |
| mordred | due to the "Cannot use v2 authentication with domain scope" command line issue | 23:44 |
| jroll | correct | 23:44 |
| mordred | cool. just making sure I'm still barking up the right tree | 23:46 |
| mordred | which I'm 95% sure has to do with env vars and the neutron client in ways I don't fully appreciate | 23:46 |
| *** nwkarsten has joined #openstack-qa | 23:46 | |
| *** rfolco has joined #openstack-qa | 23:46 | |
| mtreinish | jroll: what were causing the error messages on success then? | 23:47 |
| jroll | mtreinish: jobs that don't use an L3 plugin? | 23:47 |
| jroll | which I think means floating IPs disabled | 23:47 |
| openstackrecheck | openstack/cinder change: https://review.openstack.org/346094 failed gate-tempest-dsvm-neutron-full-ubuntu-xenial in the gate queue with an unrecognized error | 23:48 |
| jroll | maybe gate-grenade-dsvm doesn't use floating IPs, or uses nova-net? | 23:48 |
| jroll | I think it uses nova-net but idk with all the changes lately | 23:48 |
| mtreinish | jroll: it uses nova net | 23:48 |
| jroll | that would do it | 23:48 |
| mtreinish | jroll: the only jobs that use neutron have neutron in the name | 23:48 |
| mtreinish | jroll: it raises the question why do we try to call the neutron api if it's not installed | 23:49 |
| mtreinish | but, one thing at a time | 23:49 |
| jroll | mtreinish: heh, great question | 23:49 |
| jroll | mtreinish: I guess this just needs an is_service_enabled https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L243 | 23:51 |
| JayF | mtreinish: while Ironic has your attention, mind pointing a review at https://review.openstack.org/#/c/350795/ -- I have someone in OSIC working on factoring those vars outta ds-g that don't need to be there (for Ironic), but in the meantime we'd love to change our default | 23:54 |
| mtreinish | JayF: +1 | 23:56 |
| JayF | mtreinish: who has core on that, if not you? | 23:57 |
| * JayF looks it up | 23:57 | |
| mtreinish | JayF: infra-core and sdague | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
