Friday, 2017-03-17

*** yangyapeng has quit IRC00:02
*** catintheroof has joined #openstack-qa00:03
openstackgerritMerged openstack-infra/devstack-gate master: Keep working to clean up PROJECTS list  https://review.openstack.org/44662300:04
*** ihrachys has quit IRC00:13
*** catintheroof has quit IRC00:16
openstackrecheckConsole logs not available after 20:52s for gate-kolla-dsvm-build-centos-source-centos-7 442923,1,e79e4cb00:16
*** tosky has quit IRC00:18
*** zz_dimtruck is now known as dimtruck00:19
*** csomerville has joined #openstack-qa00:35
*** cody-somerville has quit IRC00:35
*** amotoki has joined #openstack-qa00:37
openstackrecheckConsole logs not available after 21:22s for gate-rally-docs-ubuntu-xenial 370040,2,87db3ad00:38
*** gouthamr has joined #openstack-qa00:38
*** amotoki has quit IRC00:41
openstackrecheckConsole logs not available after 21:46s for gate-octavia-pep8-ubuntu-xenial 441912,2,eafc95500:59
gmannoomichi: masayukig andreaf can you look into those - https://review.openstack.org/#/c/409634/  https://review.openstack.org/#/c/413426/01:00
*** catintheroof has joined #openstack-qa01:02
*** catintheroof has quit IRC01:02
*** catintheroof has joined #openstack-qa01:02
*** xiongjh1 has joined #openstack-qa01:04
*** gvrangan has quit IRC01:05
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes failing flavor access tests for Member role.  https://review.openstack.org/44679901:06
*** Apoorva has quit IRC01:06
*** armax has quit IRC01:13
*** yangyapeng has joined #openstack-qa01:16
*** sshnaidm is now known as sshnaidm|off01:19
openstackrecheckConsole logs not available after 20:29s for gate-tempest-dsvm-murano-api-ubuntu-xenial 442936,1,d72512601:20
*** vhosakot has joined #openstack-qa01:28
openstackrecheckConsole logs not available after 20:26s for gate-osc-dsvm-functional-ubuntu-xenial 440150,4,8eab60b01:40
*** amotoki has joined #openstack-qa01:45
*** xiongjh1_ has joined #openstack-qa01:47
*** xiongjh1 has quit IRC01:47
*** amotoki has quit IRC01:50
*** xiongjh1_ has quit IRC01:53
*** xiongjh1_ has joined #openstack-qa01:53
*** jamielennox is now known as jamielennox|away01:56
*** rfolco_ has joined #openstack-qa01:57
*** rfolco_ has quit IRC01:57
openstackrecheckConsole logs not available after 20:37s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 442691,3,79e294202:01
*** jamielennox|away is now known as jamielennox02:01
*** xiongjh1_ has quit IRC02:04
*** xiongjh1_ has joined #openstack-qa02:05
*** dimtruck is now known as zz_dimtruck02:06
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes instance actions compute rbac test failing for Member role.  https://review.openstack.org/44680802:09
openstackgerritFelipe Monteiro proposed openstack/patrole master: Partially revert removal of time.sleep if v3 auth enabled in conf.  https://review.openstack.org/44681302:20
openstackrecheckConsole logs not available after 20:02s for gate-networking-sfc-functional-dsvm-ubuntu-xenial 442140,4,13f833102:21
*** vhosakot has quit IRC02:32
*** vhosakot has joined #openstack-qa02:32
*** yamahata has quit IRC02:35
*** catintheroof has quit IRC02:36
openstackrecheckConsole logs not available after 20:06s for gate-networking-sfc-functional-dsvm-ubuntu-xenial 442882,6,786ed0102:41
*** hongbin_ has joined #openstack-qa02:50
*** gouthamr has quit IRC03:01
openstackrecheckConsole logs not available after 20:05s for gate-barbican-pep8-ubuntu-xenial 442344,3,59a019c03:01
*** ihrachys has joined #openstack-qa03:14
*** ihrachys has quit IRC03:15
openstackrecheckConsole logs not available after 20:06s for gate-congress-dsvm-api-mysql-ubuntu-xenial 442249,1,803754c03:21
*** ihrachys has joined #openstack-qa03:22
*** ihrachys has quit IRC03:23
*** zz_dimtruck is now known as dimtruck03:27
openstackrecheckConsole logs not available after 20:04s for gate-tacker-pep8-ubuntu-xenial 424425,6,d9c4adc03:42
*** vhosakot_ has joined #openstack-qa03:44
*** vhosakot has quit IRC03:46
*** amotoki has joined #openstack-qa03:46
*** amotoki has quit IRC03:51
*** markvoelker_ has quit IRC03:53
*** udesale has joined #openstack-qa04:00
openstackrecheckConsole logs not available after 20:04s for gate-tempest-dsvm-networking-sfc-ubuntu-xenial 442882,7,ffe3c3304:02
*** yamahata has joined #openstack-qa04:13
*** alex_xu has quit IRC04:17
openstackrecheckConsole logs not available after 20:13s for gate-networking-sfc-python27-ubuntu-xenial 442883,5,f9f7c4804:22
*** alex_xu has joined #openstack-qa04:23
*** dimtruck is now known as zz_dimtruck04:35
*** hongbin_ has quit IRC04:39
openstackrecheckConsole logs not available after 20:06s for gate-cross-cinder-python27-ubuntu-xenial 442061,5,878bbd204:42
*** ihrachys has joined #openstack-qa04:49
*** ihrachys has quit IRC04:50
*** xiongjh1_ has quit IRC04:51
*** cody-somerville has joined #openstack-qa04:54
*** csomerville has quit IRC04:54
*** csomerville has joined #openstack-qa04:58
*** cody-somerville has quit IRC04:58
*** yangyape_ has joined #openstack-qa05:00
openstackrecheckConsole logs not available after 20:07s for gate-tempest-dsvm-networking-sfc-ubuntu-xenial 442140,5,9b6240405:02
*** yangyapeng has quit IRC05:02
*** csomerville has quit IRC05:12
openstackgerritJulian Edwards proposed openstack-dev/devstack master: Make running_in_container work with LXD  https://review.openstack.org/44621705:21
openstackrecheckConsole logs not available after 20:03s for gate-nova-pep8-ubuntu-xenial 442295,10,cd2909405:22
*** shausy has joined #openstack-qa05:34
*** erlon has quit IRC05:35
*** nadya has joined #openstack-qa05:41
openstackrecheckConsole logs not available after 20:04s for gate-networking-odl-python27-ubuntu-xenial 334250,3,e1ff32e05:42
*** nadya has quit IRC05:46
*** yangyape_ has quit IRC05:48
*** yangyapeng has joined #openstack-qa05:51
openstackrecheckConsole logs not available after 20:04s for gate-rally-docs-ubuntu-xenial 370040,3,e5efcd006:02
*** yangyapeng has quit IRC06:06
*** gvrangan has joined #openstack-qa06:06
*** tuanluong has joined #openstack-qa06:11
*** gcheresh_ has joined #openstack-qa06:13
*** k4n0 has joined #openstack-qa06:15
*** yangyapeng has joined #openstack-qa06:18
openstackrecheckConsole logs not available after 20:04s for gate-tempest-dsvm-monasca-python-mysql-full-ubuntu-xenial 440847,8,6da868b06:22
*** yangyapeng has quit IRC06:24
*** yangyapeng has joined #openstack-qa06:26
*** abehl has joined #openstack-qa06:32
*** sridharg has joined #openstack-qa06:37
*** gcheresh_ has quit IRC06:38
*** zhouyf6_ has left #openstack-qa06:40
*** pgadiya has joined #openstack-qa06:42
openstackrecheckConsole logs not available after 20:11s for gate-cliff-python27-ubuntu-xenial 442419,1,8f96b4706:43
*** pgadiya has left #openstack-qa06:45
*** r-mibu has quit IRC06:47
*** r-mibu has joined #openstack-qa06:52
openstackgerritJeremy Zhang proposed openstack/tempest master: Add test case for update volume encryption type  https://review.openstack.org/44686407:02
openstackrecheckConsole logs not available after 20:20s for gate-taskflow-python35 442949,1,7ee6fcd07:03
*** k4n0 has quit IRC07:06
*** markus_z has joined #openstack-qa07:12
*** e0ne has joined #openstack-qa07:16
*** JayF has quit IRC07:20
*** nadya has joined #openstack-qa07:21
openstackrecheckConsole logs not available after 20:07s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 442905,1,9151b5607:23
*** jbernard has quit IRC07:23
*** jbernard has joined #openstack-qa07:23
*** vhosakot_ has quit IRC07:25
*** zz_dimtruck is now known as dimtruck07:26
*** nadya_ has joined #openstack-qa07:27
*** nadya has quit IRC07:29
*** JayF has joined #openstack-qa07:33
*** tesseract has joined #openstack-qa07:34
*** dimtruck is now known as zz_dimtruck07:36
*** zhufl has quit IRC07:36
*** zhufl has joined #openstack-qa07:37
*** e0ne has quit IRC07:40
*** abehl has quit IRC07:42
*** pcaruana has joined #openstack-qa07:43
openstackrecheckConsole logs not available after 20:06s for gate-networking-odl-python27-ubuntu-xenial 442872,3,f151a9a07:43
*** amotoki has joined #openstack-qa07:48
*** amotoki has quit IRC07:52
openstackgerritGhanshyam Mann proposed openstack/tempest master: Separate volume v2 transfers service clients  https://review.openstack.org/43800607:53
*** tmatsu has quit IRC07:59
*** tmatsu has joined #openstack-qa08:00
*** hferenc has quit IRC08:01
openstackrecheckConsole logs not available after 20:10s for gate-trove-functional-dsvm-mysql-ubuntu-xenial 442871,1,6e7f8d308:03
*** tmatsu has quit IRC08:04
*** rossella_s has joined #openstack-qa08:05
*** gfidente has joined #openstack-qa08:11
*** gcheresh_ has joined #openstack-qa08:13
*** gcheresh_ has quit IRC08:18
*** gcheresh_ has joined #openstack-qa08:23
openstackrecheckConsole logs not available after 20:05s for gate-telemetry-dsvm-integration-panko-ubuntu-xenial 442293,2,cfdc5ac08:23
*** tmatsu has joined #openstack-qa08:24
*** tmatsu has quit IRC08:28
*** abehl has joined #openstack-qa08:30
*** shausy has quit IRC08:33
*** shausy has joined #openstack-qa08:34
*** amoralej|off is now known as amoralej08:38
*** udesale has quit IRC08:41
*** udesale has joined #openstack-qa08:41
*** aarefiev_afk is now known as aarefiev08:43
openstackrecheckConsole logs not available after 20:05s for gate-cross-cinder-python27-ubuntu-xenial 442061,5,d34730208:44
*** hferenc has joined #openstack-qa08:47
*** k4n0 has joined #openstack-qa08:54
*** yamamoto has quit IRC08:57
*** shardy has joined #openstack-qa09:01
openstackrecheckConsole logs not available after 20:10s for gate-heat-dsvm-functional-convg-mysql-lbaasv2-non-apache-ubuntu-xenial 442943,1,b8ee5e309:04
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543509:06
*** Serlex has joined #openstack-qa09:06
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543509:09
*** dtantsur|sick is now known as dtantsur09:10
*** david-lyle_ has joined #openstack-qa09:18
*** david-lyle has quit IRC09:18
openstackrecheckConsole logs not available after 20:16s for gate-ironic-docs-ubuntu-xenial 441625,3,0b0ccc009:24
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543509:31
prateekping andreaf, please let me know when you have some bandwidth to discuss about https://review.openstack.org/#/c/445419/09:32
prateekping dpaterson, please let me know when you have some bandwidth to discuss about https://review.openstack.org/#/c/445419/09:32
*** yamamoto has joined #openstack-qa09:35
*** yamamoto has quit IRC09:38
*** yamamoto has joined #openstack-qa09:38
*** gcheresh_ has quit IRC09:41
*** yamamoto has quit IRC09:42
openstackrecheckConsole logs not available after 20:08s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 433497,6,024e0bf09:44
*** amotoki has joined #openstack-qa09:48
*** amotoki has quit IRC09:53
*** e0ne has joined #openstack-qa09:54
*** nadya_ has quit IRC09:56
*** mriedem has quit IRC09:58
*** mriedem has joined #openstack-qa09:58
*** nadya has joined #openstack-qa10:03
*** mriedem has quit IRC10:03
*** rcernin has joined #openstack-qa10:04
openstackrecheckConsole logs not available after 20:11s for gate-trove-functional-dsvm-mysql-ubuntu-xenial 412299,5,8df684110:04
*** tosky has joined #openstack-qa10:05
*** aojea has joined #openstack-qa10:10
*** yangyapeng has quit IRC10:16
*** shausy has quit IRC10:16
*** shausy has joined #openstack-qa10:16
openstackrecheckConsole logs not available after 20:07s for gate-os-client-config-requirements 436388,3,9c0953810:25
openstackgerritMerged openstack-dev/devstack master: Updated from generate-devstack-plugins-list  https://review.openstack.org/44633310:27
andreafprateek: hi - what's up?10:31
prateekHi andreaf, thanks for replying10:32
prateeki am working on adding a --save-state option in tempest run10:32
andreafyes I remember we talked about it10:33
prateekand making save state as default option like tempest run --no-save-state and only then the save state would be skipped10:33
prateekwhile i am able to run tempest run and the save state works fine in cmd10:34
prateekandreaf, that is good then, i would have less explaining10:34
prateek:)10:34
prateekbut the UT fails in test_run.py10:34
andreafprateek: so while I understand the reason why you want that to be the default, I don't think we should make it default - not yet at least - because that would have an impact on the workflow of people using tempest run today10:35
toskyuhm, why?10:36
prateekandreaf, okk, i would change that, another thing i wanted to ask was the failing UT10:36
prateekhttp://paste.openstack.org/show/603102/10:36
prateekthe cmd is successful but the UT fails10:36
toskyif the difference is just a file more with the saved state, people who don't use it will not even notice10:36
andreaftosky: well I'm not sure - it will add time to the run for one thing10:37
prateektosky, that sounds good, the difference is just a new file being created10:37
toskyandreaf: is it so slow? :)10:37
andreaftosky: I'm don10:37
*** cdent has joined #openstack-qa10:38
andreaftosky: I don't know I guess it may depend on what it's in your cloud10:38
toskyI don't remember it being noticeably slow - the discovery phase of the tests can be slow as well10:38
prateekandreaf, tosky , it is pretty quick when i run10:38
toskyprateek: can you try it with a big cluster with more resources?10:38
prateekandreaf, yup , should depend on the cloud as well10:38
prateektosky, sure would do10:38
prateekandreaf, another thing i wanted to ask was how do i debug a tempest test, since it does not mock and uses python module to run the command10:39
prateekhttp://paste.openstack.org/show/603102/10:39
prateekandreaf, tosky ^10:39
prateekit says that the test fails when it runs tempest run --regex passing10:40
prateekin the UT10:40
prateekbut when i run it on the command line it runs fine10:40
prateekandreaf, tosky , this is when i run tempest run --regex passing10:41
prateekhttp://paste.openstack.org/show/603103/10:41
*** yamamoto has joined #openstack-qa10:41
prateekand this is the failed UT saying that tempest run --regex passing fails http://paste.openstack.org/show/603102/10:42
andreafprateek: yeah I don't like those unit tests much, they are very platform specific I think - I had problems before with them running on macos for instance10:42
prateekandreaf, :/10:42
andreafprateek: but at least they keep the run command from breaking :)10:42
andreafprateek: you're welcome to refactor them and not use subprocess if you want10:43
prateekandreaf, that sounds good, would give it a shot10:43
prateekthanks !10:43
*** tuanluong has quit IRC10:44
andreafprateek: it terms of debugging, try to debug up to where the subprocess is invoked and check the exact command invoked10:44
andreafprateek: that should allow you to reproduce the issue10:44
openstackrecheckConsole logs not available after 20:08s for gate-ec2-api-python35 442779,1,80efa1d10:45
andreafprateek: note that if you have a tempest.conf in place that may hide issues in the unit tests10:45
*** edannon has joined #openstack-qa10:45
prateekandreaf, i tried entering it using s in pdb and was lost then10:45
andreafprateek: when unit tests are executed there is no tempest.conf in place, so make sure you don't have one when trying to reproduce the issue10:45
prateekandreaf, yup, i have a tempest.conf, i would try without that10:45
prateekand that might be the reason it would be pulling values i dont have otherwise10:46
prateekandreaf, ahhh10:46
andreafprateek: heh I'm lazy I usually debug with pycharm :P10:46
*** rcernin has quit IRC10:46
prateekandreaf, i got a pycharm professional today only so would try doing that with pycharm today :)10:46
andreafheh it's handy for debugging I think10:47
prateekthanks andreaf :)10:47
andreafprateek: thank you for working on that10:47
prateek:)10:47
*** udesale has quit IRC10:48
*** yamahata has quit IRC10:54
openstackrecheckConsole logs not available after 20:12s for gate-grenade-dsvm-ceilometer-ubuntu-xenial 442781,1,9701c9811:05
openstackrecheckConsole logs not available after 20:14s for gate-neutron-pep8-ubuntu-xenial 442375,3,e77ee2711:25
*** yangyapeng has joined #openstack-qa11:26
*** aojea_ has joined #openstack-qa11:28
*** aojea has quit IRC11:31
*** gcheresh_ has joined #openstack-qa11:32
openstackrecheckConsole logs not available after 20:17s for gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial 442215,3,91e8a7711:45
*** gszasz has joined #openstack-qa11:48
*** amotoki has joined #openstack-qa11:49
*** shausy has quit IRC11:51
*** shausy has joined #openstack-qa11:51
*** amotoki has quit IRC11:53
*** shardy is now known as shardy_lunch12:04
*** nadya has quit IRC12:04
openstackrecheckConsole logs not available after 20:16s for gate-neutron-python27-ubuntu-xenial 411741,21,a81f0bf12:06
*** aysyd has joined #openstack-qa12:06
*** rossella_s has quit IRC12:11
*** rossella_s has joined #openstack-qa12:12
*** edannon has quit IRC12:13
*** bkopilov has quit IRC12:16
*** Serlex has quit IRC12:17
openstackrecheckConsole logs not available after 20:14s for gate-ec2-api-pep8-ubuntu-xenial 361172,14,8f8056112:26
*** nadya has joined #openstack-qa12:31
*** erlon has joined #openstack-qa12:33
*** markvoelker has joined #openstack-qa12:35
openstackgerritJordan Pittier proposed openstack-infra/devstack-gate master: Don't run nova-cert by default  https://review.openstack.org/44698612:39
*** fandi has joined #openstack-qa12:39
openstackrecheckConsole logs not available after 20:09s for gate-rally-python27-ubuntu-xenial 370040,4,41aa28412:46
*** nadya has quit IRC12:47
*** shausy has quit IRC12:48
*** felipemonteiro_ has joined #openstack-qa12:48
openstackgerritMerged openstack-dev/devstack stable/ocata: Fix install_libvirt for other RHEL-based distros  https://review.openstack.org/44539512:49
*** dustins has joined #openstack-qa12:51
*** mriedem has joined #openstack-qa12:55
*** yamamoto has quit IRC12:55
*** yamamoto has joined #openstack-qa12:56
*** awaugama has joined #openstack-qa12:58
openstackrecheckConsole logs not available after 20:09s for gate-kuryr-kubernetes-python27-ubuntu-xenial 436877,7,62f42d113:06
*** catintheroof has joined #openstack-qa13:10
*** catintheroof has quit IRC13:10
*** catintheroof has joined #openstack-qa13:10
*** nadya has joined #openstack-qa13:11
*** tmatsu has joined #openstack-qa13:11
*** esberglu has joined #openstack-qa13:14
*** tmatsu has quit IRC13:16
*** bkopilov has joined #openstack-qa13:18
*** nadya has quit IRC13:20
*** gouthamr has joined #openstack-qa13:23
*** nadya has joined #openstack-qa13:25
openstackrecheckConsole logs not available after 20:11s for gate-cinder-pep8-ubuntu-xenial 442288,5,5ce162c13:27
*** ihrachys has joined #openstack-qa13:27
*** nadya_ has joined #openstack-qa13:27
*** nadya has quit IRC13:28
*** amotoki has joined #openstack-qa13:29
*** rfolco has joined #openstack-qa13:29
*** dave-mccowan has joined #openstack-qa13:30
*** nadya_ has quit IRC13:33
*** amotoki has quit IRC13:33
*** nadya has joined #openstack-qa13:34
*** bkopilov has quit IRC13:34
*** ihrachys has quit IRC13:38
*** ihrachys_ has joined #openstack-qa13:38
*** weshay_pto is now known as weshay_sick13:40
*** aysyd has quit IRC13:46
openstackrecheckConsole logs not available after 20:06s for gate-tempest-dsvm-neutron-full-ubuntu-xenial-ocata 442905,1,c1ef51013:47
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate master: Revert "Keep working to clean up PROJECTS list"  https://review.openstack.org/44700413:47
*** bkopilov has joined #openstack-qa13:48
openstackgerritJoe Hakim Rahme proposed openstack/tempest master: Add tests for compute v2.40 microversion  https://review.openstack.org/44700513:48
openstackgerritDavid Rabel proposed openstack-dev/devstack master: Create /opt/stack and make it home directory  https://review.openstack.org/44700613:49
*** gvrangan has quit IRC13:55
*** shardy_lunch is now known as shardy13:56
*** dustins has quit IRC13:59
*** jdandrea has joined #openstack-qa14:01
*** diltram has quit IRC14:05
openstackrecheckConsole logs not available after 20:10s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 433497,7,328113614:07
*** hongbin has joined #openstack-qa14:08
*** jdandrea has quit IRC14:16
*** nadya has quit IRC14:19
*** prateek has quit IRC14:22
*** diltram_ has joined #openstack-qa14:23
*** Serlex has joined #openstack-qa14:25
*** gcheresh_ has quit IRC14:26
openstackrecheckConsole logs not available after 20:15s for gate-cliff-python27-ubuntu-xenial 442419,2,d51935314:27
openstackgerritMerged openstack-dev/devstack master: Added list of mlock-using processes to peakmem_tracker output  https://review.openstack.org/43447014:31
*** zz_dimtruck is now known as dimtruck14:31
*** amoralej is now known as amoralej|lunch14:34
openstackgerritDavid Rabel proposed openstack-dev/devstack master: Create /opt/stack and make it home directory  https://review.openstack.org/44700614:35
sdaguejlvillal: is this a change you want to backport - https://review.openstack.org/#/c/44517614:35
jlvillalsdague, I can if desired.14:36
jlvillalsdague, I had noticed it when looking at log files14:36
openstackgerritSean Dague proposed openstack-dev/devstack master: Make running_in_container work with LXD  https://review.openstack.org/44621714:38
sdaguejlvillal: your call, I wasn't sure if you wanted it back on stable branches for ironic14:39
sdagueit's the kind of targetted changes that I will often take all the way back so we don't have to ponder that master and stable work differently14:39
openstackgerritJohn L. Villalovos proposed openstack-dev/devstack stable/ocata: Use string comparison to compare UUID values  https://review.openstack.org/44701514:40
jlvillalsdague, ^^^14:40
openstackgerritJohn L. Villalovos proposed openstack-dev/devstack stable/newton: Use string comparison to compare UUID values  https://review.openstack.org/44701614:41
jlvillalsdague, and ^^^14:41
sdaguejlvillal: thanks14:42
jlvillalsdague, You're welcome14:42
*** yamamoto has quit IRC14:43
*** tosky has quit IRC14:43
*** tosky has joined #openstack-qa14:44
*** yamamoto has joined #openstack-qa14:45
*** sc68cal has quit IRC14:47
openstackrecheckConsole logs not available after 20:08s for gate-grenade-dsvm-ceilometer-ubuntu-xenial 442781,1,e1de4de14:47
*** raj_sing- is now known as raj_singh14:49
*** yamamoto has quit IRC14:50
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate master: Revert "Keep working to clean up PROJECTS list"  https://review.openstack.org/44700414:51
*** fandi has quit IRC14:53
*** jdandrea has joined #openstack-qa14:57
*** k4n0 has quit IRC15:00
*** markus_z has quit IRC15:02
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate master: Revert "Keep working to clean up PROJECTS list"  https://review.openstack.org/44700415:04
*** Serlex has quit IRC15:06
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543515:06
openstackrecheckConsole logs not available after 20:13s for gate-python-neutronclient-python27-ubuntu-xenial 442980,1,33edd2115:07
*** mriedem is now known as mriedem_afk15:10
*** dansmith is now known as superdan15:10
sdaguedtroyer: any idea what TRACK_DEPENDS is really for?15:12
*** armax has joined #openstack-qa15:12
*** amoralej|lunch is now known as amoralej15:13
openstackgerritDavid Rabel proposed openstack-dev/devstack master: Add Debian testing to known Distros  https://review.openstack.org/44702715:15
*** dave-mccowan has quit IRC15:15
sdaguedhellmann: I realize you are about to cut out on vacation, but in looking through https://review.openstack.org/#/c/444775 it appears that the infrastructure to run split has some short comings. Because libraries getting installed that were done at a global level (which will also hit LIBS_FROM_GIT) actually need to be double installed15:21
sdagueand... as I stare at the real solution for that ... my brain mildly melts15:21
dhellmannsdague : does https://review.openstack.org/#/c/418135/ not solve that?15:25
dhellmannmaybe a similar check needs to happen somewhere else?15:26
sdaguedhellmann: it does for the LIBS_FROM_GIT thing, but it doesn't for things like install_keystone_middleware15:26
dhellmannaha, I see what you mean now15:27
dhellmannI guess I didn't realize we had anything in place installing packages that were not in requirements lists15:27
sdagueyeh, we do that based on feature selection15:27
dhellmannoy15:28
openstackrecheckConsole logs not available after 20:10s for gate-grenade-dsvm-ceilometer-ubuntu-xenial 441734,2,55503c415:28
sdaguewell, all this predates the feature flagging in setup.py15:28
dhellmannyeah15:28
*** dsariel has joined #openstack-qa15:28
sdagueand not a lot of projects moved over to it15:28
dhellmannno, I just didn't know devstack well enough to know that was a thing15:28
*** dave-mccowan has joined #openstack-qa15:28
dhellmannso that's an option, then. can we move those dependencies into "extras" in the projects and have devstack use them?15:29
sdagueanyway, I kind of want to figure out if this is going to completely block heat moving forward, or it just means they'll be missing like 10% coverage15:29
sdaguedhellmann: sure, in this case it would be a swift patch to get keystonemiddleware into an optional depends15:29
dhellmannis the only "special" case this middleware thing with swift?15:29
sdaguedhellmann: it's the only one thus found15:29
dhellmannsince swift is not yet installed under python 3?15:30
dhellmannok, so an expedient solution would be to put in a pip_install_2_and_3 function that does what I did in the setup_dev_lib function, then call that to install keystonemiddleware15:30
dhellmannI'm not sure which is better long term, that or adding extras to projects15:31
sdagueit would be, but again, this stuff is getting weird. And we said we'd delete it by end of cycle. And my experience in deleting weird stuff is people start using it bug for bug in jobs to get behavior you were not expecting15:31
dhellmannI can see not wanting to encode dependencies directly in devstack15:31
*** armax has quit IRC15:32
dhellmannso you think maybe living with the test coverage issues for now is better?15:32
sdaguedepending on how big those gaps are, yes15:32
dhellmannI think that's one of 3 reasonable options15:32
dhellmannyeah, I could live with that15:32
dhellmannI have no idea how much coverage is at issue15:33
sdagueright, I'd like that data before making a call at how much work around we'd do15:33
dhellmannnow, what do we do in queens when swift is still not running on python3?15:33
* dtroyer catching up...15:33
openstackgerritFelipe Monteiro proposed openstack/patrole master: Identity V3 rbac_base method refactor  https://review.openstack.org/44605515:33
dtroyerTRACK_DEPENDS was an early way to get the list of python deps for … something?  I think CI used it at one point, I doubt they still do15:33
dhellmannmaybe using the extras solution is better, long term15:33
openstackgerritMerged openstack/patrole master: Fixes instance actions compute rbac test failing for Member role.  https://review.openstack.org/44680815:33
dtroyersdague: ^^^15:34
sdaguedhellmann: yeh, extras helps15:34
*** Apoorva has joined #openstack-qa15:34
dhellmannis heat's setup for devstack within the devstack repo?15:34
dhellmanna 4th option is to do some custom stuff in their devstack plugin15:35
dhellmannthough that feels like something we'll end up repeating for other projects15:35
dhellmannat which point we might as well just put it right into devstack directly15:35
sdaguethough, honestly, I think if the answer is "we want the whole python ecosystem to support python3" and swift is really far behind there, and we're getting deployers that are going python3 only, we're just going to need to decide what that all means15:35
sdaguedhellmann: it's not in devstack, it's a plugin15:35
dhellmanneven if we do put extras into swift, we have to change devstack to use that target15:35
*** felipemonteiro_ has quit IRC15:35
sdagueyeh15:36
dhellmannI think I've talked myself into supporting a special case in install_swift for now.15:36
sdaguefrom a cleanliness and a "what the heck is going on" perspective, I'd kind of like devstack to be either pure python2 or python315:36
dhellmannmaybe not with a generic function, just some local logic15:36
dhellmannI would too15:36
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes failing flavor access tests for Member role.  https://review.openstack.org/44679915:37
dhellmannwe are extremely unlikely to have all projects on 3 by the end of pike15:37
dhellmannwe will have many, but not all15:37
sdaguesure15:37
dhellmannso does that mean moving the "set up swift" stuff out of devstack and putting it somewhere else? if so, where?15:37
sdaguebut the real issue is just what things do those projects depend on15:37
dhellmannor do we say that we install swift from system packages?15:38
dhellmannlike our other dependencies15:38
dhellmannthat doesn't seem ideal for integration testing15:38
sdagueyeh, I don't know15:38
sdagueanyway, lets get heat data and see15:38
dhellmannwhat other projects need swift in their tests? glance?15:39
sdaguethere is a swift backend that's not being tested in glance, but again, it's optional15:39
dhellmannok15:40
openstackgerritClark Boylan proposed openstack-infra/devstack-gate master: Handle trailing commas in source libs list  https://review.openstack.org/44703715:40
*** Serlex has joined #openstack-qa15:40
openstackgerritRick Bartra proposed openstack/patrole master: Update Cinder test that incorrectly handles 404  https://review.openstack.org/44615515:41
*** mriedem_afk is now known as mriedem15:42
openstackgerritJordan Pittier proposed openstack-dev/devstack master: Apache Keystone Template: reduce the number of processes to 3  https://review.openstack.org/44591015:43
openstackgerritClark Boylan proposed openstack-infra/devstack-gate master: Handle trailing commas in source libs list  https://review.openstack.org/44703715:45
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Install psutil before starting memory_tracker  https://review.openstack.org/44704015:45
openstackrecheckConsole logs not available after 20:16s for gate-tacker-python27-ubuntu-xenial 439719,3,4a0b1a915:48
*** yamamoto has joined #openstack-qa15:48
openstackgerritMerged openstack-dev/devstack master: Xen: restrict devstack VM use upto 8 vCPUs.  https://review.openstack.org/44534815:49
openstackgerritMerged openstack-dev/devstack master: FAQ: remove references to stable/kilo: use stable/ocata instead  https://review.openstack.org/44592115:49
openstackgerritMerged openstack-dev/devstack master: Install same packages for neutron-* services as for q-*  https://review.openstack.org/44157115:49
*** aarefiev is now known as aarefiev_afk15:51
*** aysyd has joined #openstack-qa15:56
*** yamamoto has quit IRC15:56
*** dsariel has quit IRC15:57
jrollsdague: dhellmann: jfyi, there are ironic jobs that depend on swift as a glance backend16:01
sdaguejroll: hard dependency?16:02
sdagueand, is it all of them?16:02
jrollsdague: yes, for some, not all16:02
sdaguebecause the concern mostly is about python3 porting16:02
jrollright, we'll be able to mostly test ironic on py3 without swift being happy with py316:02
jrollbut I wanted to give that data point, since doug asked16:03
jroll(it's a hard dependency of one of our deploy modes, to answer that question in more detail)16:03
*** abehl has quit IRC16:06
openstackgerritMerged openstack-dev/devstack master: Use string comparison to compare UUID values  https://review.openstack.org/44517616:07
openstackgerritMerged openstack-dev/devstack master: Remove glare from glance  https://review.openstack.org/44245216:07
openstackgerritMerged openstack-dev/devstack master: Fix IPv6 provider networks  https://review.openstack.org/42444416:07
openstackrecheckConsole logs not available after 20:17s for networking-odl-coverage-ubuntu-xenial 421127,15,cac43ad16:08
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Set up sudoers to never require password for mlock_report.py  https://review.openstack.org/44704816:09
*** nadya has joined #openstack-qa16:10
*** Apoorva has quit IRC16:12
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Install psutil before starting memory_tracker  https://review.openstack.org/44704016:15
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Set up sudoers to never require password for mlock_report.py  https://review.openstack.org/44704816:16
*** dustins has joined #openstack-qa16:18
andreafsdague: https://review.openstack.org/#/c/446986/ ?16:20
*** rossella_s has quit IRC16:22
*** aojea_ has quit IRC16:28
openstackrecheckConsole logs not available after 20:10s for gate-neutron-pep8-ubuntu-xenial 433637,10,5c33f3316:28
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Deprecate Cinder v1 API option  https://review.openstack.org/44623816:34
sdagueandreaf: +A16:35
*** sridharg has quit IRC16:36
andreafsdague: thanks!16:39
andreafsdague: and if you have a sec for another d-g patch, this is for the scenario job: https://review.openstack.org/#/c/442565/16:39
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes router external_fixed_ip tests sometimes failing with Conflict.  https://review.openstack.org/44706116:43
openstackgerritKen'ichi Ohmichi proposed openstack-dev/devstack stable/mitaka: Set Cinder's api_v3 False on Temepst setting  https://review.openstack.org/44706216:43
dhellmannjroll : thanks. I think we're going to have to figure out how to support projects that want to run tests against swift while swift needs python 2. There are several options for that, none of them particularly appealing.16:45
*** abehl has joined #openstack-qa16:47
openstackrecheckConsole logs not available after 20:10s for gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial 434431,8,664257e16:49
*** dustins has quit IRC16:53
*** dustins has joined #openstack-qa16:57
*** e0ne has quit IRC16:59
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Change Cinder api_v3 config True on default  https://review.openstack.org/44706717:00
*** armax has joined #openstack-qa17:01
oomichiandreaf: I separate cinder api config patch into https://review.openstack.org/#/c/446238/ and https://review.openstack.org/#/c/44706717:02
andreafoomichi: cool, thanks17:02
oomichiandreaf: the later needs a devstack patch: https://review.openstack.org/#/c/447062/17:03
andreafoomichi: heh sure - I would have just waited for Mitaka EOL, it's less than a month a way - but ok17:04
*** Apoorva has joined #openstack-qa17:05
oomichiandreaf: haha, a month is long for me to forget it easily :)17:06
openstackrecheckConsole logs not available after 20:10s for gate-tempest-dsvm-networking-ovn-ovs-master 422269,10,0cf2c1517:09
oomichiandreaf: btw about https://review.openstack.org/#/c/444472 Now copying of manager.py seems to stop and I guess we could remove old data_utils now17:10
oomichiandreaf: if old manager.py try to be merged after the above patch, the gate will fail and we just need to update the patch17:10
oomichiandreaf: it doesn't seem to hurt the gates of plugin projects17:11
oomichithe latest master branches of all projects don't use the old data_utils now17:12
*** dtantsur is now known as dtantsur|afk17:13
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes failing flavor access tests for Member role.  https://review.openstack.org/44679917:15
andreafoomichi: heh ok... but this data_utils cleanup is a bit getting in the way of making us able to change manager.py17:16
*** pcaruana has quit IRC17:18
andreafoomichi: but you're right the remaining patches would not even merge after https://review.openstack.org/#/c/444472 merges17:18
andreafoomichi: I'll update all that tomorrow morning which should be quiet17:19
oomichiandreaf: yeah, that takes long time than we expected. maybe because the patch is huge due to copying raw manager.py without cleanup17:20
*** nadya has quit IRC17:20
oomichiandreaf: reviewers(including me) tend to avoid *red* patches17:20
oomichiandreaf: ok, I am waiting for, thanks :)17:21
jrolldhellmann: we should have most of that code path covered soon, without swift, there would just be small bits around temporary (signed) swift URLs not covered. I'm comfortable with that while we wait for swift17:26
openstackgerritJoe Hakim Rahme proposed openstack/tempest master: Add tests for compute v2.40 microversion  https://review.openstack.org/44700517:27
openstackrecheckConsole logs not available after 20:21s for gate-functional-dsvm-magnum-swarm-ubuntu-xenial 443002,1,97778d917:29
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Install psutil before starting memory_tracker  https://review.openstack.org/44704017:34
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack master: Set up sudoers to never require password for mlock_report.py  https://review.openstack.org/44704817:35
*** csomerville has joined #openstack-qa17:36
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes server fault thrown by delete password compute rbac test.  https://review.openstack.org/44708017:43
*** yamahata has joined #openstack-qa17:44
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes server fault thrown by delete password compute rbac test.  https://review.openstack.org/44708017:44
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove set_nic_state from old remove_client  https://review.openstack.org/44708117:45
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 rbac_base method refactor  https://review.openstack.org/44605517:46
*** csomerville has quit IRC17:47
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes server fault thrown by delete password compute rbac test.  https://review.openstack.org/44708017:48
openstackgerritFelipe Monteiro proposed openstack/patrole master: Fixes server fault thrown by delete password compute rbac test.  https://review.openstack.org/44708017:48
openstackrecheckConsole logs not available after 20:11s for gate-dragonflow-python35 431422,15,7e3577117:49
*** gszasz has quit IRC17:56
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 (ext) Tests - Oauth Consumers  https://review.openstack.org/44622917:57
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove set_nic_state from old remote_client  https://review.openstack.org/44708117:58
*** gvrangan has joined #openstack-qa17:58
openstackgerritNicolas Helgeson proposed openstack/tempest master: Identity V3-ext Oauth1 Consumers Client  https://review.openstack.org/43381818:06
*** abehl has quit IRC18:06
oomichiandreaf: do we need to keep -2 on https://review.openstack.org/#/c/446238 ?18:09
openstackrecheckConsole logs not available after 20:10s for gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial 442215,3,9459e1518:10
clarkbhello tempest reviewers, can I get review on https://review.openstack.org/#/c/446205/4 ? it dumps the ssh key as configured by cirros cloud init replacement thing to help debug ssh failures18:10
clarkbyou can see it working at http://logs.openstack.org/21/446221/4/check/gate-tempest-dsvm-neutron-full-ssh/1d3199f/console.html#_2017-03-16_22_39_13_176695 where I injected an artificial failure18:10
*** armax has quit IRC18:10
*** nadya has joined #openstack-qa18:12
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 Tests - Roles  https://review.openstack.org/44287018:13
*** armax has joined #openstack-qa18:14
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 tests - Regions  https://review.openstack.org/44287318:15
*** e0ne has joined #openstack-qa18:16
*** nadya has quit IRC18:17
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove assign_static_ip from old remote_client  https://review.openstack.org/44709318:17
openstackgerritDavid Rabel proposed openstack-dev/devstack master: Do not use libvirt-bin package anymore  https://review.openstack.org/44709418:22
*** aojea has joined #openstack-qa18:22
openstackrecheckConsole logs not available after 20:09s for gate-experimental-functional-dsvm-magnum-k8s-multinode-ubuntu-xenial 443002,1,0a8102b18:30
*** gfidente has quit IRC18:37
*** ihrachys has joined #openstack-qa18:38
clarkbsdague: working on getting you numbers for the apache tuning change. For the RSS column in http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/screen-peakmem_tracker.txt.gz is that in kilobytes?18:39
clarkbihrachys_: ^ I think you may know too?18:39
*** ihrachys_ has quit IRC18:41
clarkbya I think it must be because 1.7% of 8GB is roughly 140000MB or 136ish MB18:42
*** david-lyle_ is now known as david-lyle18:43
sdagueyeh, RSS is K I'm pretty sure18:44
ihrachysclarkb: not sure. note that the service is borked right now, see patches in https://review.openstack.org/#/q/topic:collect-mlock-stats-in-gate18:44
*** e0ne has quit IRC18:45
clarkbihrachys: I'm just interested in RSS is that correct?18:46
clarkbihrachys: for specific processes18:46
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 Tests - Domains  https://review.openstack.org/44623018:47
openstackgerritClark Boylan proposed openstack-dev/devstack master: Tune apache connection limits down  https://review.openstack.org/44674118:47
clarkbsdague: ^ there18:47
ihrachysclarkb: well RSS gives you some measure, but I think it may not be ideal. see https://pythonhosted.org/psutil/#psutil.Process.memory_full_info18:48
openstackrecheckConsole logs not available after 20:22s for gate-grenade-dsvm-ceilometer-ubuntu-xenial 442703,1,aba6e9218:50
*** tesseract has quit IRC18:50
sdagueclarkb: +@18:52
clarkbthanks18:52
openstackgerritMerged openstack-dev/devstack stable/ocata: Use string comparison to compare UUID values  https://review.openstack.org/44701518:55
openstackgerritMerged openstack-infra/devstack-gate master: Don't run nova-cert by default  https://review.openstack.org/44698618:55
clarkband looks like nova-cert removal is worth ~95MB18:57
*** cody-somerville has joined #openstack-qa18:57
*** tosky has quit IRC18:58
*** catintheroof has quit IRC18:59
*** gvrangan has quit IRC18:59
*** aojea has quit IRC19:02
clarkbihrachys: sdague also other idea is/was make sure ksm is enabled (but then reading up on that I got to thinking that maybe xen locks those pages for us whcih could explain the locked memory issues?)19:02
*** csomerville has joined #openstack-qa19:02
clarkbor possibly unmerging of pages may be what triggers our OOMs?19:03
sdagueso in our use case, I can't imagine ksm is going to be super helpful19:04
sdaguethese guests are really small19:04
sdagueit's going to be a lot of work on the dedup there19:04
clarkblooks like it is already running19:04
clarkbso maybe it explains why we only OOM on xen19:04
clarkbor at least seems more common there19:05
sdagueyeh19:05
sdaguewell, on xen they need the dom019:05
*** cody-somerville has quit IRC19:05
sdaguethere is more mem overhead overall there19:06
*** catintheroof has joined #openstack-qa19:06
* clarkb watches a running gate job's /sys/kernel/mm/ksm files out of curiousity19:07
clarkbcurrently nothing is shared19:07
*** dustins has quit IRC19:08
*** amoralej is now known as amoralej|off19:09
sdagueI still don't get this OOM when we have so much swap left, something else is going on to get us there19:09
*** dustins has joined #openstack-qa19:09
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543519:09
clarkbya so you can't swap locked memory19:09
sdagueright, but is everything locked?19:09
clarkbI think that is what ihrachys is trying to figure out. However on reading up on ksm the initial implementation locked the shared pages so they couldn't be swapped. Current implementation allows swapping them but the undedeup when pages back to ram19:10
openstackrecheckConsole logs not available after 20:08s for gate-nova-python27-ubuntu-xenial 443003,2,0c4e98b19:10
clarkbso wondering if ksm under us is potentually to blame19:10
clarkbthough I guess weshouldn't care about what is under us if the VM is really VMing19:10
sdaguemaybe, but again, the most memory that could be tied up there is like 8*64M19:11
sdagueI want to understand why the python processes aren't getting swapped19:11
clarkbsdague: at the nested layer yes, but what if our hypervisor from the cloud has locked our pages and that is leaking?19:11
clarkbso we boot 10 ubuntu xenial things and run devstack on them on one hypervisor19:12
sdaguebut that would prevent host swapping19:12
clarkbthats a lot of potentially shared pages being deduped and maybe locked? (this is hand wavy at this point)19:12
clarkbsdague: yes, unless the abstraction is leaky19:12
ihrachysclarkb: yeah that's what I poke, why we don't swap19:12
sdaguethe guest doesn't coop with the host19:12
ihrachysthere is plenty of swap memory19:12
sdaguenot on linux19:12
clarkbsdague: even with Xen?19:13
clarkbsince as far as I can tell we do swap properly on KVMbacked jobs19:13
clarkbwe see that with trove on osic for example. They use gigabytes of swap at times19:13
sdagueoh... sorry, I didn't fully understand that point19:13
sdagueso this is only a rax issue19:13
clarkbI think so19:13
clarkbihrachys: ^ is that true?19:13
*** dustins has quit IRC19:14
ihrachysclarkb: that's what I heard, but I just trusted the info, haven't checked myself19:14
sdagueok, so that would be an important fact to nail down19:14
clarkbI do know we swap successfully in the trove jobs on kvm (osic)19:14
clarkbto the tune of 2 gigabytes of swap usage19:14
*** aojea has joined #openstack-qa19:14
clarkbwhich can lead trove jobs to time out but they don't seem to OOM on KVM19:15
clarkbamrith: ^ that right?19:15
*** gcheresh_ has joined #openstack-qa19:16
clarkbDefault: 100 (chosen for demonstration purposes) (guess which value appears to be used by ubuntu?)19:29
openstackrecheckConsole logs not available after 20:11s for gate-cliff-python27-ubuntu-xenial 442419,3,859830219:30
*** tosky has joined #openstack-qa19:31
*** aojea has quit IRC19:34
clarkbI guess it scans a new set of pages every 20 milliseconds19:37
*** aojea has joined #openstack-qa19:40
*** shardy has quit IRC19:44
*** vhosakot has joined #openstack-qa19:46
clarkbI keep learning things. ksmd is present as a kernel worker process, but the run file in sysfs is set to 0 not 119:47
clarkbso we mgiht actually try toggling that bit in devstack to save memory too (it should affect more than just virtual machines, but at the expense of cpu overhead)19:47
openstackrecheckConsole logs not available after 20:18s for gate-zaqar-docs-ubuntu-xenial 439944,7,9424e2619:51
openstackgerritClark Boylan proposed openstack-dev/devstack master: Use ksm if running libvirt guests  https://review.openstack.org/44711919:55
sdagueclarkb: well, follow on question, how much of our pool is rax right now?19:57
clarkboff the top of my head ~1/5?19:57
clarkbmostly ^ is an experiment for now. Want to see how it does with "real" workload19:58
*** dimtruck is now known as zz_dimtruck19:58
*** gcheresh_ has quit IRC19:58
sdaguedo we have a query somewhere for the oom?19:58
clarkbsdague: ya its top of e-r19:59
sdaguethe top is "tempest fails with ssh"19:59
clarkbhttp://status.openstack.org/elastic-recheck/#1656386 it moved since yesterday19:59
clarkbnot sure if good or bad thing19:59
clarkbre tempest fails with ssh I have https://review.openstack.org/446205 to try and debug more19:59
sdagueso on the oom it does not appear to be rax oriented20:01
sdaguehttps://dl.dropboxusercontent.com/1/view/s5hiybfg8uwbj9w/Apps/Shutter/screenshot_353.png20:02
clarkbkk20:02
sdagueso, that discounts xen being relevant20:02
clarkbya20:02
*** awaugama has quit IRC20:04
*** dave-mccowan has quit IRC20:07
*** cdent has quit IRC20:09
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Enable memory_tracker in gate for Pike  https://review.openstack.org/43447420:11
*** dustins has joined #openstack-qa20:11
openstackrecheckConsole logs not available after 21:30s for gate-tempest-dsvm-networking-odl-beryllium-snapshot-v2driver 443018,1,57afdac20:12
*** gvrangan has joined #openstack-qa20:20
clarkbsdague: ihrachys enabling ksm has some impact http://paste.openstack.org/show/603183/ but will need to compare the higher level memory consumption numbers for a better idea of how important it is20:22
openstackgerritMerged openstack/patrole master: Fixes failing flavor access tests for Member role.  https://review.openstack.org/44679920:23
clarkblooks like ~50MB at times shared? also thats a 10 second sample rate so could miss when its larger20:24
andreafoomichi: sorry I didn't have time to re-review yet - I removed the -2 I will check it tomorrow morning20:29
*** zz_dimtruck is now known as dimtruck20:30
*** k4n0 has joined #openstack-qa20:30
openstackgerritMh Raies proposed openstack/patrole master: Adding compute server tests  https://review.openstack.org/44543520:31
clarkbgot up to pages_sharing:48899 * 4KB == thats almost 200MB20:32
*** ihrachys has quit IRC20:32
oomichiandreaf: ok, tomorrow is weekend for you. So it is fine next week :)20:34
openstackrecheckConsole logs not available after 22:23s for gate-networking-bgpvpn-python27-ubuntu-xenial 425933,3,f297de120:35
openstackgerritMerged openstack-infra/devstack-gate master: Handle trailing commas in source libs list  https://review.openstack.org/44703720:41
openstackgerritNicolas Helgeson proposed openstack/patrole master: Identity V3 Tests - Domains  https://review.openstack.org/44623020:43
*** Serlex has quit IRC20:50
*** csomerville has quit IRC20:50
*** csomerville has joined #openstack-qa20:50
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate master: DNR/TMP Increase tempest concurrency to 4  https://review.openstack.org/44713620:54
openstackrecheckConsole logs not available after 21:44s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 442547,2,2ef1d5520:56
*** gcheresh_ has joined #openstack-qa21:00
openstackgerritChi Lo proposed openstack/patrole master: Enhance test_server_actions_rbac with create image actions.  https://review.openstack.org/44714121:04
*** aysyd has quit IRC21:05
*** k4n0 has quit IRC21:06
clarkbcomparing dstat output between the py35 job before adding ksm and after ksm is enabled the old ends up with 45MB free at its lowest point and ksm new ends up with 119MB free at lowest point21:07
clarkbnow its possible that n-crt accounts for most of that21:07
clarkbya n-crt is not enabled when ksm was enabled but was before. will need a new baseline to compare against21:08
openstackgerritClark Boylan proposed openstack-dev/devstack master: DNM use as baseline memory comparator  https://review.openstack.org/44714421:10
*** dustins_ has joined #openstack-qa21:11
*** dustins has quit IRC21:12
*** catintheroof has quit IRC21:16
*** k4n0 has joined #openstack-qa21:18
openstackrecheckConsole logs not available after 22:06s for gate-networking-ovn-python27-ubuntu-xenial 443057,1,f103e1a21:19
andreaflayer4 job is broken - sahara is not pre-cloned and devstack dies http://logs.openstack.org/86/437286/2/gate/gate-tempest-dsvm-layer4-ubuntu-xenial/b9a081e/logs/devstacklog.txt.gz#_2017-03-17_10_54_52_87221:21
andreaftosky: ^^^ any idea?21:21
clarkbandreaf: thats like due to a recent change in devstack-gate21:21
toskyandreaf: uhm, no idea21:22
andreafclarkb21:22
toskyit was working until yesterday21:22
andreafclarkb: oh ok so you / someone is already on top of it?21:22
clarkbsee https://git.openstack.org/cgit/openstack-infra/devstack-gate/commit/?id=bd082d261a23ab2ddc6f3b1b0f470552beb2013e21:22
clarkbno whoever is repsonsible for the job will need to fix the job and enable sahara properly21:22
toskylet me recheck which change went it21:22
clarkband if that isn't the proper way to enable sahara someone should fix devstack-gate21:22
clarkbbasically we don't want to clone and setup sahara every time, so we use the if sahara is enabled check then clone and set up sahara else dont21:23
toskyclarkb: ehm, so was sahara cloned every time??21:23
andreafclarkb: ok yeah that makes sense21:23
clarkbtosky: yes it has been for years21:24
toskyI was always a bit confused by the value of PROJECTS: so whatever was there, was always cloned?21:24
andreafclarkb, tosky: but we don't run sahara in layer4 anymore afaik, so I'm confused why it fails21:24
clarkbI am slowly trying to clean that up in devstack-gate as it results in job setup that is minutes longer than necessary21:24
clarkbtosky: yes21:24
clarkbtosky: next on the cleanup list is all of oslo21:24
clarkbsince those are only needed in the oslo src jobs21:24
clarkbbut we still clone and set them up on every single devstack job run21:24
toskyclarkb: ok, out of curiosity: if I decide to ignore DEVSTACK_GATE_SAHARA and add sahara to PROJECTS in all the sahara jobs (and sahara-dashboard when relevant), is it the same?21:25
clarkbtosky: yes, thats probably the ideal way to fix it21:25
clarkbtosky: then you can push a change to d-g to remove the special casing as it won't be necessary21:25
*** alex_xu has quit IRC21:25
toskyclarkb: I hope there won't be other changes like this, but did you send an email with the [sahara] tag before merging?21:26
clarkbtosky: no and yes there will be many more changes like this21:26
clarkbwe really need to stop doing this21:26
clarkband unfortuantely with something like 8k jobs the only real sane way to do it is try your best and fix it when things break21:26
clarkb(because I can't read 8k jobs to make sure sahara is enabled properly)21:26
toskyfine with that, but if you would have put Telles (the PTL), or Vitaly, Evgeny or me in cc in the review, we would have acted in advance21:26
toskyand the job would have been fixed not now, but even before your patch would have been merged21:27
clarkbbut we wouldn't have known21:27
toskyjust a bit of communication more, is it too much to ask?21:27
clarkbwe would have looked at it and said "its checking if sahara is enableed and cloning it. great"21:27
toskywell, it is a change which changes when Sahara is cloned, so just please put someone from the team in cc21:27
clarkbok21:28
toskyfine, now I'm going to bake a patch21:28
*** alex_xu has joined #openstack-qa21:28
andreaftosky: so for the layer4 job - that's owned by QA since it's sahara heat and trove21:28
andreaftogether21:28
andreafI would just add in there the DEVSTACK_GATE_SAHARA flag I guess?21:29
toskyandreaf: yes, or just add sahara to the PROJECTS21:29
toskymaybe even better21:29
clarkbandreaf: yes either enable sahara using the flag or add it to PROJECTS then we can remove the flag altogether once its no longer needed21:29
openstackgerritRick Bartra proposed openstack/patrole master: Enhance validation decorator with error code  https://review.openstack.org/44404221:29
toskyandreaf: what did we say about that layer4 job? Is moving away the heat test the only blocker to its removal?21:30
clarkbthe change to sahara was considered very safe because we were using the existing api to modify the behavior21:30
clarkbbut apparently that api isn't actually used?21:30
toskyandreaf: but let me check if I have to fix sahara tests too, so we can send just one review21:31
andreaftosky: well I'm not sure about the recent history of layer4 - it installs the sahara plugin which I'm not sure why21:32
toskyandreaf: and yes, I have to fix them21:32
andreaftosky: I thought it would only tests things in tempest21:32
toskyandreaf: yes, and I think mtreinish told me that it's going to be killed at some point, when everything there is out of tempest21:32
toskybut even now I'm not sure why sahara and trove bits are still there21:33
*** gouthamr has quit IRC21:33
toskyandreaf: so, I need to fix also sahara jobs, so I can send a review also for layer421:33
tosky.21:33
andreaftosky: ok thanks21:33
toskyclarkb: is heat also going to be moved behind a variable, or can I rely on its availability in PROJECTS?21:34
andreaftosky, clarkb: probably the cleaner thing would be to have no variable at all, just have the jobs maintain their list of needed project21:35
clarkbtosky: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh?id=bd082d261a23ab2ddc6f3b1b0f470552beb2013e#n61 the current plan is to have heat in the default list21:35
clarkbtosky: you can see its organized alreay based on what should go away and what should stay21:35
clarkb(the reason for that is I think heat is still a base devstack service?)21:35
*** dustins_ has quit IRC21:36
toskyI can never be sure :)21:36
toskyit's not one of the Core Six21:36
openstackgerritRick Bartra proposed openstack/patrole master: Enhance validation decorator with error code  https://review.openstack.org/44404221:36
toskyI asked thinking about tempest21:36
clarkbtosky: I'd be open to moving it too but currently the comments in the list break down the plan21:36
andreafclarkb: I'm not even sure why we have tempest-lib in there... I guess there may things that depend on it still?21:38
clarkbandreaf: well everything that was in there was because we didn't do this correctly for years and anytime we had a new project we just added it to the list21:38
clarkbthen we spent 10 minutes cloning it all and checking out the correct ref21:38
clarkbso over the last few months I have slowly been lceaning it up to reduce that overhead21:39
toskyif we consider only core projects and almost-core, there is a test-requirements.txt in nova, but I sent a patch to remove it :)21:39
toskyand we just removed tempest-lib from RDO21:39
toskyso I guess it's a candidate for the removal21:39
clarkbthe idea when I first set up taht organization was to have anything default supported by devstack in there21:39
clarkbso that default d-g worked with default devstack21:39
clarkbits likely that things ahve moved on teh devstack side since I did the split21:39
andreafclarkb: well we used to pre-fetch all repos in that list on test VM images once a day21:41
*** esberglu has quit IRC21:41
openstackrecheckConsole logs not available after 22:19s for gate-heat-pep8-ubuntu-xenial 439433,3,a9bb28621:41
*** trevormc_ has joined #openstack-qa21:41
andreafclarkb: so cloning was just a fetch of one day worth of changes21:41
andreafclarkb: did that change?21:41
clarkbandreaf: no we prefetched all the git repos onto the images once a day (we still do that)21:41
toskyoh, correction: sahara jos have been fixed by Vitaly already (Icb3c18ab955b53c028008d6c374301c1e6ca0ccd)21:41
toskyhe added DEVSTACK_GATE_SAHARA21:41
toskywe can sort out this later21:41
clarkbandreaf: but still doing the delta checking, grabbing new commits, and checking out the correct ref is slow21:42
andreafclarkb: ok yeah that's what I meant - so does it really take 10min to update all repo?21:42
clarkbandreaf: not today, but it did when I started this work21:42
clarkbandreaf: I think we can get it down to like 2 minutes or so if we get to the currently specified desired list21:42
clarkbI've already removed grenade and all of the python clients and a few other randome things21:43
andreafclarkb: ok I see - but  won't these changes affect the prefecthing? or is it the list maintained separately?21:43
clarkbthe prefetching is for every repo on gerrit21:43
andreafclarkb: ah cool21:43
clarkb(we actually filter out attic repos but effectively every repo)21:44
clarkbits part of why our images are so large21:44
andreaftosky: ok so I'll do the same for now on layer421:44
trevormc_hey guys trove is experiencing a similar issue21:44
*** gcheresh_ has quit IRC21:44
toskyandreaf: too late :)21:44
trevormc_http://logs.openstack.org/92/443692/3/check/gate-trove-functional-dsvm-mysql-ubuntu-xenial/6e64ca5/logs/devstacklog.txt.gz#_2017-03-17_16_04_09_22721:44
clarkbtrevormc_: you'll need to enable trove if you want to use trove DEVSTACK_GATE_TROVE=121:44
toskyandreaf: https://review.openstack.org/44714721:45
trevormc_yeah I saw the patch, should i submit it?21:45
* clarkb grumps that the public interfaces in devstack-gate aren't used21:45
andreaftosky: thanks!21:45
andreafsdague: this is needed to unblock tempest gate (leayer4 is busted atm): https://review.openstack.org/#/c/447147/121:46
toskyI was already halfway through the changes; I will check with Vitaly and the others about moving away from DEVSTACK_GATE_SAHARA21:46
clarkbtrevormc_: but also you can remove DEVSTACK_GATE_TROVE completely if its no longer required as a public interface then just add trove things to PROJECTS directly21:46
andreafit's a d-g patch21:46
clarkbandreaf: I can get it21:46
andreafclarkb: ok thanks21:46
andreafI was looking for a 2nd reviewer :)21:47
*** k4n0 has quit IRC21:47
clarkbI will single approve since its simple and fixes issue21:47
andreafwe can do a follow up patch to remove the DEVSTACK_GATE_SAHARA flag completely21:47
andreafclarkb: heh fine with me, thank you!21:48
andreaftosky, clarkb: thank you for a quick solution21:48
toskyandreaf: or remove the layer4 job :)21:48
toskythat was the hope21:48
andreafyeah I will work on that21:48
*** cody-somerville has joined #openstack-qa21:49
andreafI might make it non voting or so, to keep an eye on plugins when doing changes21:49
andreafbut I'm not sure yet what's the best way to do that21:49
toskywhat is the blocker? The removal of heat?21:49
*** csomerville has quit IRC21:51
andreaftosky: I haven't looked into it really21:53
andreaftosky: it used to be heat but I'll have to check21:54
*** trevormc_ has quit IRC21:54
andreafclarkb: something's wrong with the user-data script ? http://logs.openstack.org/21/446221/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/018859d/console.html#_2017-03-16_22_28_57_83874321:54
clarkbandreaf: it did that on an ood patchset21:55
clarkbbasically first char was newline so didnt match the exec prefix21:56
clarkbcurrebt patchset should work and I linked example in review21:56
toskyandreaf: I think I remember now: sahara is still there for the older branches, where sahara-tests was not used; but for sure when Mitaka is EOL it can be removed21:56
amrithclarkb, I don't think that's the reason21:56
*** gouthamr has joined #openstack-qa21:57
*** k4n0 has joined #openstack-qa21:59
andreafclarkb: didn't you want to dump ip route and ip a as well?21:59
andreafclarkb: if you are happy with the public key we can merge this change and add more data later if needed21:59
clarkbandreaf: turns out thats already dumped :) but then reading more I saw it was public key auth failing so trying to get that data22:00
clarkbthe cirros image logs net info further along in the console log I just had to read more22:00
andreafheh I just saw Jordan's comment22:00
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove set_nic_state from old remote_client  https://review.openstack.org/44708122:02
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove assign_static_ip from old remote_client  https://review.openstack.org/44709322:02
clarkbya I was chasing the wrong thing but now think its on track for being useful22:02
andreaf+2 on it22:03
openstackrecheckConsole logs not available after 22:09s for gate-telemetry-dsvm-integration-panko-ubuntu-xenial 442293,2,ec3798f22:03
clarkbafazekas: re your question its the cirros user that fails ssh so not sure the others are interesting?22:04
andreafoomichi: remote client is another thing which is used by scenario testss so if we move that to tempest.lib we will break all plugins - so you'll have to go through debcollector or something to keep it in tempest as well until it's changed on plugin side22:05
*** yamahata has quit IRC22:10
openstackrecheckConsole logs not available after 22:08s for gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial 442864,1,532f2b722:25
openstackgerritVitaly Gridnev proposed openstack-infra/devstack-gate master: Add sahara to features.yaml in subnode  https://review.openstack.org/44715822:30
*** gvrangan has quit IRC22:34
*** timello has quit IRC22:44
*** timello has joined #openstack-qa22:47
openstackrecheckConsole logs not available after 21:53s for gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial 434769,4,40600a622:47
*** dimtruck is now known as zz_dimtruck22:51
*** superdan is now known as dansmith22:57
openstackrecheckConsole logs not available after 22:16s for gate-neutron-python27-ubuntu-xenial 395504,19,876444223:09
*** aojea has quit IRC23:10
*** erlon has quit IRC23:15
*** k4n0 has quit IRC23:22
*** markvoelker has quit IRC23:26
*** zz_dimtruck is now known as dimtruck23:29
openstackrecheckConsole logs not available after 22:20s for gate-cloudkitty-python35 411860,3,72b591523:32
*** dimtruck is now known as zz_dimtruck23:33
*** zz_dimtruck is now known as dimtruck23:37
openstackgerritKen'ichi Ohmichi proposed openstack/tempest master: Remove assign_static_ip from old remote_client  https://review.openstack.org/44709323:39
openstackgerritLiang Hao proposed openstack/tempest master: Add test_create_update_network_port_security_enabled in test_networks.py  https://review.openstack.org/44583923:39
openstackgerritLiang Hao proposed openstack/tempest master: Add test_create_port_with_port_security_enabled in test_ports.py  https://review.openstack.org/44580623:41
*** yangyapeng has quit IRC23:46
openstackrecheckConsole logs not available after 23:09s for gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial 433546,10,e8d64a123:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!