| *** marun has quit IRC | 00:01 | |
| *** mlavalle has quit IRC | 00:06 | |
| *** jpich has joined #openstack-qa | 00:12 | |
| *** weshay has quit IRC | 00:13 | |
| *** dkranz has quit IRC | 00:18 | |
| *** oomichi has joined #openstack-qa | 00:35 | |
| *** jpich_ has joined #openstack-qa | 00:36 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Fix to avoid undefined variables in initial sanity checks https://review.openstack.org/117475 | 00:36 |
|---|---|---|
| *** jpich has quit IRC | 00:40 | |
| *** dimtruck is now known as zz_dimtruck | 00:46 | |
| *** dmorita has joined #openstack-qa | 00:51 | |
| *** zz_dimtruck is now known as dimtruck | 01:06 | |
| openstackgerrit | Daisuke Morita proposed a change to openstack/tempest: Add client response checking for object storage service https://review.openstack.org/117193 | 01:11 |
| *** mestery_ is now known as mestery | 01:14 | |
| openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: large ops test should be derived from OfficialClientTest https://review.openstack.org/119670 | 01:25 |
| openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: large ops test should be derived from OfficialClientTest https://review.openstack.org/119670 | 01:29 |
| *** yamahata has joined #openstack-qa | 01:36 | |
| *** Poornima has joined #openstack-qa | 01:37 | |
| openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: Migrate test_large_ops to tempest client https://review.openstack.org/113807 | 01:39 |
| *** nosnos has joined #openstack-qa | 01:51 | |
| *** dims has joined #openstack-qa | 01:54 | |
| *** jpich_ is now known as jpich | 01:57 | |
| *** HenryG_ is now known as HenryG | 02:01 | |
| *** jecarey has joined #openstack-qa | 02:21 | |
| *** bkopilov has quit IRC | 02:30 | |
| openstackgerrit | Rabi Mishra proposed a change to openstack/tempest: Tempest test changes for heat-cli changes https://review.openstack.org/104473 | 02:35 |
| *** dims has quit IRC | 02:56 | |
| *** oomichi is now known as oomichi_away | 02:57 | |
| *** oomichi_away has quit IRC | 03:02 | |
| openstackgerrit | A change was merged to openstack/tempest: javelin: implement resource destruction https://review.openstack.org/104986 | 03:05 |
| *** jecarey has quit IRC | 03:08 | |
| *** cmyster has quit IRC | 03:16 | |
| *** cmyster has joined #openstack-qa | 03:21 | |
| *** dimtruck is now known as zz_dimtruck | 03:21 | |
| *** swartulv has quit IRC | 03:23 | |
| openstackgerrit | Masayuki Igawa proposed a change to openstack/tempest: Migrate test_encrypted_cinder_volumes to tempest client https://review.openstack.org/111905 | 03:31 |
| *** chandan_kumar has joined #openstack-qa | 03:35 | |
| *** Poornima has quit IRC | 03:37 | |
| *** chandan_kumar has quit IRC | 03:44 | |
| *** bkopilov has joined #openstack-qa | 04:00 | |
| *** neeti has joined #openstack-qa | 04:01 | |
| openstackgerrit | Yuiko Takada proposed a change to openstack/tempest: Add baremetal API tests about console https://review.openstack.org/117715 | 04:12 |
| *** Poornima has joined #openstack-qa | 04:13 | |
| *** torandu has joined #openstack-qa | 04:17 | |
| *** neeti has quit IRC | 04:19 | |
| openstackgerrit | ajay proposed a change to openstack/tempest: Improve nova CLI existing tests https://review.openstack.org/101767 | 04:21 |
| openstackgerrit | A change was merged to openstack-dev/devstack: FAQ: Removed LXC not being supported on Fedora https://review.openstack.org/106927 | 04:38 |
| openstackgerrit | A change was merged to openstack/tempest: Fix response body format of orchestration_client to dict https://review.openstack.org/117713 | 04:40 |
| *** yfried has quit IRC | 04:46 | |
| *** arnaud has joined #openstack-qa | 04:48 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-dev/devstack: Use mod_version to clean-up apache version matching https://review.openstack.org/104125 | 05:14 |
| *** chandankumar has joined #openstack-qa | 05:15 | |
| openstackgerrit | varun kumar yadav proposed a change to openstack/tempest: Test to update port with CIDR value https://review.openstack.org/113861 | 05:20 |
| *** arnaud has quit IRC | 05:32 | |
| *** amotoki has joined #openstack-qa | 05:34 | |
| *** flaper87|afk is now known as flaper87 | 05:39 | |
| *** nosnos has quit IRC | 05:57 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-dev/devstack: Use mod_version to clean-up apache version matching https://review.openstack.org/104125 | 05:59 |
| *** nosnos_ has joined #openstack-qa | 06:00 | |
| *** arnaud has joined #openstack-qa | 06:07 | |
| *** nmagnezi has joined #openstack-qa | 06:07 | |
| *** nmagnezi has quit IRC | 06:07 | |
| *** nmagnezi has joined #openstack-qa | 06:08 | |
| *** k4n0 has joined #openstack-qa | 06:18 | |
| *** yfried has joined #openstack-qa | 06:30 | |
| *** k4n0 has quit IRC | 06:33 | |
| *** neeti has joined #openstack-qa | 06:41 | |
| *** k4n0 has joined #openstack-qa | 06:47 | |
| yfried | andreaf_: ping | 06:47 |
| yfried | masayukig: ping? | 06:58 |
| *** mpaolino has joined #openstack-qa | 06:59 | |
| *** SridharG has joined #openstack-qa | 07:01 | |
| *** mpaolino has quit IRC | 07:04 | |
| *** k4n0 has quit IRC | 07:09 | |
| *** Madan has joined #openstack-qa | 07:10 | |
| *** evgenyf has joined #openstack-qa | 07:12 | |
| *** jpich has quit IRC | 07:14 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Simplify ironic-python-agent download https://review.openstack.org/117384 | 07:16 |
| *** Longgeek has joined #openstack-qa | 07:16 | |
| *** afazekas_ has joined #openstack-qa | 07:22 | |
| *** arnaud has quit IRC | 07:22 | |
| *** e0ne has joined #openstack-qa | 07:22 | |
| *** e0ne has quit IRC | 07:24 | |
| *** tkammer has joined #openstack-qa | 07:24 | |
| *** jordanP has joined #openstack-qa | 07:26 | |
| *** luqas has joined #openstack-qa | 07:26 | |
| *** Longgeek has quit IRC | 07:33 | |
| *** k4n0 has joined #openstack-qa | 07:35 | |
| *** jlanoux has joined #openstack-qa | 07:36 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-dev/devstack: Add support for redis to Zaqar's lib https://review.openstack.org/119379 | 07:42 |
| openstackgerrit | varun kumar yadav proposed a change to openstack/tempest: Test to update port with CIDR value https://review.openstack.org/113861 | 07:44 |
| *** tkammer has quit IRC | 07:47 | |
| *** jhenner has joined #openstack-qa | 07:48 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-dev/devstack: Specialize Zaqar's cleanup function https://review.openstack.org/119708 | 07:49 |
| openstackgerrit | Flavio Percoco proposed a change to openstack-dev/devstack: Specialize Zaqar's cleanup function https://review.openstack.org/119708 | 07:50 |
| *** mpaolino has joined #openstack-qa | 07:50 | |
| *** jlibosva has joined #openstack-qa | 07:56 | |
| *** shardy_afk is now known as shardy | 07:56 | |
| *** jlibosva has quit IRC | 07:58 | |
| *** jlibosva has joined #openstack-qa | 08:00 | |
| *** gszasz has joined #openstack-qa | 08:00 | |
| *** marun has joined #openstack-qa | 08:02 | |
| *** safchain has joined #openstack-qa | 08:04 | |
| *** Longgeek has joined #openstack-qa | 08:07 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:18 | |
| *** ZZelle has quit IRC | 08:19 | |
| *** ZZelle has joined #openstack-qa | 08:20 | |
| *** e0ne has joined #openstack-qa | 08:20 | |
| *** jlibosva has quit IRC | 08:21 | |
| *** jlibosva has joined #openstack-qa | 08:22 | |
| *** Dafna has joined #openstack-qa | 08:24 | |
| *** tkammer has joined #openstack-qa | 08:27 | |
| *** andreaf_ is now known as andreaf | 08:29 | |
| yfried | andreaf: are you online? | 08:33 |
| *** marun has quit IRC | 08:34 | |
| andreaf | yfried: almost - going for a coffee brain didn't boot up properly yet ;) | 08:35 |
| andreaf | yfried: 10min | 08:36 |
| yfried | andreaf: tnx. could you pls ping me when you and your brain are working in sync? | 08:36 |
| *** aix has joined #openstack-qa | 08:39 | |
| andreaf | yfried: hi | 08:44 |
| yfried | andreaf: hi | 08:44 |
| andreaf | what's up? | 08:44 |
| yfried | andreaf: any chance you could help me push my 2 client patches so we can finish that bp? | 08:45 |
| yfried | andreaf: https://review.openstack.org/#/c/112199/ https://review.openstack.org/113124 | 08:45 |
| yfried | andreaf: citrix bot is giving it -1 for unkown reasons | 08:46 |
| andreaf | jenkin's failing as well on https://review.openstack.org/#/c/113124/ | 08:46 |
| andreaf | yfried: they're in my todo list already I'll try to review them today | 08:46 |
| yfried | andreaf: yeah. I've IDed that failure. will push a fix once my setup is ready to retest | 08:46 |
| andreaf | yfried ok | 08:47 |
| yfried | andreaf: I was hoping to catch you online so I can fix the issues you'd find immediately. any chance for that? | 08:47 |
| andreaf | yfried: sorry I cannot look at them right now I'll ping you when I do - what TZ are you in? | 08:49 |
| yfried | UTC+3 | 08:49 |
| yfried | andreaf: I'm online for the coming 4hrs. let me know if you happen to review any of the patches | 08:50 |
| andreaf | yfried: ok | 08:50 |
| yfried | andreaf: tnx | 08:51 |
| *** luqas has quit IRC | 08:51 | |
| *** chandankumar has quit IRC | 08:52 | |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Migrate advanced server network to tempest clients https://review.openstack.org/113124 | 08:52 |
| *** ildikov has joined #openstack-qa | 08:52 | |
| yfried | andreaf: do you happen to know who'se in charge of citrix bot? | 08:52 |
| andreaf | yfried: I think there is a wiki page with that info somewhere | 08:55 |
| yfried | https://wiki.openstack.org/wiki/Debugging_XenServer_CI_failures | 08:55 |
| yfried | andreaf: that's a single a guy and he's not online | 08:55 |
| yfried | andreaf: do you happen to have 3min for an unrelated conversation? | 08:57 |
| *** Poornima has quit IRC | 08:59 | |
| *** Poornima_ has joined #openstack-qa | 08:59 | |
| *** safchain has quit IRC | 09:00 | |
| *** sergeysh has joined #openstack-qa | 09:00 | |
| openstackgerrit | Martin André proposed a change to openstack-dev/devstack: Source lib/dstat in unstack.sh https://review.openstack.org/119715 | 09:05 |
| *** qba73 has joined #openstack-qa | 09:08 | |
| *** chandankumar has joined #openstack-qa | 09:08 | |
| andreaf | yfried: yes | 09:15 |
| yfried | andreaf: PMing you | 09:15 |
| andreaf | yfried: ? | 09:16 |
| yfried | andreaf: Private Message :) | 09:17 |
| *** gfidente has joined #openstack-qa | 09:17 | |
| *** mkollaro has joined #openstack-qa | 09:22 | |
| *** sergeysh has quit IRC | 09:24 | |
| *** sergeysh has joined #openstack-qa | 09:25 | |
| *** luqas has joined #openstack-qa | 09:26 | |
| *** mkollaro has quit IRC | 09:36 | |
| *** mkollaro has joined #openstack-qa | 09:44 | |
| *** Poornima_ has quit IRC | 09:45 | |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Migrate security_groups_basic to tempest clients https://review.openstack.org/112199 | 09:53 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Migrate advanced server network to tempest clients https://review.openstack.org/113124 | 09:54 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Adds status check for FloatingIP in scenarios https://review.openstack.org/102700 | 09:54 |
| *** Poornima_ has joined #openstack-qa | 09:59 | |
| *** e0ne has quit IRC | 10:00 | |
| *** safchain has joined #openstack-qa | 10:02 | |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack-dev/devstack: (DO NOT MERGE) test run using mysqlconnector https://review.openstack.org/114207 | 10:03 |
| *** luqas has quit IRC | 10:04 | |
| *** neeti has quit IRC | 10:08 | |
| *** yamahata has quit IRC | 10:09 | |
| *** e0ne has joined #openstack-qa | 10:12 | |
| *** safchain has quit IRC | 10:12 | |
| openstackgerrit | Masayuki Igawa proposed a change to openstack/tempest: Fix unstable assertion in test_cinder_endpoints https://review.openstack.org/117742 | 10:15 |
| *** chandankumar has quit IRC | 10:17 | |
| *** tkammer has quit IRC | 10:18 | |
| *** chandankumar has joined #openstack-qa | 10:22 | |
| jordanP | if a tempest core dev have 5 min, I'd like to review of https://review.openstack.org/#/c/116622/ (3 lines diff). Thanks | 10:29 |
| jordanP | s/to/a/ | 10:29 |
| *** tkammer has joined #openstack-qa | 10:30 | |
| andreaf | jordanP: so this happens when tenant isolation is turned off? | 10:31 |
| jordanP | andreaf, hummm, dunno, it happened with a fresh devstack with default settings :) | 10:32 |
| *** gfidente has quit IRC | 10:33 | |
| andreaf | jordanP: uhm odd I would expect the orchestration test to run on a newly created tenant, which should not see objects created by demo user - however when running without tenant isolation demo user is used for all tests | 10:35 |
| *** gfidente has joined #openstack-qa | 10:35 | |
| jordanP | andreaf, so another valid fix for this would be to force tenant_isolationn ? | 10:40 |
| *** mpaolino has quit IRC | 10:54 | |
| andreaf | jordanP, well, I guess so, but I like your patch better, I think force_tenant_isolation should only be used where really needed | 10:54 |
| *** safchain has joined #openstack-qa | 10:57 | |
| andreaf | jordanP, I just wanted to understand how the problem reproduced | 10:57 |
| *** dims has joined #openstack-qa | 11:00 | |
| openstackgerrit | Ashish Kumar Gupta proposed a change to openstack/tempest: Fwaas API Test Enhancement https://review.openstack.org/102820 | 11:04 |
| *** luqas has joined #openstack-qa | 11:08 | |
| *** dims has quit IRC | 11:16 | |
| *** dims has joined #openstack-qa | 11:16 | |
| *** dims_ has joined #openstack-qa | 11:17 | |
| *** Poornima_ has quit IRC | 11:19 | |
| *** dims has quit IRC | 11:21 | |
| *** e0ne has quit IRC | 11:22 | |
| *** mwagner_lap has quit IRC | 11:23 | |
| *** rfolco has joined #openstack-qa | 11:24 | |
| *** luqas has quit IRC | 11:34 | |
| *** mmedvede has joined #openstack-qa | 11:45 | |
| openstackgerrit | A change was merged to openstack/tempest: Remove force_tenant_isolation=True from test that doesn't need it https://review.openstack.org/106160 | 11:56 |
| *** mwagner_lap has joined #openstack-qa | 12:02 | |
| *** Madan has quit IRC | 12:05 | |
| *** yfried has quit IRC | 12:06 | |
| *** dmorita has quit IRC | 12:06 | |
| *** nosnos_ has quit IRC | 12:13 | |
| *** nosnos has joined #openstack-qa | 12:14 | |
| *** dims_ has quit IRC | 12:16 | |
| *** dims has joined #openstack-qa | 12:16 | |
| *** weshay has joined #openstack-qa | 12:17 | |
| *** nosnos has quit IRC | 12:18 | |
| *** aysyd has joined #openstack-qa | 12:19 | |
| *** dkranz has joined #openstack-qa | 12:20 | |
| *** mkoderer_ is now known as mkoderer | 12:20 | |
| *** dims_ has joined #openstack-qa | 12:20 | |
| *** dims has quit IRC | 12:21 | |
| *** adalbas has joined #openstack-qa | 12:24 | |
| *** luqas has joined #openstack-qa | 12:27 | |
| *** safchain has quit IRC | 12:29 | |
| *** ildikov_ has joined #openstack-qa | 12:31 | |
| *** ildikov has quit IRC | 12:34 | |
| *** Albert has joined #openstack-qa | 12:43 | |
| Albert | Hello | 12:43 |
| Albert | If I want to submit a small improvement into the tempest core code, how I do have to proceed, in my opinion it is too small for a blueprint... | 12:44 |
| *** yamahata has joined #openstack-qa | 12:45 | |
| *** yfried has joined #openstack-qa | 12:45 | |
| jordanP | Albert, if there's a bug or could be a bug, submit a bug report | 12:46 |
| jordanP | ortherwise, submit a patch and you'll see | 12:46 |
| Albert | Thanks JordanP | 12:47 |
| Albert | I'll take a look into the patch | 12:48 |
| *** bkopilov has quit IRC | 12:48 | |
| *** gszasz has quit IRC | 13:01 | |
| *** tkammer has quit IRC | 13:03 | |
| *** Longgeek has quit IRC | 13:08 | |
| *** mpaolino has joined #openstack-qa | 13:08 | |
| *** Poornima has joined #openstack-qa | 13:08 | |
| *** e0ne has joined #openstack-qa | 13:09 | |
| *** dustins has joined #openstack-qa | 13:10 | |
| *** Longgeek has joined #openstack-qa | 13:11 | |
| *** gszasz has joined #openstack-qa | 13:12 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Cleanup documentation for cli execute code https://review.openstack.org/119518 | 13:14 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Add docs for docstrings in cli to sphinx output https://review.openstack.org/119519 | 13:15 |
| *** tkammer has joined #openstack-qa | 13:16 | |
| *** sabeen has joined #openstack-qa | 13:20 | |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Migrate advanced server network to tempest clients https://review.openstack.org/113124 | 13:23 |
| openstackgerrit | John Davidge proposed a change to openstack-dev/devstack: Add IPv6 support for tenant data network https://review.openstack.org/87987 | 13:26 |
| *** mriedem has joined #openstack-qa | 13:27 | |
| openstackgerrit | YAMAMOTO Takashi proposed a change to openstack-dev/devstack: ofagent: Switch to IptablesFirewallDriver https://review.openstack.org/108915 | 13:27 |
| *** safchain has joined #openstack-qa | 13:29 | |
| *** mriedem1 has joined #openstack-qa | 13:35 | |
| *** mriedem has quit IRC | 13:36 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Cleanup documentation for cli execute code https://review.openstack.org/119518 | 13:42 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Add docs for docstrings in cli to sphinx output https://review.openstack.org/119519 | 13:42 |
| openstackgerrit | Albert vico proposed a change to openstack/tempest: Improvements into ssh and remote_client classes Now it is possible to create tunneled ssh connections through a list of gateways and provide particular timeout to exec_command https://review.openstack.org/119762 | 13:47 |
| *** masayukig has quit IRC | 13:49 | |
| *** jecarey has joined #openstack-qa | 13:55 | |
| *** jaypipes has joined #openstack-qa | 13:55 | |
| *** ChanServ sets mode: +o jaypipes | 13:55 | |
| *** masayukig has joined #openstack-qa | 13:55 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Cleanup documentation for cli execute code https://review.openstack.org/119518 | 13:57 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest-lib: Add docs for docstrings in cli to sphinx output https://review.openstack.org/119519 | 13:57 |
| *** salv-orlando has joined #openstack-qa | 13:58 | |
| *** markmcclain has joined #openstack-qa | 14:02 | |
| *** markmcclain has quit IRC | 14:03 | |
| *** beekneemech is now known as bnemec | 14:04 | |
| *** markmcclain has joined #openstack-qa | 14:05 | |
| *** nmagnezi has quit IRC | 14:10 | |
| *** zz_jgrimm is now known as jgrimm | 14:12 | |
| *** ilyashakhat has joined #openstack-qa | 14:12 | |
| *** toure_ has joined #openstack-qa | 14:13 | |
| *** dkranz has quit IRC | 14:18 | |
| mtreinish | DinaBelova: on https://review.openstack.org/#/c/115212/ it looks like ceilometer is eating all of the nodes ram | 14:19 |
| *** zz_dimtruck is now known as dimtruck | 14:21 | |
| *** larsks has left #openstack-qa | 14:24 | |
| openstackgerrit | Albert vico proposed a change to openstack/tempest: Improvements into ssh and remote_client classes https://review.openstack.org/119762 | 14:24 |
| *** atiwari has joined #openstack-qa | 14:25 | |
| *** mdenny has quit IRC | 14:25 | |
| *** mdenny has joined #openstack-qa | 14:25 | |
| *** dkranz has joined #openstack-qa | 14:30 | |
| *** yamahata has quit IRC | 14:33 | |
| *** superdan is now known as dansmith | 14:34 | |
| *** cdent has joined #openstack-qa | 14:34 | |
| openstackgerrit | Mikhail S Medvedev proposed a change to openstack/tempest: Skip orchestration tests if Neutron is not available https://review.openstack.org/118454 | 14:38 |
| openstackgerrit | Chmouel Boudjnah proposed a change to openstack-dev/devstack: Fix another_role variable for swift https://review.openstack.org/118105 | 14:41 |
| openstackgerrit | Swapnil Kulkarni proposed a change to openstack/tempest: Updated compute keypair clients for bp client-checks-success https://review.openstack.org/116435 | 15:00 |
| *** SridharG has left #openstack-qa | 15:04 | |
| *** yamahata has joined #openstack-qa | 15:05 | |
| *** enikanorov_ has joined #openstack-qa | 15:09 | |
| *** enikanorov_ has quit IRC | 15:09 | |
| *** enikanorov_ has joined #openstack-qa | 15:10 | |
| *** chandankumar has quit IRC | 15:10 | |
| *** aix has quit IRC | 15:13 | |
| *** salv-orlando has quit IRC | 15:14 | |
| *** aix has joined #openstack-qa | 15:15 | |
| *** afazekas_ has quit IRC | 15:17 | |
| *** gszasz has quit IRC | 15:19 | |
| *** toure_ has quit IRC | 15:20 | |
| *** hemna__ is now known as hemna | 15:20 | |
| openstackgerrit | Michele Paolino proposed a change to openstack-dev/devstack: Support for libvirt/QEMU tar releases https://review.openstack.org/108714 | 15:21 |
| *** gszasz has joined #openstack-qa | 15:21 | |
| *** toure_ has joined #openstack-qa | 15:22 | |
| *** mlavalle has joined #openstack-qa | 15:23 | |
| *** markmcclain has quit IRC | 15:26 | |
| openstackgerrit | John Davidge proposed a change to openstack-dev/devstack: Add IPv6 support for tenant data network https://review.openstack.org/87987 | 15:26 |
| *** safchain has quit IRC | 15:26 | |
| *** tkammer has quit IRC | 15:27 | |
| *** rbak has quit IRC | 15:32 | |
| *** k4n0_ has joined #openstack-qa | 15:35 | |
| *** k4n0 has quit IRC | 15:37 | |
| *** tkammer has joined #openstack-qa | 15:39 | |
| *** ildikov_ has quit IRC | 15:41 | |
| *** toure_ has quit IRC | 15:42 | |
| *** amotoki_ has joined #openstack-qa | 15:42 | |
| *** yamahata has quit IRC | 15:43 | |
| *** toure_ has joined #openstack-qa | 15:43 | |
| *** mwagner_lap has quit IRC | 15:46 | |
| *** safchain has joined #openstack-qa | 15:47 | |
| *** salv-orlando has joined #openstack-qa | 15:48 | |
| *** tkammer has quit IRC | 15:50 | |
| *** evgenyf has quit IRC | 15:50 | |
| *** gfidente has quit IRC | 15:58 | |
| *** markmcclain has joined #openstack-qa | 15:59 | |
| *** toure_ has quit IRC | 16:01 | |
| *** toure_ has joined #openstack-qa | 16:02 | |
| *** jlanoux has quit IRC | 16:03 | |
| *** jlibosva has quit IRC | 16:04 | |
| *** arnaud has joined #openstack-qa | 16:04 | |
| *** pcrews has joined #openstack-qa | 16:05 | |
| *** e0ne has quit IRC | 16:06 | |
| *** andreaf is now known as andreaf_ | 16:09 | |
| mlavalle | andreaf_: you around? | 16:12 |
| *** chandankumar has joined #openstack-qa | 16:12 | |
| *** toure_ has quit IRC | 16:17 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 16:22 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Change httpd Keystone access log to keystone_access.log https://review.openstack.org/116133 | 16:23 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Keystone access log in key-access screen https://review.openstack.org/116134 | 16:23 |
| *** chandankumar has quit IRC | 16:30 | |
| *** Albert has quit IRC | 16:34 | |
| *** yfried has quit IRC | 16:37 | |
| *** suneelb has joined #openstack-qa | 16:39 | |
| *** jordanP has quit IRC | 16:42 | |
| *** chandankumar has joined #openstack-qa | 16:51 | |
| *** e0ne has joined #openstack-qa | 16:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/tempest: Updated from global requirements https://review.openstack.org/110110 | 16:54 |
| *** tkammer has joined #openstack-qa | 16:55 | |
| *** atiwari has quit IRC | 17:00 | |
| *** cbader has joined #openstack-qa | 17:02 | |
| *** Poornima has quit IRC | 17:04 | |
| *** chandankumar has quit IRC | 17:06 | |
| *** qba73 has quit IRC | 17:08 | |
| *** yjiang5 has joined #openstack-qa | 17:12 | |
| *** harlowja_away is now known as harlowja | 17:14 | |
| *** chandankumar has joined #openstack-qa | 17:16 | |
| *** atiwari has joined #openstack-qa | 17:19 | |
| *** gszasz has quit IRC | 17:21 | |
| *** markmcclain has quit IRC | 17:24 | |
| *** yfried has joined #openstack-qa | 17:32 | |
| *** markmcclain has joined #openstack-qa | 17:35 | |
| *** toure_ has joined #openstack-qa | 17:36 | |
| *** mwagner_lap has joined #openstack-qa | 17:36 | |
| *** aysyd has quit IRC | 17:36 | |
| *** aysyd has joined #openstack-qa | 17:39 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-dev/devstack: Switch config to use Ironic driver from Nova's tree https://review.openstack.org/119844 | 17:43 |
| *** yjiang5 has quit IRC | 17:50 | |
| *** safchain has quit IRC | 17:53 | |
| *** mpaolino has quit IRC | 17:53 | |
| *** cdent_ has joined #openstack-qa | 17:54 | |
| *** cdent has quit IRC | 17:56 | |
| *** cdent_ is now known as cdent | 17:56 | |
| *** rbak has joined #openstack-qa | 17:57 | |
| adam_g | sdague, dtroyer_zz jogo i had to make a couple last changes friday to address issues that came up running it on the jenkins slaves via experimental job, but should be ready for final review https://review.openstack.org/#/c/111859/ | 18:01 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-dev/devstack: Install tempest-lib as part of install_tempest https://review.openstack.org/119850 | 18:03 |
| smoser | salv-orlando, mtu should be honored in just released 0.3.3 | 18:08 |
| openstackgerrit | Sridhar Gaddam proposed a change to openstack/tempest: Improve IPV6 parity in Security Group testcases https://review.openstack.org/94130 | 18:09 |
| sdague | mtreinish: is there a d-g change for tempest-lib? | 18:19 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Migrate cli test framework to tempest-lib https://review.openstack.org/117649 | 18:20 |
| *** k4n0_ has quit IRC | 18:20 | |
| mtreinish | sdague: not yet | 18:20 |
| sdague | the devstack job can't pass until it's in nodepool | 18:21 |
| sdague | which means d-g part has landed | 18:21 |
| sdague | we block cloning in d-g | 18:21 |
| *** aix has quit IRC | 18:21 | |
| mtreinish | sdague: yeah, I know I'm still pushing things out. (mostly as I remember to do them which is fairly random) | 18:21 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Add tempest-lib to the projects list https://review.openstack.org/119862 | 18:23 |
| mtreinish | sdague: ^^^ | 18:23 |
| mtreinish | sdague: I also still need to push the first version to pypi and add it to g-r otherwise I won't be able to list it in the tempest reqs when the time comes | 18:24 |
| sdague | sure, but that won't matter to get the testing up on the first go around | 18:25 |
| mtreinish | no, it's just the other things I've thought of so far :) | 18:25 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-dev/devstack: Switch config to use Ironic driver from Nova's tree https://review.openstack.org/119844 | 18:27 |
| *** k4n0_ has joined #openstack-qa | 18:37 | |
| *** tkammer has quit IRC | 18:38 | |
| *** chandankumar has quit IRC | 18:42 | |
| openstackgerrit | vigneshvar proposed a change to openstack-dev/devstack: Reset no_proxy when the node is installed behind a proxy https://review.openstack.org/117985 | 18:43 |
| *** openstackgerrit has quit IRC | 18:51 | |
| *** mkollaro has quit IRC | 18:51 | |
| *** wendar_ is now known as wendar | 18:58 | |
| *** mkollaro has joined #openstack-qa | 19:03 | |
| jogo | mtreinish: so what is running java | 19:03 |
| jogo | ? | 19:03 |
| mtreinish | dunno? I'm assuming it's a dep pulled in by something | 19:03 |
| mtreinish | like sahara | 19:03 |
| mtreinish | for hadoop | 19:04 |
| jogo | sdague: we are running out of memory in dsvm jobs | 19:04 |
| jogo | mtreinish: http://logs.openstack.org/27/100927/18/check/check-tempest-dsvm-neutron-full/946ec0f/logs/ps.txt.gz | 19:04 |
| jogo | zookeeper? | 19:04 |
| mtreinish | that's hadoop stuff | 19:05 |
| jogo | wtf | 19:05 |
| mtreinish | jogo: https://wiki.apache.org/hadoop/ZooKeeper | 19:05 |
| jogo | mtreinish: gah | 19:05 |
| jogo | sdague: ^ | 19:05 |
| jogo | so that is really an issue | 19:06 |
| jogo | w.r.t. to the TCs idea of minimizing the number of services OpenStack depends on | 19:06 |
| *** rockyg has joined #openstack-qa | 19:06 | |
| *** openstackgerrit has joined #openstack-qa | 19:07 | |
| jogo | I need to find some lunch, but want to send out a email on this? | 19:07 |
| mtreinish | yeah, it's a big chunk of ram, but I'm wondering why this one job pushed us over the edge | 19:07 |
| jogo | mtreinish: very possible | 19:08 |
| mtreinish | because we've been running with sahara for a while | 19:08 |
| jogo | it may have not been the straw that broke the camels back but its the 500 lb obese person on the camel | 19:08 |
| jogo | clarkb: ^ | 19:09 |
| mtreinish | jogo: that's a fair point, although a bigger consumer of ram was nova-api | 19:11 |
| jogo | mtreinish: wait really? | 19:11 |
| clarkb | jogo: ya see the rss column | 19:11 |
| jogo | wow | 19:12 |
| mtreinish | jogo: http://paste.openstack.org/show/108431/ | 19:12 |
| mtreinish | I just trimmed the log to the rss column and ran sort on it | 19:12 |
| jogo | why 25 nova-api's | 19:12 |
| mtreinish | the bigger consumers are at the bottom | 19:12 |
| mtreinish | yeah, that's a good question why are there so many | 19:13 |
| *** mestery has quit IRC | 19:13 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1366931 https://review.openstack.org/119879 | 19:13 |
| jogo | mtreinish: all good questions, so lunch time for me | 19:14 |
| jogo | dansmith: ^ | 19:14 |
| jogo | nova-api is a memory hungry monster | 19:14 |
| *** mestery has joined #openstack-qa | 19:14 | |
| jogo | and we run a ton of of them | 19:14 |
| * jogo wonders off to lunch | 19:14 | |
| jogo | mtreinish: I'll dig into this after lunch | 19:14 |
| jogo | mriedem1: ^ | 19:15 |
| mtreinish | jogo: ok, I'll keep looking too | 19:15 |
| *** mestery has quit IRC | 19:15 | |
| *** nmagnezi has joined #openstack-qa | 19:15 | |
| *** Sukhdev has joined #openstack-qa | 19:15 | |
| *** mestery has joined #openstack-qa | 19:15 | |
| mtreinish | jogo, sdague, mriedem1: http://logs.openstack.org/12/115212/5/check/check-tempest-dsvm-full/11a1af6/logs/screen-n-api.txt.gz it looks like we're starting nova-api 3 times | 19:18 |
| mtreinish | which would explain why we're seeing 24 workers... | 19:18 |
| mriedem1 | 8 cpus? | 19:18 |
| mriedem1 | so 3 * 8 | 19:18 |
| mriedem1 | that's not including the metadata workers, since that's not a pg job | 19:19 |
| mtreinish | mriedem1: yeah, the node in that job triggered oom-killer and jogo noticed that there were too many nova-api processes | 19:20 |
| *** yfried has quit IRC | 19:28 | |
| mtreinish | mriedem1, jogo: hmm, it looks like a passing job does that too... http://logs.openstack.org/44/118744/3/check/check-tempest-dsvm-full/1678a1a/logs/screen-n-api.txt.gz | 19:28 |
| mriedem1 | mtreinish: 1 parent, 1 child osapi compute worker and 1 ec2 api worker maybe? | 19:33 |
| mriedem1 | *1 parent 2 children | 19:33 |
| mriedem1 | + 8 conductor workers | 19:33 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1366931 https://review.openstack.org/119879 | 19:35 |
| mtreinish | mriedem1: yeah, it's 1 ec2, 1 osapi_compute, and 1 metadata | 19:36 |
| mtreinish | because in config enabled_apis=[ec2, osapi_compute, metadata] | 19:37 |
| mtreinish | mriedem1: logs.openstack.org/12/115212/5/check/check-tempest-dsvm-full/11a1af6/logs/screen-n-api.txt.gz#_2014-09-08_11_50_07_206 | 19:37 |
| mtreinish | so that explains the 24 workers | 19:38 |
| openstackgerrit | Travis Tripp proposed a change to openstack-dev/devstack: Initialize metadata definitions catalog https://review.openstack.org/110955 | 19:39 |
| mtreinish | mriedem1: is that the expected behavior if you set api workers to N, it's N * # of enabled apis? | 19:39 |
| *** salv-orlando has quit IRC | 19:40 | |
| *** ajo_ has joined #openstack-qa | 19:42 | |
| adam_g | FWIW the OOM fails look isolated to RAX, at least according to the query in https://launchpad.net/bugs/1366931 | 19:42 |
| mtreinish | adam_g: it might be because the rax nodes have extra procs running on them by default | 19:43 |
| adam_g | mtreinish, yeah, a oom-killer message referencing nova-agent made me wonder | 19:44 |
| mriedem1 | mtreinish: i believe so | 19:44 |
| mriedem1 | mtreinish: but i doubt anyone thought about how it's really enabling that for 3 things rather than just 1 | 19:44 |
| mtreinish | mriedem1: well it looks like there are 3 different worker # options for each of them, so it might be we just set it to 8 for each in devstack | 19:47 |
| mtreinish | mriedem1: oh it's http://git.openstack.org/cgit/openstack/nova/tree/nova/service.py#n336 I'll push out a patch to make it so it divides that over the number of enabled apis | 19:51 |
| mriedem1 | so 8 / 3 in this case | 19:53 |
| mriedem1 | mtreinish: that seems sane | 19:54 |
| *** toure_ has quit IRC | 19:55 | |
| *** ajo_ has quit IRC | 19:59 | |
| *** k4n0_ has quit IRC | 20:05 | |
| *** ajo_ has joined #openstack-qa | 20:06 | |
| *** markmcclain has quit IRC | 20:08 | |
| *** nmagnezi has quit IRC | 20:08 | |
| *** e0ne has quit IRC | 20:09 | |
| jogo | mriedem1 mtreinish: sounds like we should change nova's default behaviour about how many processes to spin up | 20:15 |
| jogo | so its num_cpu total workers | 20:15 |
| jogo | not per API endpoint | 20:15 |
| mriedem1 | jogo: well you could do a floor value of nproc / enabled_apis | 20:16 |
| mriedem1 | with 8 cpus that'd be 2 workers each | 20:16 |
| mriedem1 | but then you'd have 8 conductor workers also | 20:17 |
| *** Shrews has quit IRC | 20:17 | |
| *** Shrews has joined #openstack-qa | 20:17 | |
| jogo | mriedem1: thats OK | 20:18 |
| jogo | mriedem1: if the goal is to try to potentially use up all the CPUs, nproc/ enabled_apis makes more sense | 20:18 |
| jogo | as for why nova-api is a memory moster | 20:19 |
| mriedem1 | jogo: sounds like we need osprofiler! :) | 20:20 |
| jogo | mriedem1: actually a regular python profiler here | 20:24 |
| cdent | jogo I was just moaning to eglynn earlier today about the cost of so many service spinning up num_cpu workers | 20:24 |
| cdent | it's bound to create latency | 20:24 |
| jogo | cdent: yeah its a bit much | 20:24 |
| cdent | sdague: jogo suggested a while ago that I speak to you for some direction on how to add stack/status/stack to the collection of stuff that gets saved during a grenade run in the gate | 20:26 |
| *** jhenner has quit IRC | 20:29 | |
| sdague | mtreinish: on the worker front, there is a patching hanging out somewhere to let use override API_WORKERS in devstack, mostly for setting it to 1 | 20:29 |
| jogo | sdague: I think the defailt doesn't make a ton of sense though | 20:30 |
| jogo | although one could argue its not terrible either | 20:30 |
| sdague | jogo: so that's fine, but I'd say take that out to the list to figure out sane defaults | 20:30 |
| sdague | because all the services started heading in the nova direction | 20:30 |
| jogo | sdague: so maybe devstack setting overridding the value makes ssense | 20:30 |
| sdague | jogo: well, I don't want to set an override by default | 20:31 |
| jogo | sdague: I mean in devstack-gate | 20:31 |
| sdague | but I do want to make it an option, mostly for the low memor case | 20:31 |
| sdague | jogo: yeh, I don't think we set it there unless we absolutely have to | 20:31 |
| sdague | the defaults here should be sane | 20:31 |
| sdague | cdent: can you expand on what you are trying to do? (sorry, I don't have all the context) | 20:31 |
| jogo | sdague: so there are two other issues around oom | 20:31 |
| jogo | sdague: we run zookeeper | 20:32 |
| jogo | sdague: and nova-api hogs memory | 20:32 |
| sdague | well nova-api hogging memory sounds like a bug that should be fixed | 20:32 |
| *** sbadia has quit IRC | 20:32 | |
| sdague | zookeeper is only in the unit test nodes, right? | 20:32 |
| cdent | sdague: ping me when you done talking to jogo and I'll go into more detail? | 20:33 |
| sdague | cdent: I can multiplex | 20:33 |
| jogo | sdague: yeah talking to comstud and dansmith on nova-api | 20:33 |
| cdent | k: I'm trying to debug this bug https://bugs.launchpad.net/ceilometer/+bug/1366445 in which ceilometer notification agent is not starting in the icehouse side of a grenade run, thus causing javelin2 -m create to fail while checking telemetry | 20:34 |
| cdent | the devstack log shows that it is trying to start the agent but there are no artifacts (such as the screen log) | 20:34 |
| cdent | so I was hoping to be able to look in stack/status/stack for either a *failure or *pid file to see if that was there, just for a bit more info | 20:34 |
| cdent | but as far as I can tell that's not saved so we thought perhaps that would be a useful piece to have | 20:35 |
| sdague | cdent: ok... so this is the bigger issue around using screen upstream | 20:35 |
| cdent | it probably wouldn't help solve the problem, but would provide some data | 20:35 |
| *** rockyg has quit IRC | 20:35 | |
| cdent | do tell? | 20:35 |
| sdague | under some conditions, that I'm not sure we understand, screen drops commands | 20:35 |
| sdague | so my guess is that it's dropping the notification daemon -X stuff command | 20:36 |
| sdague | we think it's load related | 20:36 |
| sdague | but it's been really hard to figure out | 20:36 |
| cdent | if you can push me in a reasonable direction I'll be a willing differ | 20:37 |
| sdague | there is an option to run devstack without screen for this reason.... however... recently it was discovered that we aren't actually capturing the run pids correctly so the stop_* commands don't work if we aren't running in screen | 20:37 |
| sdague | which is the blocker in shifting grenade to USE_SCREEN=False | 20:37 |
| *** sbadia has joined #openstack-qa | 20:37 | |
| sdague | dtroyer_zz: has some patches to redo service launch so we can capture the pid correctly, and make stop work | 20:37 |
| sdague | I'm not sure the state of them | 20:38 |
| sdague | but it's a deeper devstack refactor to do this correctly IIRC | 20:38 |
| sdague | we had largely been limping along with this deficiency but hpcloud 1.1 seems to expose it a lot more | 20:38 |
| sdague | honestly, I've only been about 1/2 following where that's at, but if you poke dtroyer I bet he could use help on it | 20:39 |
| cdent | any ideas on how/why hpcould would be impacting? | 20:39 |
| sdague | cdent: they are pretty aggressively using cgroups to constrain the cpu | 20:40 |
| sdague | that's about as good a guess as I've got | 20:40 |
| cdent | ah | 20:40 |
| jogo | wow keystone is ahog too | 20:40 |
| cdent | is there anything that's not a hog? | 20:40 |
| sdague | jogo: they are all hogs :) | 20:40 |
| cdent | jinx | 20:40 |
| jogo | cdent: swift :) | 20:40 |
| jogo | no wait | 20:40 |
| cdent | swift breaks all the rules | 20:40 |
| notmyname | hey! ;-) | 20:41 |
| cdent | :) | 20:41 |
| cdent | what timezone is dtroyer usually? | 20:42 |
| *** lakshmiS has joined #openstack-qa | 20:42 | |
| dtroyer_zz | sdague: the process-without-screen is in https://review.openstack.org/117339 and appears to be working | 20:42 |
| dtroyer_zz | cdent: US Central | 20:43 |
| cdent | apparently not zz? | 20:43 |
| dtroyer_zz | yeah, I don't know what's up with that, I think znc needs a kick | 20:43 |
| sdague | dtroyer_zz: so in that patch you only converted the cinder calls, right? | 20:44 |
| sdague | did you want to do it for all services, or wait for a second path? | 20:44 |
| sdague | patch? | 20:44 |
| sdague | I think that's why I didn't +2 it yet, that question | 20:44 |
| cdent | If there's someway I can help you with that stuff (testing or otherwise) please let me know. I seem to have become designated poke at devstack, grenade, tempest guy for at least some ceilo stuff | 20:44 |
| dtroyer_zz | that one, yes, I figured we'd do the rest once it was working | 20:44 |
| lakshmiS | Is this the right channel for tempest tests questions? cant replicate issues for https://review.openstack.org/#/c/113632/ locally even when run in parallel mode | 20:44 |
| dtroyer_zz | or we could do them all at once | 20:44 |
| sdague | dtroyer_zz: ok, I'm happy to move that into the gate and do a follow up patch | 20:45 |
| jogo | devstack ps sorted by %MEM http://paste.openstack.org/show/108458 | 20:45 |
| jogo | notmyname: swift is very light in memory usage http://paste.openstack.org/show/108458/ ! | 20:46 |
| cdent | too many workers | 20:46 |
| sdague | cdent: so... re helping here, if you want to bang out the conversion for the other services in devstack patch | 20:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/devstack: Add docs environment to tox.ini https://review.openstack.org/119907 | 20:47 |
| jogo | cdent: yeah | 20:47 |
| sdague | that should let us flip the switch, hopefully making it so we can do the non screen thing for grenade, and make your issue go away | 20:47 |
| mtreinish | jogo: heh, I think you can tell which 8 n-api workers are the metadata ones because we're running with config drive so they use less ram | 20:47 |
| cdent | yeah, I ought to be able to do that sdague, starting tomorrow (/me is GMT) | 20:47 |
| sdague | cdent: works for me | 20:47 |
| sdague | cdent: thanks! | 20:47 |
| mtreinish | although it's still using a lot for not doing anything | 20:47 |
| * cdent has a plan | 20:48 | |
| sdague | jogo: before or after tempest run? | 20:48 |
| *** HenryG has quit IRC | 20:48 | |
| jogo | sdague: after | 20:49 |
| jogo | collected from ps | 20:49 |
| jogo | from gate logs | 20:49 |
| sdague | can you collect a before? | 20:49 |
| jogo | technically a check queue patch | 20:49 |
| jogo | sdague: good idea, we don't have any data on that right now | 20:50 |
| *** markmcclain has joined #openstack-qa | 20:50 | |
| sdague | because it would be interesting to figure out how heavy things are just after start up | 20:50 |
| jogo | but I can put a patch up to do that | 20:50 |
| sdague | so... the world dump tool in devstack could be run at the end of it's run | 20:50 |
| sdague | it will give you all that in a timestamped file | 20:50 |
| *** cnesa10 has joined #openstack-qa | 20:50 | |
| sdague | then you'd have that before the run happens | 20:51 |
| sdague | it would be mostly quite interesting to figure out how much of this is completely service overhead, and how much of it is cached data that's bloating services after they've been used | 20:52 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/devstack: DO NOT MERGE: run worlddump at end of devstack https://review.openstack.org/119927 | 20:57 |
| jogo | sdague: ^ like that? | 20:57 |
| *** ajo_ has quit IRC | 20:58 | |
| sdague | jogo: yeh | 20:59 |
| cdent | is is possible to lower all the workers by fiat or does it require some consensus? | 21:00 |
| *** HenryG has joined #openstack-qa | 21:00 | |
| sdague | cdent: I think it should have concensus | 21:02 |
| sdague | also... if we just fiat it in upstream testing we're working around an issue which may or may not be a root issue that we should fix closer to the source | 21:03 |
| cdent | isn't it two different issues? | 21:03 |
| cdent | in a deployed situation it will often be the case that a service should use all the available cpus | 21:04 |
| *** mriedem1 is now known as mriedem | 21:04 | |
| sdague | cdent: sure, possibly | 21:04 |
| cdent | but in a test situation where there's just the one node, that's not the case | 21:04 |
| sdague | but it won't really hurt to have a weeks worth of comment on that fact for everyone to say "in production it's completely sane for nova-api to be using this much memory" | 21:04 |
| sdague | and say, ok, lets split the use cases | 21:05 |
| * cdent nods | 21:05 | |
| *** melwitt has joined #openstack-qa | 21:05 | |
| *** mmedvede has quit IRC | 21:05 | |
| sdague | jogo: so the java instance that's running is zookeeper? | 21:06 |
| mtreinish | sdague: that's what it looked like from ps | 21:06 |
| *** mwagner_lap has quit IRC | 21:08 | |
| sdague | hmmmm.... I'm surprised we have that on in every run | 21:08 |
| mtreinish | sdague: I just assumed it was something related to sahara | 21:09 |
| sdague | you have the logs for this run? | 21:09 |
| *** cbader has quit IRC | 21:10 | |
| mtreinish | sdague: which run? | 21:10 |
| clarkb | sdague: jogo should be one zk and the other jenkins slave client thingy | 21:10 |
| clarkb | you can probably tell which based on pid | 21:11 |
| mtreinish | clarkb: I thought the big memory one was zookeeper, but I can't remember | 21:11 |
| clarkb | it probably is. the jenkins agent is fairly lightweight considering | 21:11 |
| openstackgerrit | Lakshmi N Sampath proposed a change to openstack/tempest: Glance Metadata Definitions Catalog - Tempest tests https://review.openstack.org/113632 | 21:11 |
| sdague | clarkb: so my real question is what's installing zookeeper | 21:12 |
| sdague | because devstack is not | 21:12 |
| sdague | as far as I can tell | 21:12 |
| mtreinish | sdague: it looks like it runs on every job | 21:12 |
| sdague | mtreinish: right, so that seem bonkers | 21:12 |
| mtreinish | sdague: if it get's pulled in as a dep doesn't ubuntu default to enabling installed services | 21:12 |
| jogo | sdague: sahara is doing it | 21:13 |
| sdague | jogo: where? | 21:13 |
| sdague | sahara installs software on the host? | 21:13 |
| jogo | sdague: well the zookeeper services is only 0.3% | 21:13 |
| jogo | of MEM | 21:13 |
| jogo | sdague: not sure looking | 21:13 |
| clarkb | sdague: I can look, jogo indicated it was devstack | 21:13 |
| sdague | mtreinish: sure, but nothing I can see us pulling pulls it in | 21:14 |
| mtreinish | sdague: yeah neither can I searching for zookeeper in the devstacklog doesn't have any hits... | 21:14 |
| clarkb | it may be the base image too we install it on the unittest slaves and that may be leaking | 21:14 |
| mtreinish | but it's in the dpkg log | 21:14 |
| jogo | clarkb: that may be it | 21:14 |
| *** dkranz has quit IRC | 21:14 | |
| sdague | clarkb: yeh, that seems to be the most likely bit | 21:14 |
| clarkb | yup that appears to be it | 21:15 |
| *** Sukhdev has quit IRC | 21:15 | |
| sdague | and if so, we should probably figure out how to stop that, because there is a ton of extra stuff that gets installed on unit test nodes that I don't think we want to leak out | 21:15 |
| jogo | so I guess sahara isn't it, glad mtreinish was wrong on that | 21:15 |
| cdent | I don't think this is yet true, but at some point ceilometer is going to start (optinally) using zookeeper to do coordination for paritioning via tooz. | 21:17 |
| clarkb | sdague: I can work up a patchfor that | 21:17 |
| clarkb | sdague: should have it up shortly | 21:17 |
| sdague | cdent: sure, but in those cases we should on demand install for those configs | 21:17 |
| * cdent nods | 21:17 | |
| openstackgerrit | Lakshmi N Sampath proposed a change to openstack/tempest: Glance Metadata Definitions Catalog - Tempest tests https://review.openstack.org/113632 | 21:18 |
| *** weshay has quit IRC | 21:19 | |
| sdague | clarkb: awesome, thanks | 21:20 |
| clarkb | this is actually more fun to clean up than I expected :) but ya should be done shortlyish | 21:20 |
| sdague | clarkb: well I'm all about bringing more fun into your life :) | 21:23 |
| *** cnesa10 has quit IRC | 21:29 | |
| jogo | wow 8 {cinder-api,nova-conductor,nova-api,nova-api) take up a total of 41% of available memory | 21:33 |
| jogo | although that is rss which includes shared libraries | 21:33 |
| *** SpamapS_ is now known as SpamapS | 21:35 | |
| *** yamahata has joined #openstack-qa | 21:35 | |
| *** SpamapS has quit IRC | 21:35 | |
| *** SpamapS has joined #openstack-qa | 21:35 | |
| jogo | sdague: http://logs.openstack.org/83/119183/4/check/check-tempest-dsvm-full/ea576e7/logs/screen-dstat.txt.gz | 21:35 |
| jogo | that paints a nice picture | 21:35 |
| *** yamahata has quit IRC | 21:36 | |
| SergeyLukjanov | zookeper is installed to run tests on tooz | 21:36 |
| *** yamahata has joined #openstack-qa | 21:36 | |
| SergeyLukjanov | AFAIK it was added by jd | 21:36 |
| jogo | Free memory drops way down | 21:36 |
| jogo | SergeyLukjanov: yeah, it looks like that was leaked into devstack-gate jobs clarkb is fixing | 21:36 |
| sdague | jogo: yeh, so that would indicate that our software stack at idle is the issue | 21:37 |
| jogo | sdague: how do you conclude that? | 21:37 |
| sdague | because the mem free drop happens with nothing except the setup | 21:38 |
| sdague | nova-manage is the db sync call | 21:38 |
| *** flaper87 is now known as flaper87|afk | 21:38 | |
| jogo | sdague: ahh yeah | 21:38 |
| jogo | 19:52 is when tempest starts | 21:38 |
| sdague | we've consumed 5G of memory before the first guest launches | 21:39 |
| sdague | the guest launches always use the most process for the first few seconds of qemu | 21:39 |
| jogo | sdague: although you have to factor in cache | 21:39 |
| jogo | sdague: at 19:51 we have 3GB if data cached stuff | 21:40 |
| sdague | cached is actually decreasing this whole time | 21:40 |
| sdague | jogo: yeh, but we've evacuated a ton of it a couple minutes later | 21:40 |
| jogo | sdague: yeah aat 20:22:34 we have < 200M cached | 21:40 |
| jogo | and 96M free | 21:40 |
| jogo | so I don't think the issue is the code at rest | 21:41 |
| sdague | ah, right, fair | 21:42 |
| *** yjiang5 has joined #openstack-qa | 21:43 | |
| sdague | so, this mostly db caching? | 21:43 |
| jogo | sdague: not sure | 21:43 |
| sdague | it would explain why swift was imune | 21:44 |
| jogo | do they not use sqla? | 21:44 |
| *** mriedem1 has joined #openstack-qa | 21:44 | |
| notmyname | no | 21:44 |
| jogo | notmyname: ahh thanks | 21:45 |
| notmyname | jogo: there's no central db of anything in swift | 21:45 |
| *** zzzeek has joined #openstack-qa | 21:45 | |
| jogo | notmyname: right, you use sqllite DBs | 21:45 |
| zzzeek | hi! | 21:45 |
| *** mriedem has quit IRC | 21:45 | |
| jogo | sdague: so zzzeek is here so we can ask him | 21:45 |
| notmyname | jogo: for the sqlite DBs, we use actual SQL instead of an ORM | 21:45 |
| jogo | zzzeek: http://paste.openstack.org/show/108458/ | 21:45 |
| jogo | that is ps run at the end of a devstack-gate job | 21:46 |
| sdague | notmyname: yeh, I thought so. | 21:46 |
| sdague | so that's at least one idea... | 21:46 |
| jogo | and nova-api and some other service take up a ton of memory | 21:46 |
| zzzeek | jogo: what am i looking for here | 21:47 |
| jogo | zzzeek: not sure | 21:47 |
| zzzeek | well looks like a whole craplod of python scripts :) | 21:47 |
| jogo | zzzeek: trying to figure out why OpenStack services at rest take up so much memory | 21:47 |
| zzzeek | are we running out of memory ? | 21:47 |
| jogo | zzzeek: yes | 21:47 |
| zzzeek | jogo: well Python is a good start…. :) | 21:47 |
| jogo | zzzeek: but swift is very svelte and is python | 21:48 |
| sdague | jogo: so honestly, I expect if you want a real answer we need to figure out how to get some kind of memory profiler into the mix | 21:48 |
| zzzeek | jogo: OK. are tehse procs like, child procs of a parent or all these all independent ? | 21:48 |
| *** dims_ has quit IRC | 21:48 | |
| zzzeek | jogo: i know ==0 about how openstack runs normally | 21:48 |
| *** dims has joined #openstack-qa | 21:49 | |
| jogo | zzzeek: about to relocate, but maybe sdague can help you get on board | 21:49 |
| notmyname | sdague: I use https://github.com/gholt/experimental-python-mempeek from time to time in swift. it should work for anything that uses a wsgi pipeline | 21:49 |
| zzzeek | jogo: or are we saying, hey, we really need to run 85 python appications indepdently and still take up almost no memory :) | 21:49 |
| sdague | jogo: I'm about to drop for the night | 21:49 |
| jogo | sdague: zzzeek is Mike Bayer, zzzeek sdague is Sean Dague | 21:50 |
| zzzeek | jogo: also are we seeing these procs leak and grow unbounded or they just take up too much to start with | 21:50 |
| cdent | zzzeek: several of those 85 process seem way too big | 21:50 |
| zzzeek | what, like this one ? 1490464 16188 /usr/bin/python /usr/local/bin/swift-object-replicator /etc/swift/object-server/1.conf | 21:50 |
| jogo | zzzeek: it looks like they are growing, and don't take a lot to star with, but will know more soon | 21:50 |
| notmyname | zzzeek: what are those first two numbers? | 21:51 |
| zzzeek | according to this paste it is “SIZE” and “RSS” | 21:51 |
| jogo | zzzeek: size doesn't mean much | 21:51 |
| jogo | zzzeek: the number you want is RSS | 21:51 |
| sdague | notmyname: awesome, thanks for the mempeek bit | 21:52 |
| zzzeek | jogo: well, swapping… | 21:52 |
| jogo | bbiab going to find some coffee | 21:52 |
| jogo | zzzeek: we are hitting the oom-killer as well | 21:53 |
| notmyname | zzzeek: jogo: the swift-object-replicator will walk a filesystem and therefor cache inodes and dentries. not sure if that's a red-herring for whatever problem you are seeing | 21:53 |
| zzzeek | anyweay if youre trying to profile python scripts for memory you pretty much have to just look at python objects present, there are some python memory profiling apps out there that are probably helpful. i usually script my way through gc.get_objects() but that’s a very inconvenient way to do anything nontrivial | 21:53 |
| *** jecarey has quit IRC | 21:53 | |
| *** dims has quit IRC | 21:53 | |
| zzzeek | the other way the program can have issues is if a C library is using up a lot of memory, you wouldnt see that in gc.get_objects() | 21:54 |
| zzzeek | https://github.com/fabianp/memory_profiler looks promising | 21:55 |
| *** dims has joined #openstack-qa | 21:56 | |
| zzzeek | https://pythonhosted.org/Pympler/muppy.html http://guppy-pe.sourceforge.net/#Heapy | 21:57 |
| *** dims has quit IRC | 22:03 | |
| *** dims has joined #openstack-qa | 22:03 | |
| *** Sukhdev has joined #openstack-qa | 22:05 | |
| *** jpich has joined #openstack-qa | 22:05 | |
| jogo | zzzeek: so one possible explanation for why some processes are taking up so much memory is db caching | 22:07 |
| jogo | zzzeek: does that make any sense? | 22:07 |
| zzzeek | jogo: any kind of caching is a good candidate, are we caching…..result sets? ORM objects? queries ? | 22:08 |
| jogo | zzzeek: *shrug* at this point its just a guess | 22:08 |
| zzzeek | jogo: ORM objects are obviously more memory intensive than a simple object like a dict or a tuple. | 22:08 |
| *** dims has quit IRC | 22:08 | |
| zzzeek | jogo: I’m not familiar with what kinds of caching various apps use | 22:08 |
| zzzeek | jogo: which app are we looking at specifically? nova ? | 22:09 |
| jogo | zzzeek: for starters nova-api | 22:09 |
| zzzeek | jogo: my understanding from looking at their SQLA layer is that they create and throw out SQLAclhemy objects pretty quick, everyhing is passed into the so-called “objects” layer | 22:09 |
| zzzeek | jogo: i dont know that there’s caching | 22:09 |
| zzzeek | jogo: im aobut the last person though to consult on specific things that exist in these apps b.c. i hardly know them. but yes if theyre caching ORM objects, that has to be done carefully. | 22:10 |
| zzzeek | i thought keystone was where they have the caching going on | 22:11 |
| morganfainberg | zzzeek, we cache in keystone, some other projects use a non-dogpile memory cache that has a lot of issues (all the ones outlined for the dogpile dict-based cache) | 22:12 |
| morganfainberg | zzzeek, that is that spec i proposed to oslo to help fix down the line, but it's delayed until at least K. | 22:12 |
| zzzeek | jogo: what I’d love to do is, run nova-api and just have a console into it where we can look at heapy or similar | 22:12 |
| zzzeek | I saw some tool that allows this a while back... | 22:13 |
| *** catohornet has joined #openstack-qa | 22:13 | |
| jogo | zzzeek: keystone uses dogpile I think | 22:13 |
| jogo | zzzeek: can you do that in devstack run on a VM? | 22:13 |
| zzzeek | jogo: id have to review what techniques were available for consoling into a python process to see memory stats. | 22:14 |
| zzzeek | jogo: one thing ive done in the past for this thing is to set up a signal listener, you hit the process with the signal and it dumps someting to a text file | 22:14 |
| zzzeek | jogo: usually i used that for dumping all stack traces to find deadlocks | 22:15 |
| zzzeek | jogo: but you could dump a memory profiled result out as well.... | 22:15 |
| *** jgrimm is now known as zz_jgrimm | 22:18 | |
| zzzeek | jogo: let me see if i can try that... | 22:20 |
| *** dimtruck is now known as zz_dimtruck | 22:21 | |
| jogo | zzzeek: awesome | 22:24 |
| *** mwagner_lap has joined #openstack-qa | 22:24 | |
| jogo | zzzeek mtreinish sdague: started a ML thread | 22:26 |
| zzzeek | i like heapy but im really fighting with its stupid interface | 22:26 |
| jogo | zzzeek: thanks for looking into this | 22:31 |
| zzzeek | well if i can get something quick we’ll see | 22:31 |
| jogo | if we can find out why we are using so much memory that should be helpful | 22:31 |
| cdent | jogo zzzeek if you guys discover something can you make sure it makes its way onto the thread that jogo has started? seems like often these sorts of discoveries get lots in irc or gerrit where they are a bit harder to find | 22:32 |
| zzzeek | cdent: i have no idae if im going to learn anything here, im just poking with a stick :) | 22:32 |
| cdent | stick poking is sometimes remarkably effective | 22:32 |
| jogo | cdent: ++ | 22:32 |
| cdent | but sadly I must retire for the evening or else I will have no tomorrow | 22:33 |
| * cdent is in the wrong timezone | 22:33 | |
| cdent | night | 22:33 |
| *** cdent has quit IRC | 22:33 | |
| zzzeek | jogo: is this someting very recent or can i run with a devstack from a coulple of weeks ago | 22:39 |
| jogo | zzzeek: I couple weeks old should be fine | 22:44 |
| zzzeek | well the nova process doesnt seem to care about a SIGUSR1 handler. not sure if that’s some eventlet thing or if they are playing w signals… | 22:49 |
| zzzeek | lets go more hardcore and put it in a background thread :) | 22:49 |
| zzzeek | though that might get whacked too….eventlet.... | 22:50 |
| jogo | zzzeek: sugusr1 is already registered to guru meditation | 22:52 |
| zzzeek | jogo: yes jsut figured that out :) | 22:52 |
| jogo | zzzeek: heh its a very cool poorly documented feature | 22:53 |
| zzzeek | jogo: OK here’s a very initial dump, just starting ujp the server http://paste.openstack.org/show/108513/ | 22:54 |
| zzzeek | so this format is num of objects, total memory | 22:54 |
| zzzeek | this is just typing “nova-api”. but im not sure why i dont see anything DB related | 22:56 |
| zzzeek | doesnt connect to the DB until used somehow ? | 22:56 |
| zzzeek | got some mroe coming…hold on | 22:58 |
| *** dustins has quit IRC | 22:59 | |
| zzzeek | OK http://paste.openstack.org/show/108515/ and http://paste.openstack.org/show/108514/ | 23:00 |
| zzzeek | let me try to get it to load some DB stuff….. | 23:02 |
| jogo | zzzeek: ok so how do I make heads or tails of that? | 23:02 |
| zzzeek | well its Python objects in memory. | 23:03 |
| zzzeek | its each kind of object, the number of them, and estimated memory use of each batch | 23:03 |
| zzzeek | so what you see at the top is the largeet user of memory | 23:03 |
| *** pcrews has quit IRC | 23:03 | |
| openstackgerrit | Joe Gordon proposed a change to openstack/tempest: Add test list_servers_filtered_by_name_regex https://review.openstack.org/119081 | 23:03 |
| jogo | zzzeek: is that after you have done a bunch of API calls? | 23:04 |
| zzzeek | jogo: not yet :) | 23:04 |
| *** rbak has quit IRC | 23:04 | |
| zzzeek | jogo: let me try to get that going | 23:04 |
| *** atiwari has quit IRC | 23:05 | |
| *** pcrews has joined #openstack-qa | 23:05 | |
| *** mkollaro has quit IRC | 23:08 | |
| zzzeek | hey jogo is it the case that if one runs “nova-api”, that isn’t the process that actually talks to SQLAlchemy? I’m putting a giant “raise” right in the middle of nova/nova/db/sqlalchemy/__init__.py and it isn’t triggered | 23:13 |
| zzzeek | this is a clean devstack install | 23:14 |
| jogo | zzzeek: nova-api does talk to SQLA | 23:14 |
| jogo | it should be | 23:14 |
| *** Longgeek has quit IRC | 23:14 | |
| zzzeek | jogo: OK ….let me poke more... | 23:14 |
| jogo | if you do something like nova list | 23:14 |
| *** markmcclain has quit IRC | 23:14 | |
| zzzeek | yup did that | 23:15 |
| jogo | we may lazy load stuff so you may have to do a AP Icall | 23:15 |
| zzzeek | jogo: what “nova” call can i make that def. hits the local DB from within nova-api ? | 23:16 |
| jogo | nova list | 23:16 |
| zzzeek | ah i think i got one | 23:16 |
| zzzeek | OK | 23:16 |
| jogo | or anything with keypairs | 23:16 |
| zzzeek | weird it deosnt even import the DB package | 23:16 |
| zzzeek | ok | 23:16 |
| jogo | you can create nova keypairs | 23:16 |
| jogo | nova flavor-list | 23:17 |
| *** tkammer has joined #openstack-qa | 23:21 | |
| zzzeek | heres one with some sqlalchemy stuff in it http://paste.openstack.org/show/108536/ | 23:24 |
| zzzeek | im still trying to understand the sizes of things. it seems like lots of objects are 1048 in size but others are not. | 23:24 |
| *** hemna is now known as hemna_ | 23:26 | |
| zzzeek | so anyway jogo in this profile all that SQLA stuff is structural, directly proportional to the number of classes that are mapped and how many attributes between them all total, its fixed | 23:28 |
| *** tkammer has quit IRC | 23:32 | |
| *** aimon has joined #openstack-qa | 23:34 | |
| *** yamahata has quit IRC | 23:37 | |
| *** luqas has quit IRC | 23:40 | |
| *** arnaud has quit IRC | 23:44 | |
| *** amotoki_ has quit IRC | 23:44 | |
| *** luqas has joined #openstack-qa | 23:44 | |
| *** jaypipes has quit IRC | 23:46 | |
| *** aysyd has quit IRC | 23:48 | |
| *** weshay has joined #openstack-qa | 23:49 | |
| *** amotoki_ has joined #openstack-qa | 23:50 | |
| *** amotoki_ has quit IRC | 23:51 | |
| *** lakshmiS has quit IRC | 23:55 | |
| mtreinish | jogo: http://logs.openstack.org/94/119894/2/check/check-tempest-dsvm-full/bd0aed1/logs/ps.txt.gz from the job with the worker patch | 23:56 |
| *** rfolco has quit IRC | 23:57 | |
| mtreinish | it looks like the total nova-api footprint went down | 23:57 |
| jogo | mtreinish: did you see the ML thread I started | 23:58 |
| jogo | mtreinish: not sure if that is the right approach | 23:58 |
| jogo | but good to know | 23:58 |
| mtreinish | jogo: yeah, I saw it. I'll probably respond tomorrow | 23:59 |
| jogo | mtreinish: cool | 23:59 |
| *** dims has joined #openstack-qa | 23:59 | |
| jogo | btw does cinder v1 have filtering support for volume list? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!