*** boden has quit IRC | 00:00 | |
*** mriedem has quit IRC | 00:02 | |
*** krtaylor has joined #openstack-qa | 00:07 | |
*** dimtruck is now known as zz_dimtruck | 00:09 | |
*** zz_dimtruck is now known as dimtruck | 00:20 | |
*** jecarey_ has joined #openstack-qa | 00:27 | |
*** jecarey has quit IRC | 00:29 | |
*** dimtruck is now known as zz_dimtruck | 00:30 | |
*** zz_dimtruck is now known as dimtruck | 00:39 | |
*** boden has joined #openstack-qa | 00:56 | |
*** dims has joined #openstack-qa | 00:59 | |
*** boden has quit IRC | 01:01 | |
*** Longgeek has joined #openstack-qa | 01:04 | |
*** yamahata has joined #openstack-qa | 01:13 | |
*** dimtruck is now known as zz_dimtruck | 01:18 | |
*** zz_dimtruck is now known as dimtruck | 01:19 | |
*** dims has quit IRC | 01:26 | |
*** rodrigods_ has joined #openstack-qa | 01:27 | |
*** dims has joined #openstack-qa | 01:30 | |
*** dims_ has joined #openstack-qa | 01:31 | |
*** dims has quit IRC | 01:35 | |
*** dims_ has quit IRC | 01:40 | |
*** dims has joined #openstack-qa | 01:41 | |
*** dims has quit IRC | 01:45 | |
*** edwa66431 has joined #openstack-qa | 01:49 | |
*** edwa6643 has quit IRC | 01:51 | |
*** boden has joined #openstack-qa | 01:57 | |
*** boden has quit IRC | 02:02 | |
*** dimtruck is now known as zz_dimtruck | 02:24 | |
*** lordd has joined #openstack-qa | 02:27 | |
*** lordd has quit IRC | 02:30 | |
*** lordd has joined #openstack-qa | 02:43 | |
*** acfleury has joined #openstack-qa | 02:57 | |
*** boden has joined #openstack-qa | 02:58 | |
*** cmyster_away is now known as cmyster | 02:59 | |
*** edwa6643 has joined #openstack-qa | 03:02 | |
*** boden has quit IRC | 03:03 | |
*** edwa66431 has quit IRC | 03:06 | |
*** rodrigods_ has quit IRC | 03:09 | |
*** lordd has quit IRC | 03:24 | |
*** dkranz has quit IRC | 03:24 | |
*** jaypipes has joined #openstack-qa | 03:25 | |
*** ChanServ sets mode: +o jaypipes | 03:25 | |
*** pcrews has joined #openstack-qa | 03:26 | |
*** lordd has joined #openstack-qa | 03:29 | |
*** lordd has quit IRC | 03:39 | |
*** zz_dimtruck is now known as dimtruck | 03:41 | |
*** pcrews has quit IRC | 03:42 | |
*** edwa6643 has quit IRC | 03:46 | |
*** dimtruck is now known as zz_dimtruck | 03:51 | |
*** acfleury has quit IRC | 03:54 | |
*** chandankumar has joined #openstack-qa | 03:56 | |
*** boden has joined #openstack-qa | 03:59 | |
*** chandankumar has quit IRC | 04:02 | |
*** chandankumar has joined #openstack-qa | 04:02 | |
*** boden has quit IRC | 04:03 | |
*** Poornima has joined #openstack-qa | 04:08 | |
*** Madan has joined #openstack-qa | 04:10 | |
*** gmann has quit IRC | 04:27 | |
*** gmann has joined #openstack-qa | 04:27 | |
*** swamireddy has joined #openstack-qa | 04:31 | |
*** dims has joined #openstack-qa | 04:41 | |
*** dims has quit IRC | 04:46 | |
*** neeti has joined #openstack-qa | 04:48 | |
*** cnesa13 has quit IRC | 04:52 | |
*** Ch00k has joined #openstack-qa | 04:53 | |
*** lordd has joined #openstack-qa | 04:56 | |
*** boden has joined #openstack-qa | 05:00 | |
*** boden has quit IRC | 05:04 | |
*** Longgeek has quit IRC | 05:07 | |
*** lordd has quit IRC | 05:10 | |
*** lordd has joined #openstack-qa | 05:13 | |
*** lordd has quit IRC | 05:16 | |
*** lordd has joined #openstack-qa | 05:17 | |
*** Longgeek has joined #openstack-qa | 05:22 | |
*** Ch00k has quit IRC | 05:22 | |
*** chandankumar has quit IRC | 05:23 | |
*** chandankumar has joined #openstack-qa | 05:26 | |
*** Ch00k has joined #openstack-qa | 05:27 | |
*** yfried has joined #openstack-qa | 05:30 | |
*** lordd has quit IRC | 05:35 | |
*** lordd has joined #openstack-qa | 05:35 | |
yfried | masayukig: cyeoh: https://review.openstack.org/#/c/118232/ this is ready, could you guys please take a look? | 05:36 |
---|---|---|
yfried | andreaf_: ^ | 05:36 |
*** Ch00k has quit IRC | 05:36 | |
yfried | cyeoh: https://review.openstack.org/#/c/102700/26/tempest/services/network/resources.py,cm could you please explain your comment here? suggest an alternative? | 05:37 |
kashyap | yfried, Hi, was looking at this bug - https://bugs.launchpad.net/nova/+bug/1372049 | 05:44 |
kashyap | So, when you set these two | 05:45 |
kashyap | 'notify_nova_on_port_status_changes' 'notify_nova_on_port_data_changes' to False in nova.conf, | 05:45 |
kashyap | Above 64 guests are booted? | 05:45 |
yfried | kashyap: you have to set the relevant values in nova.conf (vif_plugging_*) as well or nova would fail on all | 05:46 |
yfried | kashyap: but assuming you do, then yes - you can boot 100 VMs and they all get ACTIVE status | 05:47 |
*** lordd has quit IRC | 05:48 | |
kashyap | yfried, Can you post your sanitized nova.conf somewhere with the offending case? Just curious, will test in my env. | 05:48 |
yfried | nova.conf? sure | 05:48 |
* kashyap off to make breakfast. | 05:49 | |
kashyap | You can post it in the bug, others debugging it might find it useful too. | 05:50 |
yfried | kashyap: please take a look at my extra comment | 05:50 |
*** evgenyf has joined #openstack-qa | 05:50 | |
kashyap | Sure, in a few. | 05:51 |
*** rdekel has joined #openstack-qa | 05:51 | |
yfried | kashyap: added conf files | 05:53 |
*** Ch00k has joined #openstack-qa | 05:57 | |
*** boden has joined #openstack-qa | 06:00 | |
*** yfried is now known as yfried|afk | 06:03 | |
*** yamamoto has quit IRC | 06:04 | |
*** flaper87 has quit IRC | 06:05 | |
*** flaper87 has joined #openstack-qa | 06:05 | |
*** yfried|afk is now known as yfried | 06:05 | |
*** boden has quit IRC | 06:05 | |
*** bkopilov has quit IRC | 06:11 | |
*** bkopilov has joined #openstack-qa | 06:11 | |
*** pkoniszewski has joined #openstack-qa | 06:14 | |
*** Ch00k has quit IRC | 06:14 | |
kashyap | yfried, Thanks. I'll test in my env today from git & see if I can reproduce it. | 06:17 |
*** pkoniszewski has quit IRC | 06:17 | |
*** yfried is now known as yfried|afk | 06:18 | |
*** rdekel has quit IRC | 06:19 | |
*** yfried|afk is now known as yfried | 06:19 | |
yfried | kashyap: tnx. let me know what you find | 06:20 |
*** k4n0 has joined #openstack-qa | 06:20 | |
*** rdekel has joined #openstack-qa | 06:21 | |
*** Ch00k has joined #openstack-qa | 06:25 | |
*** swamireddy1 has joined #openstack-qa | 06:27 | |
*** e0ne has joined #openstack-qa | 06:28 | |
*** yamamoto has joined #openstack-qa | 06:28 | |
kashyap | Sure. | 06:28 |
*** SridharG has joined #openstack-qa | 06:29 | |
*** e0ne has quit IRC | 06:30 | |
*** swamireddy has quit IRC | 06:30 | |
*** Ch00k has quit IRC | 06:31 | |
*** e0ne has joined #openstack-qa | 06:34 | |
*** nmagnezi has joined #openstack-qa | 06:34 | |
*** yamamoto has quit IRC | 06:35 | |
*** e0ne has quit IRC | 06:36 | |
*** yfried is now known as yfried|afk | 06:38 | |
*** e0ne has joined #openstack-qa | 06:40 | |
*** e0ne has quit IRC | 06:43 | |
*** vigneshvar has joined #openstack-qa | 06:45 | |
*** e0ne has joined #openstack-qa | 06:47 | |
*** e0ne has quit IRC | 06:49 | |
*** afazekas has joined #openstack-qa | 06:53 | |
*** yfried|afk is now known as yfried | 06:58 | |
yfried | anyone here familiar with grenade? | 07:01 |
*** boden has joined #openstack-qa | 07:01 | |
yfried | no idea why this fails on it every time | 07:02 |
yfried | https://review.openstack.org/#/c/122998/ | 07:02 |
*** boden has quit IRC | 07:06 | |
*** jlibosva has joined #openstack-qa | 07:15 | |
*** jordanP has joined #openstack-qa | 07:15 | |
*** e0ne has joined #openstack-qa | 07:29 | |
swamireddy1 | afazekas: Hi | 07:32 |
swamireddy1 | afazekas: In the patch https://review.openstack.org/#/c/115944/, I have added a tests for reboot_instance... | 07:33 |
*** luqas has joined #openstack-qa | 07:33 | |
swamireddy1 | afazekas: But in the tempest/thirdparty/boto/test_ec2_instance_run.py - file, in test_run_terminate_instance(), with one instance, the for loop used...The for loop used for other test cases also - with one intance | 07:35 |
swamireddy1 | afazekas: Is there any reason for using a for loop for one instance? | 07:35 |
*** jlanoux has joined #openstack-qa | 07:37 | |
*** Albert_13 has joined #openstack-qa | 07:38 | |
*** e0ne has quit IRC | 07:38 | |
*** e0ne has joined #openstack-qa | 07:40 | |
*** gszasz has joined #openstack-qa | 07:40 | |
*** Longgeek has quit IRC | 07:40 | |
*** luqas__ has joined #openstack-qa | 07:41 | |
*** luqas has quit IRC | 07:44 | |
*** mpaolino has joined #openstack-qa | 07:48 | |
afazekas | swamireddy1: :) | 07:48 |
afazekas | swamireddy1: it could work without the for loop, but in theory a reservation can have multiple instances. | 07:49 |
afazekas | asserting the number of instances in the reservation (1) and using the [0]th could work in most cases | 07:53 |
*** gszasz has quit IRC | 07:53 | |
*** tkammer has joined #openstack-qa | 07:58 | |
*** jlibosva has quit IRC | 07:59 | |
*** Longgeek has joined #openstack-qa | 08:01 | |
*** jlibosva has joined #openstack-qa | 08:01 | |
*** safchain has joined #openstack-qa | 08:02 | |
*** tkammer has quit IRC | 08:03 | |
*** gfidente has joined #openstack-qa | 08:08 | |
afazekas | swamireddy1: as dkranz pointed out that test case was created for terminating the instance without waiting for runing/ACTIVE state. might be better to create new one for reboot. | 08:09 |
*** bkopilov has quit IRC | 08:10 | |
afazekas | Looks like the usage of for loop for one instance is confusing (and slower) most places (maybe exuding termination) it can be changed to `assert len 1` and using the first[0] one. | 08:12 |
*** markus_z has joined #openstack-qa | 08:13 | |
*** e0ne has quit IRC | 08:13 | |
*** e0ne has joined #openstack-qa | 08:16 | |
*** tkammer has joined #openstack-qa | 08:17 | |
*** jlibosva1 has joined #openstack-qa | 08:18 | |
*** chandankumar has quit IRC | 08:18 | |
*** jlibosva has quit IRC | 08:21 | |
*** luqas__ has quit IRC | 08:24 | |
swamireddy1 | afazekas: Ok...I created originally as separate test case as test_run_reboot_terminate¨...later as its same as test_run_terminate test case, merged with it.. | 08:25 |
swamireddy1 | afazekas: Now Iam confused... | 08:26 |
*** boden has joined #openstack-qa | 08:27 | |
*** swamireddy2 has joined #openstack-qa | 08:29 | |
swamireddy2 | afazekas: yes, the for loop not needed for one instance, os we need to either update the same full file or remove the for loop for this patch, rest of the file can be done later | 08:29 |
*** bkopilov has joined #openstack-qa | 08:31 | |
*** swamireddy1 has quit IRC | 08:31 | |
*** k4n0 has quit IRC | 08:33 | |
*** ildikov has joined #openstack-qa | 08:36 | |
*** bkopilov has quit IRC | 08:38 | |
*** gszasz has joined #openstack-qa | 08:41 | |
*** Poornima has quit IRC | 08:41 | |
*** neeti_ has joined #openstack-qa | 08:45 | |
*** neeti has quit IRC | 08:47 | |
*** liusheng has joined #openstack-qa | 08:48 | |
*** tosky has joined #openstack-qa | 08:49 | |
*** swamireddy has joined #openstack-qa | 08:50 | |
*** k4n0 has joined #openstack-qa | 08:50 | |
*** swamireddy2 has quit IRC | 08:54 | |
*** cnesa13 has joined #openstack-qa | 08:54 | |
*** bkopilov has joined #openstack-qa | 08:55 | |
*** luqas__ has joined #openstack-qa | 08:56 | |
*** Ch00k has joined #openstack-qa | 08:57 | |
*** Ch00k has quit IRC | 08:57 | |
*** cnesa13 has quit IRC | 09:01 | |
*** cnesa13 has joined #openstack-qa | 09:02 | |
Albert_13 | hi, I'm developing custom tests, and need to automate some communications betwen servers with different protocols, and I'm considering using twisted framework, but if later on I would like to include my tests into the community, would be a burden the usage of sutch thirdparty frameworks? | 09:05 |
*** Ch00k has joined #openstack-qa | 09:06 | |
*** qba73 has joined #openstack-qa | 09:06 | |
jordanP | Albert_13, twisted is a huge framework... the best way to answer this question would be to start a new threand on the qa mailing list | 09:07 |
*** Longgeek has quit IRC | 09:08 | |
*** Ch00k has quit IRC | 09:08 | |
jordanP | Albert_13, my opininon : if contributing upstream could be beneficial to you, you should concider writting your tests the way the community does it. It will be easier for everyone | 09:08 |
Albert_13 | Thanks Jordan | 09:09 |
*** Ch00k has joined #openstack-qa | 09:09 | |
*** Longgeek has joined #openstack-qa | 09:17 | |
*** e0ne has quit IRC | 09:24 | |
*** swamireddy has quit IRC | 09:25 | |
*** e0ne has joined #openstack-qa | 09:26 | |
*** swamireddy has joined #openstack-qa | 09:27 | |
*** yfried is now known as yfried|afk | 09:29 | |
*** yfried|afk is now known as yfried | 09:29 | |
*** zz_dimtruck is now known as dimtruck | 09:34 | |
*** tkammer has quit IRC | 09:36 | |
*** Ch00k has quit IRC | 09:37 | |
*** yfried is now known as yfried|afk | 09:39 | |
*** neeti_ has quit IRC | 09:40 | |
*** neeti has joined #openstack-qa | 09:41 | |
*** chandankumar has joined #openstack-qa | 09:42 | |
*** k4n0 has quit IRC | 09:44 | |
*** dimtruck is now known as zz_dimtruck | 09:44 | |
*** swamireddy1 has joined #openstack-qa | 09:47 | |
*** yfried|afk is now known as yfried | 09:48 | |
*** tkammer has joined #openstack-qa | 09:49 | |
swamireddy1 | afazekas: what we do here with adding reboot_instance test? could please advise/ | 09:50 |
*** swamireddy has quit IRC | 09:50 | |
*** swamireddy has joined #openstack-qa | 09:51 | |
*** cnesa13 has quit IRC | 09:51 | |
*** yfried_ has joined #openstack-qa | 09:52 | |
*** yfried__ has joined #openstack-qa | 09:53 | |
*** swamireddy1 has quit IRC | 09:54 | |
*** neeti has quit IRC | 09:55 | |
*** yfried has quit IRC | 09:55 | |
*** cnesa14 has joined #openstack-qa | 09:55 | |
*** tkammer has quit IRC | 09:55 | |
*** Madan has quit IRC | 09:56 | |
*** yfried_ has quit IRC | 09:56 | |
*** andreaf_ is now known as andreaf | 10:01 | |
*** Madan has joined #openstack-qa | 10:02 | |
andreaf | afazekas, mkoderer_ : https://review.openstack.org/#/q/topic:bp/resource-cleanup,n,z - the framework for the series is merged now, and all changes passing check - they are all identical changes so should be easy to review | 10:04 |
*** luqas__ has quit IRC | 10:04 | |
*** luqas has joined #openstack-qa | 10:06 | |
*** luqas has quit IRC | 10:08 | |
*** ildikov_ has joined #openstack-qa | 10:10 | |
*** tkammer has joined #openstack-qa | 10:10 | |
*** ildikov has quit IRC | 10:12 | |
*** Longgeek has quit IRC | 10:16 | |
*** aix has joined #openstack-qa | 10:16 | |
*** luqas has joined #openstack-qa | 10:16 | |
*** Longgeek has joined #openstack-qa | 10:20 | |
*** luqas has quit IRC | 10:20 | |
*** mitz_ has joined #openstack-qa | 10:22 | |
*** k4n0 has joined #openstack-qa | 10:29 | |
*** evgenyf has quit IRC | 10:33 | |
*** Albert_13 has quit IRC | 10:34 | |
*** Albert_13 has joined #openstack-qa | 10:35 | |
*** luqas has joined #openstack-qa | 10:36 | |
*** yamahata has quit IRC | 10:37 | |
*** naggappan has joined #openstack-qa | 10:40 | |
*** e0ne has quit IRC | 10:43 | |
naggappan | hi i have done openstack-manual commit and i am clear about how to contribute now. Looking forward to commit to tempest and neutron. I checked the https://etherpad.openstack.org/p/icehouse-summit-qa-neutron | 10:45 |
naggappan | but most of them are taken by someone else | 10:45 |
naggappan | is there any thing which someone can suggest me | 10:45 |
*** zz_johnthetubagu is now known as johnthetubaguy | 10:47 | |
*** Poornima has joined #openstack-qa | 10:47 | |
*** dmellado has quit IRC | 10:48 | |
*** evgenyf has joined #openstack-qa | 10:50 | |
chandankumar | naggappan, hello | 10:50 |
chandankumar | naggappan, have you tried to get familiar with tempest? | 10:50 |
*** zigo has quit IRC | 10:50 | |
*** dmellado has joined #openstack-qa | 10:51 | |
*** zigo has joined #openstack-qa | 10:52 | |
naggappan | chandankumar, i tried to run the tempest and checked. But not gone throught the code of tempest much. But i have created my own scenarios for our office purpose | 10:52 |
naggappan | if adding more scenarios is best way to start, please point me the link where i can see the blue print of scenarios related to neutron | 10:53 |
*** yfried__ has quit IRC | 10:53 | |
chandankumar | naggappan, try to go through this link https://wiki.openstack.org/wiki/Neutron/TempestAPITests | 10:53 |
chandankumar | naggappan, like this blueprint https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests-in-icehouse | 10:54 |
naggappan | ok basic like create a vm allocate newtork,subnet automatically i have done. Fine let me gothroght blueprints of scenarios and try to get it done | 10:56 |
chandankumar | naggappan, just shoot a mail to the owner on the things you are going to work, | 10:57 |
naggappan | sure | 10:57 |
*** dims has joined #openstack-qa | 10:59 | |
*** nmagnezi has quit IRC | 11:02 | |
*** tkammer has quit IRC | 11:05 | |
*** mkollaro has joined #openstack-qa | 11:08 | |
*** mwagner_lap has quit IRC | 11:08 | |
*** yfried__ has joined #openstack-qa | 11:09 | |
*** Ch00k has joined #openstack-qa | 11:10 | |
*** bkopilov has quit IRC | 11:10 | |
*** nmagnezi has joined #openstack-qa | 11:15 | |
*** jyuso065 has quit IRC | 11:15 | |
*** neeti has joined #openstack-qa | 11:17 | |
*** tosky has quit IRC | 11:17 | |
*** tkammer has joined #openstack-qa | 11:18 | |
EmilienM | jogo: sdague: I would need a review on javelin: https://review.openstack.org/#/c/121648/ | 11:21 |
*** om has joined #openstack-qa | 11:25 | |
*** luqas has quit IRC | 11:26 | |
*** yfried__ is now known as yfried|afk | 11:28 | |
*** evgenyf has quit IRC | 11:28 | |
*** bkopilov has joined #openstack-qa | 11:28 | |
*** bkopilov has quit IRC | 11:32 | |
*** bkopilov has joined #openstack-qa | 11:33 | |
*** zz_jgrimm is now known as jgrimm | 11:33 | |
*** om has quit IRC | 11:34 | |
*** evgenyf has joined #openstack-qa | 11:40 | |
*** naggappan has quit IRC | 11:40 | |
*** e0ne has joined #openstack-qa | 11:42 | |
*** tosky has joined #openstack-qa | 11:43 | |
*** cmyster is now known as cmyster_away | 11:46 | |
*** e0ne has quit IRC | 11:50 | |
*** om has joined #openstack-qa | 11:52 | |
*** naggappan has joined #openstack-qa | 11:56 | |
*** yfried|afk is now known as yfried__ | 11:57 | |
*** e0ne has joined #openstack-qa | 11:58 | |
*** swamireddy has quit IRC | 12:02 | |
*** luqas has joined #openstack-qa | 12:03 | |
*** aysyanne has joined #openstack-qa | 12:04 | |
*** rdekel has quit IRC | 12:04 | |
*** edwa6643 has joined #openstack-qa | 12:07 | |
*** mmedvede has joined #openstack-qa | 12:13 | |
*** openstackgerrit has joined #openstack-qa | 12:14 | |
*** edwa6643 has quit IRC | 12:17 | |
*** om has quit IRC | 12:18 | |
*** edwa6643 has joined #openstack-qa | 12:20 | |
*** yfried__ is now known as yfried|afk | 12:25 | |
*** weshay has joined #openstack-qa | 12:27 | |
*** boden has quit IRC | 12:30 | |
*** boden has joined #openstack-qa | 12:30 | |
*** acfleury has joined #openstack-qa | 12:31 | |
*** yfried|afk is now known as yfried__ | 12:32 | |
*** dims has quit IRC | 12:38 | |
*** mmedvede has quit IRC | 12:38 | |
*** andreaf is now known as andreaf_ | 12:38 | |
*** dims has joined #openstack-qa | 12:38 | |
*** liusheng has quit IRC | 12:39 | |
*** nmagnezi has quit IRC | 12:43 | |
*** rfolco has joined #openstack-qa | 12:44 | |
*** mmedvede has joined #openstack-qa | 12:45 | |
*** sballe has joined #openstack-qa | 12:47 | |
*** rdekel has joined #openstack-qa | 12:48 | |
*** cnesa14 has quit IRC | 12:50 | |
*** Madan has quit IRC | 12:50 | |
*** gszasz has quit IRC | 12:53 | |
*** vigneshvar has quit IRC | 12:54 | |
*** cnesa14 has joined #openstack-qa | 12:54 | |
*** gszasz has joined #openstack-qa | 12:58 | |
*** nmagnezi has joined #openstack-qa | 12:58 | |
*** jlibosva1 is now known as jlibosva | 12:59 | |
*** ildikov has joined #openstack-qa | 13:00 | |
*** ildikov_ has quit IRC | 13:03 | |
*** toure has joined #openstack-qa | 13:03 | |
*** mestery has quit IRC | 13:09 | |
*** mestery has joined #openstack-qa | 13:09 | |
*** sballe has quit IRC | 13:12 | |
*** tkammer has quit IRC | 13:15 | |
*** swamireddy has joined #openstack-qa | 13:15 | |
*** luqas has left #openstack-qa | 13:18 | |
*** bknudson has quit IRC | 13:21 | |
yfried__ | sdague: https://bugs.launchpad.net/neutron/+bug/1372049 please let me know if the info I provided is any good | 13:22 |
yfried__ | kashyap: were you able to reproduce the bug from this morning? | 13:22 |
kashyap | yfried__, Hi, I ended up doing some "yak shaving" - as I was stuck with a Keystone issue while trying to invoke db sync | 13:23 |
kashyap | So, mostly I investigated that (it's with Juno, M3, on Fedora 21/Rawhide) - https://bugzilla.redhat.com/show_bug.cgi?id=1145113 | 13:23 |
*** mriedem has joined #openstack-qa | 13:24 | |
kashyap | yfried__, So, I'm afraid I didn't get to it yet. And, starting tomorrow my next weeks are just too crazy, too many things going on | 13:25 |
yfried__ | kashyap: np. just wondered if someone else was able to reproduce | 13:26 |
openstackgerrit | Sergey Shnaidman proposed a change to openstack/tempest: DHCPv6 network tests https://review.openstack.org/117458 | 13:26 |
*** superdan is now known as dansmith | 13:29 | |
*** tkammer has joined #openstack-qa | 13:30 | |
*** swamireddy has quit IRC | 13:37 | |
*** sballe_ has quit IRC | 13:37 | |
*** sballe has joined #openstack-qa | 13:37 | |
*** zz_dimtruck is now known as dimtruck | 13:44 | |
*** toure has quit IRC | 13:45 | |
*** toure has joined #openstack-qa | 13:45 | |
*** sabeen1 has joined #openstack-qa | 13:46 | |
*** dimtruck is now known as zz_dimtruck | 13:53 | |
*** proverma has joined #openstack-qa | 13:53 | |
*** chandankumar has quit IRC | 13:54 | |
*** bknudson has joined #openstack-qa | 13:56 | |
*** proverma has quit IRC | 13:58 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for glance failing to start on stable https://review.openstack.org/123134 | 13:58 |
sdague | mriedem: so https://bugs.launchpad.net/devstack/+bug/1303813 is showing a ton of hits, but it's not actually the bug that was intended for | 13:59 |
sdague | the real issue is the one above there | 13:59 |
sdague | how would you feel about deleting the signature for 1303813 | 13:59 |
mriedem | looking | 14:00 |
mriedem | sdague: heh, you pushed the query - feel free to remove it :) | 14:02 |
mriedem | and you invalidated the bug | 14:02 |
sdague | yeh, well it's not being helpful :) | 14:02 |
sdague | unfortunately we can't do query subtraction | 14:02 |
mriedem | sdague: i'm assuming it's a dupe of bug 1331274 | 14:02 |
sdague | actually, it's not | 14:02 |
sdague | it's actually the fail on the old side | 14:03 |
sdague | but ... close enough | 14:03 |
*** gfidente has quit IRC | 14:04 | |
*** dustins has joined #openstack-qa | 14:04 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove glance not starting signature https://review.openstack.org/123136 | 14:05 |
sdague | ok, there is the delete | 14:05 |
*** rdekel has quit IRC | 14:06 | |
*** tkammer has quit IRC | 14:08 | |
*** rbak has joined #openstack-qa | 14:08 | |
*** dkranz has joined #openstack-qa | 14:10 | |
*** andreaf has joined #openstack-qa | 14:15 | |
*** gfidente has joined #openstack-qa | 14:16 | |
*** gordc has joined #openstack-qa | 14:19 | |
*** k4n0 has quit IRC | 14:19 | |
mriedem | sdague: commented on https://review.openstack.org/#/c/123134/ | 14:20 |
*** tkammer has joined #openstack-qa | 14:20 | |
mriedem | sdague: why not tag on g-api? | 14:20 |
sdague | mriedem: module:glance is pretty effective | 14:20 |
*** sballe_ has joined #openstack-qa | 14:21 | |
mriedem | instead of the tags, since it's on g-api and g-reg | 14:21 |
sdague | yeh, I actually expect hitting the fields is faster than tags | 14:21 |
*** neeti has quit IRC | 14:21 | |
sdague | so I realize that both the fields I added don't change the hits | 14:21 |
sdague | but they do limit the search space dramatically | 14:22 |
sdague | so this should make it a very cheap query | 14:22 |
mriedem | always a flare for the dramatic with you :P | 14:22 |
*** yamahata has joined #openstack-qa | 14:23 | |
mriedem | +W on the series | 14:23 |
*** sballe_ has quit IRC | 14:23 | |
mriedem | i plan on cleaning up the closed queries today | 14:23 |
*** jecarey_ has quit IRC | 14:24 | |
*** chandankumar has joined #openstack-qa | 14:24 | |
sdague | cool | 14:24 |
*** sballe has quit IRC | 14:24 | |
sdague | mriedem: how you feeling about this - https://review.openstack.org/#/c/122150/ ? | 14:25 |
sdague | oh, that should have been in -nova | 14:25 |
sdague | but take a look regardless | 14:25 |
*** zz_dimtruck is now known as dimtruck | 14:29 | |
*** andreaf has quit IRC | 14:31 | |
*** atiwari has joined #openstack-qa | 14:31 | |
*** nmagnezi has quit IRC | 14:32 | |
bknudson | I'm going to see if I can figure out what's going on with https://bugs.launchpad.net/glance/+bug/1372422 | 14:32 |
*** acfleury has quit IRC | 14:33 | |
*** acfleury has joined #openstack-qa | 14:33 | |
bknudson | "Glance exploding on configuration parsing" | 14:34 |
*** chandankumar has quit IRC | 14:36 | |
*** yfried__ has quit IRC | 14:37 | |
bknudson | Looks like the log file in the bug is already deleted? http://logs.openstack.org/98/122998/2/check/check-grenade-dsvm-partial-ncpu-icehouse/790f63e/logs/old/screen-g-api.txt | 14:40 |
bknudson | http://logs.openstack.org/98/122998/2/check/check-grenade-dsvm-partial-ncpu-icehouse/790f63e/logs/old/screen-g-api.txt.gz worked | 14:40 |
bknudson | still no stack trace :( | 14:40 |
*** Poornima has quit IRC | 14:45 | |
bknudson | I tried starting glance master using keystoneclient.middleware and started up fine... not sure how to recreate this. | 14:45 |
*** sballe has joined #openstack-qa | 14:48 | |
*** ildikov has quit IRC | 14:48 | |
bknudson | I tried checking out glance to stable/icehouse and still doesn't fail. | 14:49 |
*** Guest75828 has joined #openstack-qa | 14:52 | |
openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Update signature for pip download timeout bug 1270710 https://review.openstack.org/123147 | 14:53 |
mriedem | sdague: jogo: ^ | 14:53 |
bknudson | pip freeze for the failing job shows "oslo.config==1.2.2.dev3.g7d68dc8 ", whereas my system has "oslo.config==1.4.0" | 14:53 |
*** jecarey has joined #openstack-qa | 14:54 | |
*** SridharG has left #openstack-qa | 14:54 | |
bknudson | I think that was it... I was able to recreate by "sudo pip install -U "oslo.config==1.2.1" | 14:56 |
mtreinish | bknudson: oh, yeah that would probably break something... | 14:56 |
bknudson | it doesn't fail with "oslo.config==1.3.0" | 14:57 |
bknudson | current keystoneclient has "oslo.config>=1.4.0" ... so that would indicate this wasn't running with the correct requirements | 14:58 |
openstackgerrit | Rob Crittenden proposed a change to openstack-dev/devstack: Configure endpoints to use SSL natively or via proxy https://review.openstack.org/98854 | 14:59 |
mtreinish | bknudson: so it's running icehouse on the old side of a grenade run so I'm not sure if the installed version matches that or not. Also that oslo.config version means it's being installed from the git repo | 15:01 |
*** kmartin has joined #openstack-qa | 15:01 | |
mtreinish | bknudson: but something definitely isn't getting installed correctly if that's the version in pip-freeze, I'd take a look at the devstack log to see what's installing that version | 15:01 |
bknudson | oslo.config has a stable/havana branch, but doesn't have stable/icehouse... it looks like grenade is checking out stable/havana. | 15:05 |
mtreinish | dhellmann: ^^^ | 15:06 |
*** naggappan has quit IRC | 15:06 | |
*** sballe_ has joined #openstack-qa | 15:06 | |
bknudson | we'll see if it tries to check out stable/icehouse | 15:06 |
*** hemnass is now known as hemna | 15:06 | |
dhellmann | mtreinish: that's odd | 15:07 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add bug for glance failing to start on stable https://review.openstack.org/123134 | 15:07 |
mtreinish | bknudson: yeah, that's probably what's causing the problems. There might be a devstack-gate, devstack, g-r, or grenade fix then. | 15:07 |
mtreinish | dhellmann: yeah, especially since this just popped up and I don't think it's on every run | 15:08 |
dhellmann | mtreinish: We may have said we would go back and create the branches when we needed them, not accounting for grenade's behavior. | 15:08 |
bknudson | http://logs.openstack.org/98/122998/2/check/check-grenade-dsvm-partial-ncpu-icehouse/790f63e/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-09-22_09_28_18_965 | 15:08 |
bknudson | it does git_has_branch openstack/oslo.config stable/icehouse | 15:08 |
bknudson | but then it looks like it checks out master anyways. | 15:09 |
*** sballe has quit IRC | 15:09 | |
dhellmann | bknudson: so the new client code doesn't work with the old oslo.config library? | 15:10 |
bknudson | dhellmann: we had a recent change to the auth token middleware (that I think we talked to you about) | 15:11 |
dhellmann | yeah, this is feeling vaguely familiar | 15:11 |
bknudson | where it converts the strings in api-paste.ini to the actual type. | 15:11 |
dhellmann | pulling options -- right | 15:11 |
*** toure has quit IRC | 15:11 | |
bknudson | and that's where it's hitting this with what looks like some pretty old oslo.config code | 15:11 |
bknudson | I think this is going to be a general issue going forward, though... grenade is running with current keystoneclient not havana/icehouse keystoneclient, but that's running with old oslo.config | 15:13 |
*** toure has joined #openstack-qa | 15:13 | |
bknudson | seems like grenade should be running with current oslo.config? | 15:13 |
dhellmann | yeah, it seems like it should either update the requirements for all of the clients or use the old clients until those requirements can be updated | 15:14 |
mkollaro | is there some way to set tox to use globally installed packages and not create a virtualenv? I know using virtualenv is the main purpose of tox, but I need to check my package dependencies and I can't use the command directly - I'd have to rewrite a lot of code | 15:14 |
bknudson | keystoneclient had oslo.config>=1.4.0 | 15:14 |
dhellmann | bknudson: right, but apparently a different version of oslo.config is installed? | 15:14 |
bknudson | dhellmann: the pip freeze output says oslo.config==1.2.2.dev3.g7d68dc8 | 15:15 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove glance not starting signature https://review.openstack.org/123136 | 15:15 |
bknudson | dhellmann: http://logs.openstack.org/98/122998/2/check/check-grenade-dsvm-partial-ncpu-icehouse/790f63e/logs/pip-freeze.txt.gz | 15:15 |
bknudson | people were saying this was intermittent... maybe we can find one that passed and check out its oslo.config version. | 15:16 |
dhellmann | bknudson: I'd have to look at the install dance grenade does, but I suspect that's caused by installing the clients, then going back and installing that oslo lib from source (that's the only way to get a dev version) | 15:16 |
dhellmann | good idea | 15:16 |
bknudson | dhellmann: I just tried `sudo pip install -U 1.2.1` | 15:17 |
bknudson | `sudo pip install -U "oslo.config==1.2.1" | 15:17 |
bknudson | so here's a recent stable/icehouse build, where oslo.config==1.4.0 -- http://logs.openstack.org/72/122772/4/check/check-grenade-dsvm/1c316e8/logs/pip-freeze.txt.gz | 15:19 |
*** mlavalle has joined #openstack-qa | 15:19 | |
*** Longgeek_ has joined #openstack-qa | 15:19 | |
dkranz | andreaf_: What was the result of how to handle all the dependent resource cleanup patches? | 15:20 |
dkranz | andreaf_: Are we just going to revew all of them as is? | 15:20 |
*** sballe_ has quit IRC | 15:21 | |
*** jamielennox has quit IRC | 15:22 | |
*** Longgeek has quit IRC | 15:22 | |
*** pcrews has joined #openstack-qa | 15:23 | |
dhellmann | bknudson: we have 2 problems here: why does this only fail for some jobs? and how do we make keystoneclient's option type conversion work for all cases? | 15:23 |
*** sballe has joined #openstack-qa | 15:28 | |
*** toure has quit IRC | 15:31 | |
*** evgenyf has quit IRC | 15:31 | |
*** toure has joined #openstack-qa | 15:31 | |
bknudson | dhellmann: I started an etherpad for notes: https://etherpad.openstack.org/p/bug1372422 | 15:35 |
*** Guest75828 is now known as annegentle | 15:38 | |
*** tkammer has quit IRC | 15:38 | |
*** Ch00k has quit IRC | 15:41 | |
*** jhenner has joined #openstack-qa | 15:43 | |
*** jhenner1 has quit IRC | 15:43 | |
openstackgerrit | A change was merged to openstack/tempest: Migrate computev2 API tests to resource_* fixtures https://review.openstack.org/121538 | 15:44 |
*** jhenner has quit IRC | 15:47 | |
*** chandankumar has joined #openstack-qa | 15:50 | |
bknudson | dhellmann: does keystoneclient need to support the icehouse version of oslo.config? | 15:52 |
*** Longgeek_ has quit IRC | 15:55 | |
dhellmann | bknudson: Maybe? I think grenade is supposed to be codifying our rules for things like that, but it seems like it has inconsistent behavior. | 15:56 |
*** jlibosva has quit IRC | 15:58 | |
*** sballe has quit IRC | 16:02 | |
*** jlanoux has quit IRC | 16:04 | |
*** yfried__ has joined #openstack-qa | 16:05 | |
*** sballe has joined #openstack-qa | 16:06 | |
*** sballe has quit IRC | 16:12 | |
*** chandankumar has quit IRC | 16:14 | |
*** e0ne has quit IRC | 16:17 | |
*** sdague has quit IRC | 16:25 | |
*** Albert_13 has quit IRC | 16:30 | |
*** markmcclain has joined #openstack-qa | 16:36 | |
notmyname | is there a single list of the hacking rules anywhere? | 16:36 |
*** aimon has quit IRC | 16:36 | |
*** mkollaro has quit IRC | 16:36 | |
*** yfried__ is now known as yfried|afk | 16:38 | |
*** yjiang5 has joined #openstack-qa | 16:39 | |
*** proverma has joined #openstack-qa | 16:40 | |
*** yfried|afk is now known as yfried__ | 16:41 | |
*** asselin has quit IRC | 16:48 | |
*** asselin has joined #openstack-qa | 16:48 | |
*** eglynn__ is now known as eglynn-officeafk | 16:52 | |
*** yfried__ is now known as yfried|afk | 16:53 | |
*** markus_z has quit IRC | 16:53 | |
*** sdague has joined #openstack-qa | 16:55 | |
*** mpaolino has quit IRC | 16:57 | |
openstackgerrit | pritesh proposed a change to openstack-dev/devstack: Remove the Cisco Nexus monolithic plugin support https://review.openstack.org/121623 | 16:59 |
*** qba73 has quit IRC | 17:03 | |
*** jordanP has quit IRC | 17:08 | |
*** afazekas has quit IRC | 17:08 | |
*** yamahata has quit IRC | 17:08 | |
*** marun has joined #openstack-qa | 17:10 | |
*** e0ne has joined #openstack-qa | 17:11 | |
openstackgerrit | Thomas Bechtold proposed a change to openstack-dev/devstack: Use correct WSGI dir on SUSE distros https://review.openstack.org/123188 | 17:11 |
*** aimon has joined #openstack-qa | 17:14 | |
*** Ch00k has joined #openstack-qa | 17:19 | |
*** harlowja_away is now known as harlowja | 17:21 | |
*** suneelb has joined #openstack-qa | 17:25 | |
*** chandankumar has joined #openstack-qa | 17:29 | |
*** melwitt has joined #openstack-qa | 17:31 | |
*** krtaylor has quit IRC | 17:32 | |
*** jpmontez has joined #openstack-qa | 17:35 | |
*** johnthetubaguy is now known as zz_johnthetubagu | 17:39 | |
*** yfried|afk is now known as yfried__ | 17:41 | |
*** krtaylor has joined #openstack-qa | 17:41 | |
*** gszasz has quit IRC | 17:43 | |
*** cnesa14 has quit IRC | 17:50 | |
*** gfidente has quit IRC | 17:52 | |
bknudson | dhellmann: the change to auth_token isn't especially complicated, so maybe this is the easiest solution: https://review.openstack.org/123198 | 17:53 |
*** aix has quit IRC | 17:55 | |
*** jpmontez has quit IRC | 17:58 | |
*** salv-orlando has joined #openstack-qa | 18:02 | |
*** proverma has quit IRC | 18:04 | |
*** harlowja has quit IRC | 18:05 | |
*** harlowja_ has joined #openstack-qa | 18:05 | |
openstackgerrit | John Davidge proposed a change to openstack-dev/devstack: Add IPv6 support for tenant data network https://review.openstack.org/87987 | 18:07 |
*** yjiang5 has quit IRC | 18:14 | |
*** chandankumar has quit IRC | 18:15 | |
*** chandankumar has joined #openstack-qa | 18:17 | |
*** marun has quit IRC | 18:18 | |
*** tosky has quit IRC | 18:19 | |
*** chandankumar has quit IRC | 18:26 | |
*** yjiang5 has joined #openstack-qa | 18:30 | |
*** e0ne has quit IRC | 18:35 | |
*** sballe has joined #openstack-qa | 18:35 | |
*** e0ne has joined #openstack-qa | 18:35 | |
openstackgerrit | John Davidge proposed a change to openstack-dev/devstack: Add IPv6 support for tenant data network https://review.openstack.org/87987 | 18:41 |
*** jpmontez has joined #openstack-qa | 18:45 | |
openstackgerrit | Baodong (Robert) Li proposed a change to openstack-dev/devstack: Add IPv6 support for tenant data network https://review.openstack.org/87987 | 18:46 |
*** kmartin has quit IRC | 18:47 | |
*** openstack has joined #openstack-qa | 18:56 | |
openstackgerrit | Armando Migliaccio proposed a change to openstack/tempest: Fix MismatchError for LB scenario test https://review.openstack.org/120533 | 18:58 |
*** sabeen1 has quit IRC | 18:59 | |
*** proverma has joined #openstack-qa | 19:00 | |
*** dustins has quit IRC | 19:00 | |
*** dustins has joined #openstack-qa | 19:03 | |
openstackgerrit | A change was merged to openstack-dev/devstack: Function for auth_token middleware config https://review.openstack.org/122882 | 19:04 |
openstackgerrit | A change was merged to openstack-dev/devstack: Configurable token hashing algorithm https://review.openstack.org/116535 | 19:05 |
*** rockyg has joined #openstack-qa | 19:07 | |
*** yfried__ is now known as yfried|afk | 19:07 | |
*** rwsu has joined #openstack-qa | 19:10 | |
*** sballe_ has joined #openstack-qa | 19:21 | |
*** proverma has quit IRC | 19:22 | |
*** sballe has quit IRC | 19:24 | |
*** rfolco has quit IRC | 19:24 | |
*** safchain has quit IRC | 19:27 | |
*** sabeen1 has joined #openstack-qa | 19:27 | |
*** sballe_ has quit IRC | 19:32 | |
*** weshay has quit IRC | 19:38 | |
*** vigneshvar has joined #openstack-qa | 19:41 | |
*** rfolco has joined #openstack-qa | 19:46 | |
*** mwagner_lap has joined #openstack-qa | 19:47 | |
*** jhenner has joined #openstack-qa | 19:48 | |
*** weshay has joined #openstack-qa | 19:51 | |
*** proverma has joined #openstack-qa | 19:51 | |
*** marun has joined #openstack-qa | 19:51 | |
*** markmcclain has quit IRC | 19:57 | |
*** jpmontez has quit IRC | 19:59 | |
*** proverma has quit IRC | 20:02 | |
*** proverma has joined #openstack-qa | 20:03 | |
*** proverma has quit IRC | 20:07 | |
*** jpmontez has joined #openstack-qa | 20:07 | |
*** proverma has joined #openstack-qa | 20:08 | |
*** e0ne has quit IRC | 20:09 | |
*** boden has quit IRC | 20:11 | |
*** rockyg has quit IRC | 20:13 | |
clarkb | mtreinish: I am looking for an reasonably stable experimental job to run on nodes whose image was built by dib. basically a sanity check before we switch more things to dib built images | 20:14 |
clarkb | mtreinish: would the tempest ceph job be good? or have any other ideas? | 20:15 |
*** sabeen1 has quit IRC | 20:18 | |
mtreinish | clarkb: hmm, I've never looked at results from the tempest ceph job | 20:19 |
clarkb | mtreinish: np I can probably dig them up | 20:19 |
clarkb | mtreinish: do you know of any tempest cahnges that are testing the experimental jobs? | 20:20 |
mtreinish | clarkb: well anything that touches marconi, or the nova v3 api will run the experimental jobs on them because those paths are only enabled in experimental jobs | 20:21 |
*** andreaf has joined #openstack-qa | 20:21 | |
mtreinish | but other than that I don't think we run it as standard operating procedure too often | 20:21 |
clarkb | thanks | 20:21 |
mtreinish | honestly, I think the stability of all the jobs are pretty similar, I'd look at jogo's gate graphs to see if anything is a bit more stable than the others | 20:22 |
clarkb | mtreinish: I think I will just add a new job then. Because it looks like we fail a lot/most of the experimental jobs | 20:24 |
mtreinish | clarkb: https://review.openstack.org/#/c/107685/ needs to run the non-isolated experimental jobs which you and jeblair just pushed through | 20:25 |
mtreinish | clarkb: yeah that's really not too surprising considering how often they get looked at | 20:25 |
*** weshay has quit IRC | 20:25 | |
andreaf | mtreinish: before I rebase that to make it testable, I'd like to see https://review.openstack.org/#/c/119060 merged - it's +2 from dkranz already, and it's co-authored by you and me so I need someone else to review is. sdague? cyeoh? | 20:29 |
*** mriedem has quit IRC | 20:29 | |
dkranz | andreaf: IMO, if two core reviewers co-author something only one other +2 should be required. So I don't see a problem with a +A from mtreinish | 20:32 |
mtreinish | andreaf: I thought that was the patch I stole the change-id from you, but none of the code was reused? I think you can give it the +A :) | 20:32 |
andreaf | mtreinish, dkranz : ok will do | 20:33 |
dkranz | andreaf: Did you see my question from earlier about the resource patch sets? Are we full-speed ahead on reviewing/merging those? | 20:33 |
andreaf | dkranz: I missed your question | 20:34 |
dkranz | andreaf: Just the one I re-asked ^^^ | 20:34 |
andreaf | dkranz: so I think we could just try and merge them quickly - they are all the same, and all passing check | 20:35 |
dkranz | andreaf: I saw briefly your comments with Sean about maybe not doing this | 20:35 |
dkranz | andreaf: ok. | 20:35 |
andreaf | dkranz: the problem was all the load I caused by rebasing the whole patchset too may times :( | 20:36 |
andreaf | dkranz: the gate is quite busy so I don't know if we need to dose them a few per day only | 20:37 |
dkranz | andreaf: ok | 20:37 |
andreaf | dkranz, the nova v3 tests are broken already (since they have been removed from gate), so they will fail on the compute v3 change, but not because of the change | 20:38 |
andreaf | dkranz, in fact I included the nova v3 tests mostly for completeness and so I could enforce the hacking rule | 20:39 |
dkranz | andreaf: I saw that. ok | 20:39 |
*** skata has joined #openstack-qa | 20:40 | |
andreaf | dkranz: if you have some more time for reviews, https://review.openstack.org/#/c/117673/ (scenario migration) would benefit from a review as well | 20:42 |
*** proverma has quit IRC | 20:45 | |
*** markmcclain has joined #openstack-qa | 20:50 | |
dkranz | andreaf, adam_g : about https://review.openstack.org/#/c/117673/, where is the run showing this test actually passed after the change? | 20:51 |
dkranz | It looks ok to me but I like to see it passing before +A | 20:52 |
adam_g | dkranz, the scenario test @ http://logs.openstack.org/73/117673/7/check/check-tempest-dsvm-ironic-pxe_ssh-nv/e90aebe/logs/testr_results.html.gz (which was failed run, but due to other issues) | 20:52 |
mtreinish | adam_g: I thought those jobs were killing the regex: http://logs.openstack.org/73/117673/7/check/check-tempest-dsvm-ironic-pxe_ssh-nv/e90aebe/console.html.gz#_2014-09-19_08_29_14_630 | 20:53 |
mtreinish | or are we not quite there yet? | 20:53 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Handle @s in log file names by renaming them https://review.openstack.org/122885 | 20:54 |
adam_g | mtreinish, after everything here https://review.openstack.org/#/q/topic:ironic_infra,n,z is merged, we can kill the regex | 20:54 |
*** Ch00k has quit IRC | 20:54 | |
adam_g | mtreinish, final patch there was just approved earlier | 20:54 |
mtreinish | adam_g: ok cool | 20:55 |
adam_g | dkranz, FYI those ironic run should pass now /w https://review.openstack.org/#/c/122832/ merged, if you want to recheck before +A | 20:55 |
*** vigneshvar has quit IRC | 20:56 | |
*** proverma has joined #openstack-qa | 20:57 | |
*** proverma has quit IRC | 20:58 | |
dkranz | adam_g: ok, so those two ironic jobs are supposed to be working. I will try to remember :) | 21:00 |
*** proverma has joined #openstack-qa | 21:01 | |
*** boden has joined #openstack-qa | 21:12 | |
*** flaper87 is now known as flaper87|afk | 21:13 | |
fungi | anybody happen to recognize whether http://logs.openstack.org/12/123112/1/check/check-tempest-dsvm-neutron-full/cdb7110/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-09-22_14_24_22_574 is benign noise from oslo.messaging leaking into n-cpu screen logs? | 21:15 |
*** edwa6643 has quit IRC | 21:15 | |
*** boden has quit IRC | 21:16 | |
jogo | so stable/icehouse is broken | 21:17 |
jogo | so thats fun | 21:17 |
fungi | always a blast, yes | 21:19 |
mtreinish | jogo, fungi: is it just the keystoneclient oslo.config thing? or is there another aspect to it? | 21:19 |
dkranz | mtreinish: Can you review https://review.openstack.org/#/c/112581/ (cleanup resources after running tempest)? I am not sure about the doc stuff there. It doesn't look quite right but I am not really familiar with that magic. | 21:20 |
mtreinish | dkranz: oh, yeah sry I forgot about that bp last week. Yeah, the docs aren't right he was closer on the first rev, before he just needed to add an automodule stanza in a new file and put that in the index | 21:21 |
mtreinish | I'll leave a comment about that | 21:22 |
openstackgerrit | Aaron Rosen proposed a change to openstack/tempest: DON"T REVIEW https://review.openstack.org/123251 | 21:23 |
jogo | mtreinish: actually don't know I didn't click on any links | 21:25 |
jogo | I thought someone else already did | 21:25 |
jogo | guess not | 21:25 |
*** gordc has left #openstack-qa | 21:26 | |
mtreinish | jogo: there was a thread about some of the failures on stable maint and I think it was just that issue oslo.config issue which has a fix in progress I think | 21:26 |
mtreinish | morganfainberg: ^^^ right? | 21:26 |
fungi | aha, here's what i was looking for... http://logs.openstack.org/12/123112/1/check/check-tempest-dsvm-neutron-full/cdb7110/logs/screen-n-api.txt.gz?level=TRACE#_2014-09-22_14_16_48_360 | 21:26 |
fungi | nova.api.openstack ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='127.0.0.1', port=9696): Read timed out. (read timeout=30) | 21:26 |
*** mmedvede has quit IRC | 21:28 | |
jogo | mtreinish: yup http://logs.openstack.org/81/119081/6/check/check-grenade-dsvm-icehouse/582a26f/logs/old/screen-g-api.txt.gz | 21:28 |
jogo | mtreinish: is at least part of it | 21:28 |
*** sabeen1 has joined #openstack-qa | 21:32 | |
*** dkranz has quit IRC | 21:33 | |
fungi | i can't seem to find a bug corresponding to that failure i ended up with... when in doubt assume a nova->neutron interaction failing is a neutron bug, or open against both projects? | 21:33 |
*** sabeen1 has quit IRC | 21:34 | |
*** sabeen1 has joined #openstack-qa | 21:35 | |
fungi | nova's neutron driver is asking for a ports list via neutronclient, and the connection to neutron times out | 21:35 |
andreaf | stevebaker, masayukig: ping - regarding the autoscaling test - https://review.openstack.org/#/c/122320/ | 21:35 |
fungi | so seems more like neutron | 21:35 |
*** toure has quit IRC | 21:36 | |
clarkb | fungi: sounds like neutron to me | 21:36 |
fungi | i'll take it deeper down the rabbit hole into neutron logs then | 21:37 |
jogo | fungi: it does sound like neutron but add nova as well | 21:37 |
jogo | just to be safe | 21:37 |
fungi | jogo: sure thing--thanks | 21:38 |
jogo | fungi: arosen is one of my go to people for nova/neutron issues | 21:38 |
jogo | arosen: ^ | 21:38 |
fungi | it's bubbling up in tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic so likely a concurrency-related something or other | 21:38 |
mtreinish | fungi: maybe not, that one spins up 2 tenants with servers in each and plays some sec group rule games to selectively block ping and ssh and test it | 21:41 |
fungi | oh, hrm | 21:41 |
fungi | well, maddeningly i don't see any errors in the q-svc screen log corresponding to that api call | 21:42 |
mtreinish | it should be relatively well isolated, but there are some issues with those test which have a couple of fixes pending (I think they're probably unrelated, mostly around relying on icmp traffic not working everywhere) | 21:42 |
mtreinish | fungi: yeah, that's always how this game goes :) | 21:43 |
openstackgerrit | Victoria Martínez de la Cruz proposed a change to openstack/tempest: Updates Tempest to match Zaqar's API name change https://review.openstack.org/123252 | 21:44 |
openstackgerrit | Victoria Martínez de la Cruz proposed a change to openstack/tempest: Updates Tempest to match Zaqar's API name change https://review.openstack.org/123252 | 21:45 |
marun | mtreinish: Have you seen this? http://www.amazon.com/Google-Tests-Software-James-Whittaker/dp/0321803027 | 21:46 |
marun | mtreinish: I'm a few chapters in and it has some pretty advice on how to approach testing that would seem pretty relevant to openstack. | 21:47 |
marun | pretty -> pretty sane | 21:47 |
marun | jaypipes, jogo ^^ | 21:47 |
mtreinish | marun: hmm, the title sounds familiar, but I haven't read it. Thanks for the suggestion, I'll take a look at it too. | 21:48 |
*** aysyanne has quit IRC | 21:48 | |
marun | mtreinish: An interesting tidbit is how google classifies tests into small/medium/large rather than using overloaded terms like unit/functional/etc | 21:48 |
*** harlowja_ has quit IRC | 21:49 | |
marun | mtreinish: On one hand it seems silly to have to create new terms, but without that there seems to be a lot of confusion around the different types of testing based on preconceived notions of what a classification like 'functional' or 'integrated' means. | 21:49 |
*** harlowja has joined #openstack-qa | 21:49 | |
*** dustins has quit IRC | 21:50 | |
jaypipes | marun: bring it to Paris :) I'll definitely steal it ;) | 21:51 |
mtreinish | marun: yeah, I can see how having less overloaded terms could help in explaining things and make the classification easier | 21:52 |
fungi | should i expect a nova driver request id passed via neutronclient to be reflected in the q-svc debug log at least? if it's not there, does that mean the neutron api server never received the request? | 21:53 |
jogo | The other is that the book is highly romanticized. It almost reads like a romance, and SETs are the heros. | 21:53 |
jogo | marun: when your done with it can I borrow it? | 21:53 |
mtreinish | fungi: nope, there's a bp for that, but I don't know what the status is | 21:54 |
mtreinish | fungi: https://blueprints.launchpad.net/nova/+spec/log-request-id-mappings | 21:54 |
clarkb | mtreinish: huh that seems like a bug not a feature | 21:54 |
marun | jogo: I've heard that comment, too. Then again, openstack infra is pretty close to the 'productivity engineering' group that SETs are a part of, and romanticizing it doesn't seem like such a bad idea if it allows an increase in available resources. | 21:55 |
clarkb | fungi: are they trying to say we should be more romantic? | 21:55 |
jogo | marun: that was a quote from a review | 21:56 |
marun | jogo: I think we could do worse than add folks with an SET focus to infra | 21:56 |
mtreinish | clarkb: sigh yeah, but it's a larger effort because it requires client changes everywhere and logging mappings between request-ids. The scope was a bit larger than a bug. | 21:56 |
*** dims_ has joined #openstack-qa | 21:56 | |
jogo | marun: yeah | 21:56 |
jogo | marun: your back in town this week right? | 21:56 |
marun | jogo: yes | 21:56 |
marun | jogo: as to borrowing it, I'm afraid I read on kindle. I'll check if it's loanable if reading on a screen doesn't dissuade you. | 21:57 |
*** dims_ has quit IRC | 21:58 | |
clarkb | jogo: also, your local library probably does ebook loaning. Its pretty awesome | 21:58 |
marun | +1 to that | 21:58 |
*** dims_ has joined #openstack-qa | 21:58 | |
jogo | hmm I don | 21:58 |
jogo | 't have a kindle | 21:58 |
* marun needs to get a library card | 21:58 | |
marun | jogo: most devices have a reader, including computers | 21:59 |
jogo | marun: lunch this week? | 21:59 |
mtreinish | jogo: there's an android kindle thing you can use | 21:59 |
marun | jogo: i don't have a kindle either | 21:59 |
jogo | mtreinish: oh nice | 21:59 |
*** rodrigods_ has joined #openstack-qa | 21:59 | |
*** dims has quit IRC | 21:59 | |
*** edwa6643 has joined #openstack-qa | 21:59 | |
marun | jogo: sure, what day works for you? | 22:00 |
*** jpmontez has quit IRC | 22:00 | |
*** proverma has quit IRC | 22:01 | |
jogo | marun: How does Thursday sound? | 22:02 |
marun | jaypipes: I'm thinking a cross-project session around in-project testing practice would be really useful. Maybe on the friday? | 22:02 |
openstackgerrit | Andrea Frittoli proposed a change to openstack/tempest: Drop autoscaling scenario test. https://review.openstack.org/123260 | 22:02 |
marun | jogo: Thursday works. When/where? I'm in the city this time. | 22:03 |
andreaf | stevebaker, masayukig : https://review.openstack.org/#/c/123260/ | 22:03 |
jogo | marun: 11:30 somewhere in the city? we can pick where late in the week | 22:03 |
*** cnesa14 has joined #openstack-qa | 22:04 | |
marun | jaypipes: I'm working on ensuring that we talk about testing either in the neutron track or on friday, but maybe a wider focus would be better. The challenge is in devising a strategy for delivery that is interactive rather than transmissive. | 22:04 |
marun | jogo: wfm, I'll put it on my calendar. | 22:04 |
fungi | clarkb: i don't have a romantic bone in my body | 22:05 |
jogo | marun: great | 22:06 |
marun | fungi: ...yet ;) | 22:06 |
clarkb | fungi: just convert your funny bone | 22:06 |
fungi | that'd be one helluva retrofit | 22:06 |
*** boden has joined #openstack-qa | 22:13 | |
*** mmedvede has joined #openstack-qa | 22:17 | |
*** boden has quit IRC | 22:17 | |
jogo | mtreinish: tempest doesn't make sure instances get deleted fully does it? | 22:22 |
*** rodrigods_ has quit IRC | 22:29 | |
*** proverma has joined #openstack-qa | 22:30 | |
mtreinish | jogo: it depends, in most cases it waits for deletion by looping a get until it gets a 404 | 22:34 |
mtreinish | there might be some places where it doesn't do that | 22:34 |
*** jgrimm is now known as zz_jgrimm | 22:35 | |
andreaf | mtreinish: it seems that a few tests are failing in the neutron non-isolated job, I doubt it's related to test-accounts, I'll have a look tomorrow: http://logs.openstack.org/60/119060/6/experimental/check-tempest-dsvm-neutron-full-non-isolated/25600e7/console.html#_2014-09-22_22_25_15_524 | 22:35 |
mtreinish | jogo: it does that for most async resource deletions. That's actually been a periodic issue with cinder in that some volumes take longer than the timeout to disappear | 22:35 |
mtreinish | andreaf: yeah the nightlies for that job have beeen failing for a while. I haven't had a chance to dig into it | 22:36 |
mtreinish | andreaf: although that patch doesn't actually use the new backwards compat cred provider yet does it? It just adds the mechanism for it | 22:37 |
*** jaypipes has quit IRC | 22:41 | |
*** jecarey has quit IRC | 22:41 | |
*** marun has quit IRC | 22:46 | |
*** marun has joined #openstack-qa | 22:48 | |
*** rbak has quit IRC | 22:56 | |
*** proverma has quit IRC | 23:01 | |
*** bknudson has quit IRC | 23:02 | |
marun | mtreinish: Here's the relevant tott post about google's test classifications: http://googletesting.blogspot.com/2010/12/test-sizes.html | 23:03 |
marun | jogo: ^^ | 23:03 |
*** cnesa14 has quit IRC | 23:03 | |
*** acfleury has quit IRC | 23:04 | |
mtreinish | marun: hmm, by that chart tempest in the gate is like medium. | 23:04 |
mtreinish | I would have assumed it was large | 23:04 |
*** dimtruck is now known as zz_dimtruck | 23:05 | |
*** KurtMartin has joined #openstack-qa | 23:05 | |
*** KurtMartin is now known as kumartin | 23:06 | |
marun | mtreinish: I don't think the chart itself is enough... | 23:09 |
*** jpich has joined #openstack-qa | 23:10 | |
mtreinish | marun: haha, fair enough. It was just colorful so it grabbed my attention | 23:10 |
marun | mtreinish: Tempest is an outlier in their classification, though. | 23:11 |
mtreinish | marun: yeah I guess it would be | 23:13 |
openstackgerrit | Aaron Rosen proposed a change to openstack/tempest: Replace wait with communicate to avoid potential deadlock https://review.openstack.org/123280 | 23:13 |
marun | mtreinish: I think unit ~= small and functional ~= medium for in-project tests. I don't think any project but swift has in-tree tests that could be considered large. | 23:13 |
*** boden has joined #openstack-qa | 23:13 | |
marun | mtreinish: I think tempest scenario tests could properly be considered large, being end-to-end, so long as the suggestions for execution time limits are ignored. | 23:14 |
marun | 900+ second limits would be terrible! | 23:15 |
*** sballe has joined #openstack-qa | 23:15 | |
*** sballe has quit IRC | 23:15 | |
notmyname | marun: large == extensive coverage, lots of lines? | 23:16 |
mtreinish | marun: heh, yeah our current limit when we kill an individual test is 500 sec. Although it's more likely to fail on an internal timeout before that | 23:16 |
openstackgerrit | Aaron Rosen proposed a change to openstack/tempest: Replace wait with communicate to avoid potential deadlock https://review.openstack.org/123280 | 23:17 |
mtreinish | so adding another 400sec would make everyone very upset | 23:17 |
marun | notmyname: I think that's part of it anyway. | 23:18 |
notmyname | marun: oh. I see the link now | 23:18 |
*** boden has quit IRC | 23:18 | |
marun | notmyname: iirc the swift functional tests are pretty involved in a way that most openstack projects don't attempt to test. | 23:18 |
notmyname | marun: swift functional tests are a client that can be run against any arbitrary swift endpoint (including public cloud providers) | 23:19 |
notmyname | marun: but we also have "probe tests" which are like cluster integration tests (white box testing. eg write 3 replicas, kill one, make sure replication puts it back). unfortunately, those aren't yet run in the gate | 23:19 |
notmyname | and, fwiw, on my saio, functests: "Ran 307 tests in 125.440s". unittests: "Ran 3155 tests in 106.491s". probetests: "Ran 19 tests in 457.813s" | 23:21 |
marun | notmyname: We're thinking of how to simulating multi-node with fault injection for Neutron with containers... Not sure how that would work with Swift. | 23:21 |
marun | notmyname: It's good to see a healthy split between different types of testing. | 23:22 |
marun | notmyname: I'm trying to move Neutron in that direction - we try to do too much in unit tests to the point that nobody really understands that a unit test is supposed to have subsecond execution. :/ | 23:22 |
notmyname | cool | 23:23 |
marun | notmyname: Why don't the probe tests run in the gate? | 23:23 |
marun | notmyname: Is there a technical limitation? | 23:23 |
notmyname | I'm proud of our test coverage in swift, but we could certainly be better about some of the unit testing. there are parts that really (_really_) stretch the definition of unit tests | 23:23 |
marun | mtreinish: 500s! Although, the way things run in the gate resource starvation is a significant factor in execution time. | 23:25 |
notmyname | marun: ya, the probe tests require the swift all-in-one setup (ie number of processes, filesystem mountpoints, presence of certain scripts). and the gate is (in general) "run it in devstack". so nobody has taken the time to either rewrite the probetests or rewrite devstack-swift or update the gate to run a SAIO | 23:26 |
openstackgerrit | David Lyle proposed a change to openstack/tempest: Restoring dashboard tests https://review.openstack.org/123281 | 23:26 |
marun | notmyname: ah, right. | 23:26 |
marun | notmyname: I've started thinking of devstack as being an impediment to new kinds of testing. | 23:27 |
mtreinish | marun: honestly I haven't see us hit a 500sec per test timeout in well over a year. It's more likely we'll timeout on the 196sec we allocate for build timeouts or ssh timeouts. | 23:27 |
marun | mtreinish: ah, fair enough. | 23:27 |
marun | notmyname: I think it would be helpful if projects could define their deployment semantics using a common framework that something like devstack could consume, but that could be reused to support testing of the projects in isolation. | 23:28 |
mtreinish | marun: I also should probably fix the total time reporting from a tempest run because IIRC the number we print is the sum of all the individual tests instead of the end_time - start_time | 23:29 |
*** proverma has joined #openstack-qa | 23:29 | |
*** atiwari has quit IRC | 23:29 | |
mtreinish | right now the output on the console run X tests in Y sec makes it seem like things are really slow... | 23:29 |
notmyname | marun: I totally support testing projects in isolation. I've wanted that for a long time | 23:30 |
marun | mtreinish: Hmmm, and is the summation what appears in the jenkins results? | 23:30 |
notmyname | marun: eg there's no reason to do multiple neutron test runs with different DB backends on every swift patch. there's no way a swift patch could break that. but yet that's what happens :-( | 23:30 |
mtreinish | marun: the jenkins results are end_time - start_time, but that also includes the devstack time and other related setup | 23:31 |
marun | notmyname: doing dependency analysis on each patch is something I'm planning on pushing for. | 23:31 |
notmyname | s/neutron// | 23:31 |
marun | notmyname: I'm hoping to start small - limited the integrated jobs that run for Neutron, e.g. don't run integrated jobs if only the test tree was updated. | 23:31 |
marun | mtreinish: ah, ok. | 23:31 |
notmyname | marun: so here's a variation on your proposed "define your deployment semantics" (which to me sounds like "give us a puppet/chef script".. | 23:32 |
marun | mtreinish: +1 to providing both elapsed and total time | 23:33 |
*** yamahata has joined #openstack-qa | 23:34 | |
notmyname | marun: never mind. I tried typing out my idea, and it's a bad idea and I feel bad. | 23:34 |
notmyname | marun: I support your idea :-) | 23:34 |
marun | notmyname: heh | 23:34 |
marun | notmyname: I'm sure it was well-intentioned. | 23:35 |
marun | notmyname: deployment is always a pain point. | 23:35 |
*** kumartin is now known as kmartin_away | 23:35 | |
notmyname | marun: I just realized that my idea would still require something like your's anyway. eg "give me 4 mountpoints at A,B,C,D" | 23:36 |
notmyname | marun: isn't that what heat is supposed to solve? ;-) | 23:36 |
marun | notmyname: hah | 23:36 |
marun | notmyname: so many ways to solve similar problems, I can never keep track | 23:36 |
marun | notmyname: I do like the idea of each project owning its deployment semantics, though. | 23:37 |
notmyname | openstack projects are required to provide a heat template for their testing. the heat version used is whatever was in the last integrated release | 23:37 |
notmyname | marun: +1000 to that! | 23:37 |
marun | notmyname: I think we'll start seeing responsibility devolve away from devstack as projects take more responsibility for testing. | 23:38 |
marun | In the end, the world will be perfect. ;) | 23:38 |
marun | notmyname: I'm not familiar enough with heat to understand the implications of that. | 23:39 |
notmyname | ya, me either | 23:40 |
marun | notmyname: Would it be possible to provide a template that would deploy swift? i.e. replace puppet/chef/etc? | 23:40 |
jogo | mtreinish: re: deleting instances, so any case where an instance fails to delete should cause tempest to fail (in theory, not in reality)? | 23:41 |
*** zz_dimtruck is now known as dimtruck | 23:42 | |
jogo | marun mtreinish: https://github.com/Netflix/Hystrix | 23:42 |
notmyname | marun: I don't know if such a thing exists today. the 2 ways I've seen to easlity set up swift are https://github.com/swiftstack/vagrant-swift-all-in-one and http://serverascode.com/2014/06/12/run-swift-in-docker.html | 23:43 |
jogo | something we may want to consider, Shevek from nebula pointed it out to me a while back | 23:43 |
jogo | hmm its java | 23:50 |
jogo | it sounds very useful though: https://github.com/Netflix/Hystrix/wiki | 23:52 |
jogo | lifeless: https://github.com/Netflix/Hystrix/wiki | 23:53 |
lifeless | jogo: +1 | 23:53 |
lifeless | jogo: thats a great distributed system design pattern | 23:53 |
jogo | lifeless: so its in java | 23:54 |
openstackgerrit | Aaron Rosen proposed a change to openstack/tempest: Use processutils and remove duplicate ping code https://review.openstack.org/123285 | 23:54 |
jogo | lifeless: I think you should port it into python :) | 23:54 |
lifeless | jogo: I know :) | 23:54 |
lifeless | jogo: possibly indeed. | 23:54 |
jogo | lifeless: yeah it sounds like something that would help make OpenStack much more reliable | 23:54 |
lifeless | jogo: I was pondering a python impl of raft actually | 23:54 |
lifeless | since I've come to coclude that ampq is the source of much fragility in our design | 23:55 |
marun | jogo, lifeless: http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/ | 23:55 |
jogo | lifeless: raft? | 23:55 |
lifeless | jogo: and I'd be included to start at the bottom | 23:55 |
marun | We need to spread this as far and wide as possible :) | 23:55 |
lifeless | jogo: read marun's link. | 23:55 |
lifeless | jogo: erm actually perhaps not that one; let me see | 23:55 |
jogo | lifeless: http://raftconsensus.github.io/ ? | 23:56 |
marun | lifeless: would love to have more/better info if you have it | 23:56 |
lifeless | jogo: ah, I mean kafka | 23:56 |
jogo | http://kafka.apache.org/ ? | 23:56 |
lifeless | jogo: marun: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying | 23:56 |
lifeless | ^ that is totally wortha read | 23:56 |
clarkb | kafka has python bindings somewhere. not sure how good they are | 23:56 |
lifeless | it brought together a bunch of things I already knew | 23:56 |
marun | danke | 23:57 |
lifeless | clarkb: I meant impl, since openstack is so java averse. | 23:57 |
openstackgerrit | Miguel Lavalle proposed a change to openstack/tempest: Migrate test_load_balancer_basic to tempest client https://review.openstack.org/118232 | 23:57 |
clarkb | lifeless: I mean why should we care if you have a java service? we run an erlang service as part of this thing | 23:57 |
lifeless | marun: also 'release it' is a good book to read about operational readiness of things like openstack | 23:57 |
lifeless | clarkb: I agree. | 23:57 |
lifeless | clarkb: I am not endorsing the averseness. | 23:57 |
lifeless | clarkb: I am merely acknowledging that we're languagist en mass. | 23:58 |
*** hemna is now known as hemnass | 23:58 | |
marun | clarkb: java is painful | 23:58 |
lifeless | clarkb: TBH the JVM is a hell of a lot better operationally than CPython, due to all the things that make getting started with it in the first place hard (like fixed memory footprints) | 23:58 |
clarkb | marun: but its a thing you just run and never have to look at the guts of | 23:59 |
lifeless | clarkb: but - its a much steeper learning curve to operate for first-time folk | 23:59 |
clarkb | lifeless: btw https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907 has an assignee | 23:59 |
marun | clarkb: like gerrit? :p | 23:59 |
clarkb | marun: no like rabbit. when did you write erlang last? | 23:59 |
*** aimon has quit IRC | 23:59 | |
clarkb | or elasticsearch | 23:59 |
clarkb | mysql and postgres fall in the same bunch too though they are Cish | 23:59 |
marun | good thing I don't have problems with rabbit | 23:59 |
marun | their contribution model is terrile | 23:59 |
marun | terrible | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!