| *** rongze has quit IRC | 00:03 | |
| *** thedodd has joined #openstack-infra | 00:07 | |
| *** mgagne has quit IRC | 00:07 | |
| *** pcrews has quit IRC | 00:11 | |
| *** nati_ueno has joined #openstack-infra | 00:12 | |
| *** dcramer_ has quit IRC | 00:14 | |
| *** hogepodge has joined #openstack-infra | 00:22 | |
| *** hogepodge has quit IRC | 00:24 | |
| *** pcrews has joined #openstack-infra | 00:26 | |
| *** senk has joined #openstack-infra | 00:31 | |
| *** yamahata_ has quit IRC | 00:32 | |
| *** senk1 has joined #openstack-infra | 00:34 | |
| *** senk has quit IRC | 00:36 | |
| *** yamahata_ has joined #openstack-infra | 00:42 | |
| *** jcooley_ has joined #openstack-infra | 00:48 | |
| *** jcooley_ has quit IRC | 00:50 | |
| *** jcooley_ has joined #openstack-infra | 00:52 | |
| *** senk has joined #openstack-infra | 00:53 | |
| *** senk1 has quit IRC | 00:54 | |
| *** yamahata_ has quit IRC | 00:55 | |
| *** bpokorny has joined #openstack-infra | 00:55 | |
| bknudson | ok, I've got a single test translated from keystone to tempest scenario | 00:56 |
|---|---|---|
| morganfainberg | does anyone know if in oslo.logging it's valid to do something like log.debug('%(id)s', id=value) ? | 00:57 |
| morganfainberg | instead of using the explicit dict semantic. it looks like pythong logging supports it | 00:58 |
| * morganfainberg goes digging. | 00:58 | |
| clarkb | morganfainberg: if python logging supports it then yes | 00:58 |
| *** senk has quit IRC | 00:58 | |
| clarkb | oslo.logging is basically python logging | 00:58 |
| morganfainberg | clarkb, thought so. | 00:58 |
| morganfainberg | clarkb, i just realized how much clearer it is to use kwargs instead of explicit dict... and feel dumb for not seeing it earlier | 00:58 |
| morganfainberg | clarkb, that is if it works, let me go test it. | 00:59 |
| bknudson | https://review.openstack.org/#/c/60724/ | 00:59 |
| clarkb | bknudson: is that where the novaclient tests live? | 00:59 |
| *** sdake has quit IRC | 01:00 | |
| clarkb | bknudson: I think the novaclient tests are under tempest/cli | 01:00 |
| clarkb | but ++ to progress | 01:00 |
| morganfainberg | clarkb, bah, doesn't work in python logging *grumbles* seems like a silly thing to not "do" *at least in py27* | 01:01 |
| morganfainberg | clarkb, oh well. /me constructs dicts instead | 01:01 |
| bknudson | clarkb: I wasn't sure where to put it so just put it in scenarios since that what was mentioned | 01:01 |
| bknudson | maybe there should be a new directory for python API tests. | 01:03 |
| clarkb | bknudson: I believe that is what the cli/ dir exists for | 01:04 |
| *** nati_ueno has quit IRC | 01:04 | |
| clarkb | I could be wrong though, asking the tempest devs would be best | 01:04 |
| bknudson | cli doesn't seem like a bad place at all. | 01:04 |
| *** guohliu has joined #openstack-infra | 01:06 | |
| *** jcooley_ has quit IRC | 01:08 | |
| morganfainberg | bknudson = super awesome for tackling the testing for keystone in tempest. (just my opinion though) | 01:08 |
| bknudson | morganfainberg: so my goal is to take the tests essentially "as is" | 01:09 |
| morganfainberg | bknudson, nod. which is sooooooo much better than we have | 01:09 |
| bknudson | morganfainberg: not sure if you've thought about this... but we could instead go with starting from scratch. | 01:09 |
| morganfainberg | bknudson, since it's in tempest then. | 01:09 |
| bknudson | morganfainberg: because the keystoneclient tests in keystone are not for testing keystoneclient. | 01:09 |
| morganfainberg | bknudson, well, i think starting from scratch might not be the right approach - we have decent coverage | 01:10 |
| bknudson | they test the server... but I don't want to lose any coverage. | 01:10 |
| morganfainberg | bknudson, and doing make-work for the sake of makework is silly. | 01:10 |
| bknudson | morganfainberg: I think we're on the same page. | 01:10 |
| morganfainberg | bknudson, woo :) | 01:10 |
| bknudson | but I think what's going to happen is we get everything to tempest and then take back what's not actually testing the client. | 01:11 |
| *** jcooley_ has joined #openstack-infra | 01:11 | |
| bknudson | which might even be most of the tests... I'm not that familiar with them | 01:11 |
| bknudson | morganfainberg: dict(id=value) isn't that much worse than id=value. | 01:13 |
| clarkb | or shift the non client tests into the other parts of tempest | 01:13 |
| *** nati_ueno has joined #openstack-infra | 01:14 | |
| *** pcrews has quit IRC | 01:14 | |
| bknudson | maybe it would be better to clean up the keystoneclient tests in keystone first... separate the client tests from the unit tests. | 01:15 |
| morganfainberg | bknudson, that might be true | 01:16 |
| *** mgagne has joined #openstack-infra | 01:16 | |
| *** mgagne has joined #openstack-infra | 01:16 | |
| *** bpokorny has left #openstack-infra | 01:17 | |
| *** bpokorny1 has joined #openstack-infra | 01:19 | |
| *** paul-- has joined #openstack-infra | 01:26 | |
| *** cyeoh_ has joined #openstack-infra | 01:26 | |
| *** cyeoh_ is now known as cyeoh | 01:26 | |
| *** sdake has joined #openstack-infra | 01:28 | |
| *** bingbu has joined #openstack-infra | 01:29 | |
| *** yaguang has joined #openstack-infra | 01:31 | |
| *** guohliu has quit IRC | 01:31 | |
| *** sarob has joined #openstack-infra | 01:36 | |
| *** CaptTofu has joined #openstack-infra | 01:40 | |
| *** bpokorny1 has quit IRC | 01:40 | |
| *** thedodd has quit IRC | 01:43 | |
| *** julim has joined #openstack-infra | 01:45 | |
| *** nosnos has joined #openstack-infra | 01:48 | |
| *** nosnos has quit IRC | 01:48 | |
| *** nati_ueno has quit IRC | 01:49 | |
| *** nosnos has joined #openstack-infra | 01:49 | |
| *** loq_mac has joined #openstack-infra | 01:51 | |
| *** leifmadsen has quit IRC | 01:51 | |
| *** sarob has quit IRC | 01:52 | |
| *** julim has quit IRC | 01:55 | |
| *** jcooley_ has quit IRC | 01:55 | |
| *** yamahata_ has joined #openstack-infra | 01:57 | |
| *** senk has joined #openstack-infra | 01:59 | |
| *** dcramer_ has joined #openstack-infra | 02:00 | |
| *** jcooley_ has joined #openstack-infra | 02:03 | |
| *** senk has quit IRC | 02:03 | |
| *** bingbu has quit IRC | 02:06 | |
| *** nati_ueno has joined #openstack-infra | 02:15 | |
| *** bingbu has joined #openstack-infra | 02:19 | |
| *** rongze has joined #openstack-infra | 02:26 | |
| *** marun has joined #openstack-infra | 02:28 | |
| *** marun has quit IRC | 02:32 | |
| *** zhiyan has joined #openstack-infra | 02:45 | |
| *** jcooley_ has quit IRC | 02:56 | |
| *** guohliu has joined #openstack-infra | 02:58 | |
| *** sdake has quit IRC | 02:58 | |
| *** senk has joined #openstack-infra | 03:00 | |
| *** rcleere has joined #openstack-infra | 03:02 | |
| *** senk has quit IRC | 03:04 | |
| *** jcooley_ has joined #openstack-infra | 03:04 | |
| *** zhiyan has quit IRC | 03:04 | |
| *** guohliu has quit IRC | 03:06 | |
| *** bingbu has quit IRC | 03:14 | |
| *** nati_ueno has quit IRC | 03:15 | |
| *** mriedem has quit IRC | 03:16 | |
| *** sarob has joined #openstack-infra | 03:20 | |
| *** sarob has quit IRC | 03:24 | |
| *** bingbu has joined #openstack-infra | 03:26 | |
| *** mwagner_lap has quit IRC | 03:27 | |
| *** sdake has joined #openstack-infra | 03:27 | |
| *** zhiyan has joined #openstack-infra | 03:31 | |
| *** nati_ueno has joined #openstack-infra | 03:32 | |
| *** nati_ueno has quit IRC | 03:39 | |
| *** mwagner_lap has joined #openstack-infra | 04:09 | |
| *** mrodden has joined #openstack-infra | 04:10 | |
| *** mrodden has quit IRC | 04:10 | |
| *** Daviey has quit IRC | 04:12 | |
| *** sarob has joined #openstack-infra | 04:20 | |
| *** sarob has quit IRC | 04:24 | |
| *** yaguang has quit IRC | 04:24 | |
| *** Daviey has joined #openstack-infra | 04:25 | |
| *** paul-- has quit IRC | 04:26 | |
| *** paul-- has joined #openstack-infra | 04:26 | |
| *** nosnos_ has joined #openstack-infra | 04:28 | |
| *** chandankumar has joined #openstack-infra | 04:28 | |
| *** zehicle has joined #openstack-infra | 04:28 | |
| *** chandankumar has quit IRC | 04:29 | |
| *** zehicle_at_dell has quit IRC | 04:31 | |
| *** nosnos has quit IRC | 04:31 | |
| *** chandankumar has joined #openstack-infra | 04:35 | |
| *** AaronG_Zzz is now known as AaronGr | 04:38 | |
| *** AaronGr is now known as AaronG_Zzz | 04:42 | |
| *** paul-- has quit IRC | 04:49 | |
| *** dcramer_ has quit IRC | 04:52 | |
| *** sdake has quit IRC | 04:53 | |
| *** rongze has quit IRC | 04:54 | |
| *** boris-42 has joined #openstack-infra | 05:03 | |
| *** rcleere has quit IRC | 05:09 | |
| *** mrodden has joined #openstack-infra | 05:11 | |
| *** yaguang has joined #openstack-infra | 05:12 | |
| *** UtahDave has joined #openstack-infra | 05:19 | |
| *** nicedice_ has quit IRC | 05:21 | |
| *** sarob has joined #openstack-infra | 05:25 | |
| *** yamahata_ has quit IRC | 05:25 | |
| *** senk has joined #openstack-infra | 05:32 | |
| *** senk has quit IRC | 05:36 | |
| *** rongze has joined #openstack-infra | 05:37 | |
| *** talluri has joined #openstack-infra | 05:40 | |
| *** rongze has quit IRC | 05:42 | |
| *** talluri has quit IRC | 05:42 | |
| *** talluri has joined #openstack-infra | 05:42 | |
| *** tian has quit IRC | 05:43 | |
| *** elasticio has joined #openstack-infra | 05:44 | |
| *** senk has joined #openstack-infra | 05:44 | |
| *** senk has quit IRC | 05:49 | |
| *** elasticio has quit IRC | 05:51 | |
| *** elasticio has joined #openstack-infra | 05:51 | |
| *** marun has joined #openstack-infra | 05:53 | |
| *** nosnos_ has quit IRC | 05:57 | |
| *** nosnos has joined #openstack-infra | 05:58 | |
| *** elasticio has quit IRC | 06:06 | |
| *** mgagne has quit IRC | 06:14 | |
| *** sarob has quit IRC | 06:14 | |
| *** Daviey has quit IRC | 06:33 | |
| *** senk has joined #openstack-infra | 06:38 | |
| *** rongze has joined #openstack-infra | 06:39 | |
| *** senk1 has joined #openstack-infra | 06:40 | |
| *** Daviey has joined #openstack-infra | 06:41 | |
| *** zhiyan has quit IRC | 06:41 | |
| *** senk1 has quit IRC | 06:42 | |
| *** senk has quit IRC | 06:42 | |
| *** rongze has quit IRC | 06:45 | |
| *** sarob has joined #openstack-infra | 06:45 | |
| *** senk has joined #openstack-infra | 06:46 | |
| *** pblaho has joined #openstack-infra | 06:49 | |
| *** senk has quit IRC | 06:50 | |
| *** sarob has quit IRC | 06:50 | |
| *** senk has joined #openstack-infra | 06:51 | |
| *** tian has joined #openstack-infra | 06:54 | |
| *** rongze has joined #openstack-infra | 06:59 | |
| *** nosnos has quit IRC | 07:01 | |
| *** nosnos has joined #openstack-infra | 07:01 | |
| *** thomasbiege has joined #openstack-infra | 07:02 | |
| *** jcooley_ has quit IRC | 07:18 | |
| *** sarob has joined #openstack-infra | 07:20 | |
| *** sarob has quit IRC | 07:25 | |
| *** Ryan_Lane has quit IRC | 07:26 | |
| *** tian has quit IRC | 07:28 | |
| *** jcooley_ has joined #openstack-infra | 07:29 | |
| *** davidhadas has quit IRC | 07:32 | |
| *** yolanda has joined #openstack-infra | 07:34 | |
| *** zhiyan has joined #openstack-infra | 07:39 | |
| *** tian has joined #openstack-infra | 07:39 | |
| *** jcooley_ has quit IRC | 07:41 | |
| *** jcooley_ has joined #openstack-infra | 07:42 | |
| *** ljjjustin has joined #openstack-infra | 07:46 | |
| *** jcooley_ has quit IRC | 07:46 | |
| *** thomasbiege has quit IRC | 07:50 | |
| *** flaper87|afk is now known as flaper87 | 07:56 | |
| *** Ryan_Lane has joined #openstack-infra | 08:03 | |
| *** afazekas has quit IRC | 08:03 | |
| *** senk has quit IRC | 08:05 | |
| *** senk has joined #openstack-infra | 08:07 | |
| *** jcoufal has joined #openstack-infra | 08:07 | |
| *** Ryan_Lane has quit IRC | 08:08 | |
| *** pvo has quit IRC | 08:08 | |
| *** Guest98834 has quit IRC | 08:08 | |
| *** hughsaunders has quit IRC | 08:09 | |
| *** ljjjustin has quit IRC | 08:10 | |
| *** hashar has joined #openstack-infra | 08:14 | |
| *** Ryan_Lane has joined #openstack-infra | 08:14 | |
| *** senk has quit IRC | 08:15 | |
| *** senk has joined #openstack-infra | 08:16 | |
| *** Ryan_Lane has quit IRC | 08:19 | |
| *** marun has quit IRC | 08:19 | |
| *** fbo_away is now known as fbo | 08:19 | |
| *** sarob has joined #openstack-infra | 08:20 | |
| *** hughsaunders has joined #openstack-infra | 08:24 | |
| *** sarob has quit IRC | 08:25 | |
| *** bingbu has quit IRC | 08:25 | |
| *** nprivalova has joined #openstack-infra | 08:32 | |
| *** marun has joined #openstack-infra | 08:35 | |
| *** yongli is now known as yongli_away | 08:41 | |
| *** afazekas has joined #openstack-infra | 08:43 | |
| *** nati_ueno has joined #openstack-infra | 08:43 | |
| *** thomasbiege has joined #openstack-infra | 08:50 | |
| *** senk has quit IRC | 08:54 | |
| *** jpich has joined #openstack-infra | 09:05 | |
| *** UtahDave has quit IRC | 09:05 | |
| *** plomakin has quit IRC | 09:06 | |
| *** yaguang has quit IRC | 09:07 | |
| *** yassine has joined #openstack-infra | 09:11 | |
| *** vkozhukalov has joined #openstack-infra | 09:12 | |
| *** senk has joined #openstack-infra | 09:19 | |
| *** sarob has joined #openstack-infra | 09:20 | |
| *** nati_ueno has quit IRC | 09:22 | |
| *** derekh has joined #openstack-infra | 09:25 | |
| *** sarob has quit IRC | 09:25 | |
| *** hughsaunders has quit IRC | 09:28 | |
| *** davidhadas has joined #openstack-infra | 09:28 | |
| *** thomasbiege has quit IRC | 09:31 | |
| *** hughsaunders has joined #openstack-infra | 09:32 | |
| *** johnthetubaguy has joined #openstack-infra | 09:35 | |
| *** hughsaunders has quit IRC | 09:52 | |
| *** dmakogon_ is now known as denis_makogon | 09:54 | |
| *** yaguang has joined #openstack-infra | 09:54 | |
| *** davidhadas has quit IRC | 09:55 | |
| *** senk has quit IRC | 10:01 | |
| *** hughsaunders has joined #openstack-infra | 10:02 | |
| *** ljjjustin has joined #openstack-infra | 10:06 | |
| *** nosnos has quit IRC | 10:07 | |
| *** nosnos has joined #openstack-infra | 10:07 | |
| *** aardvark has joined #openstack-infra | 10:10 | |
| *** WarrenUsui has quit IRC | 10:12 | |
| *** johnthetubaguy has quit IRC | 10:15 | |
| *** johnthetubaguy has joined #openstack-infra | 10:15 | |
| *** sarob has joined #openstack-infra | 10:20 | |
| *** ljjjustin has quit IRC | 10:21 | |
| *** sarob has quit IRC | 10:21 | |
| *** sarob has joined #openstack-infra | 10:22 | |
| *** pblaho has quit IRC | 10:23 | |
| *** zhiyan has quit IRC | 10:23 | |
| *** sarob has quit IRC | 10:26 | |
| *** zhiyan has joined #openstack-infra | 10:27 | |
| *** paul-- has joined #openstack-infra | 10:33 | |
| *** johnthetubaguy1 has joined #openstack-infra | 10:33 | |
| *** johnthetubaguy has quit IRC | 10:36 | |
| *** nprivalova has quit IRC | 10:40 | |
| *** jorisroovers has joined #openstack-infra | 10:45 | |
| *** zhiyan has quit IRC | 10:46 | |
| *** yamahata_ has joined #openstack-infra | 10:49 | |
| *** jorisroovers has quit IRC | 10:53 | |
| *** jorisroovers has joined #openstack-infra | 10:53 | |
| *** nprivalova has joined #openstack-infra | 10:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:53 | |
| *** paul-- has quit IRC | 10:54 | |
| *** ilyashakhat has joined #openstack-infra | 10:55 | |
| *** fbo is now known as fbo_away | 11:06 | |
| *** zhiyan has joined #openstack-infra | 11:13 | |
| *** zhiyan has quit IRC | 11:14 | |
| *** yamahata_ has quit IRC | 11:14 | |
| *** rongze has quit IRC | 11:17 | |
| *** lillie has joined #openstack-infra | 11:19 | |
| *** lillie is now known as Guest33069 | 11:19 | |
| *** pvo has joined #openstack-infra | 11:19 | |
| *** sarob has joined #openstack-infra | 11:20 | |
| *** sarob has quit IRC | 11:25 | |
| *** rongze has joined #openstack-infra | 11:26 | |
| *** markmc has joined #openstack-infra | 11:26 | |
| vkozhukalov | How can I login into devstack-gate VM via ssh? Is it possible at all? I need my hook script to be able to run some custom commands inside VM. | 11:33 |
| *** adalbas has joined #openstack-infra | 11:34 | |
| *** amotoki has joined #openstack-infra | 11:40 | |
| *** rfolco has joined #openstack-infra | 11:42 | |
| *** talluri has quit IRC | 11:42 | |
| *** ArxCruz has joined #openstack-infra | 11:42 | |
| *** plomakin has joined #openstack-infra | 11:44 | |
| *** talluri has joined #openstack-infra | 11:45 | |
| vkozhukalov | Is it supposed that devstack-gate hook is launched inside test VM? | 11:52 |
| *** nati_ueno has joined #openstack-infra | 11:52 | |
| *** rongze has quit IRC | 11:58 | |
| *** talluri has quit IRC | 12:00 | |
| *** tma996 has joined #openstack-infra | 12:02 | |
| *** nati_ueno has quit IRC | 12:07 | |
| *** yaguang has quit IRC | 12:07 | |
| *** nati_ueno has joined #openstack-infra | 12:07 | |
| *** nati_ueno has quit IRC | 12:12 | |
| *** nprivalova has quit IRC | 12:14 | |
| *** nprivalova has joined #openstack-infra | 12:15 | |
| *** adalbas has quit IRC | 12:15 | |
| *** johnthetubaguy1 is now known as johnthetubagu | 12:16 | |
| *** johnthetubagu is now known as johnthetubaguy | 12:16 | |
| *** nati_ueno has joined #openstack-infra | 12:17 | |
| *** Abhishe__ has joined #openstack-infra | 12:20 | |
| *** sarob has joined #openstack-infra | 12:20 | |
| *** Abhishe__ has quit IRC | 12:21 | |
| *** sarob has quit IRC | 12:25 | |
| *** paul-- has joined #openstack-infra | 12:26 | |
| *** SushilKM has joined #openstack-infra | 12:29 | |
| *** sdague_ has quit IRC | 12:29 | |
| *** adalbas has joined #openstack-infra | 12:36 | |
| *** rongze has joined #openstack-infra | 12:40 | |
| *** fbo_away is now known as fbo | 12:45 | |
| vkozhukalov | Now, I see. Every test is launched already inside VM which is jenkins worker (slave). | 12:46 |
| vkozhukalov | So, I do not need ssh. | 12:46 |
| *** julim has joined #openstack-infra | 12:48 | |
| *** hashar has quit IRC | 12:51 | |
| *** xchu has joined #openstack-infra | 12:53 | |
| *** mkerrin has joined #openstack-infra | 12:54 | |
| *** flaper87 is now known as flaper87|afk | 13:06 | |
| *** guohliu has joined #openstack-infra | 13:07 | |
| *** nprivalova has quit IRC | 13:10 | |
| *** mwagner_lap has quit IRC | 13:11 | |
| *** mugsie has quit IRC | 13:11 | |
| *** nosnos has quit IRC | 13:13 | |
| *** nprivalova has joined #openstack-infra | 13:13 | |
| *** hashar has joined #openstack-infra | 13:14 | |
| *** marun has quit IRC | 13:15 | |
| *** weshay has joined #openstack-infra | 13:18 | |
| *** guohliu has quit IRC | 13:18 | |
| *** sarob has joined #openstack-infra | 13:20 | |
| *** vipuls has joined #openstack-infra | 13:23 | |
| *** che-arne has joined #openstack-infra | 13:23 | |
| *** slong has joined #openstack-infra | 13:23 | |
| *** vipul has quit IRC | 13:23 | |
| *** ekarlso- has quit IRC | 13:23 | |
| *** ekarlso has joined #openstack-infra | 13:23 | |
| *** slong_ has quit IRC | 13:23 | |
| *** paul-- has quit IRC | 13:23 | |
| *** sarob has quit IRC | 13:25 | |
| *** rongze has quit IRC | 13:25 | |
| mordred | vkozhukalov: correct | 13:26 |
| *** thomasem has joined #openstack-infra | 13:28 | |
| *** mugsie has joined #openstack-infra | 13:29 | |
| *** mugsie has quit IRC | 13:29 | |
| *** mugsie has joined #openstack-infra | 13:29 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Adding an option to use qpid instead of rabbit or zeromq https://review.openstack.org/55829 | 13:29 |
| *** senk has joined #openstack-infra | 13:31 | |
| *** sarob has joined #openstack-infra | 13:35 | |
| *** SushilKM__ has joined #openstack-infra | 13:36 | |
| *** SushilKM has quit IRC | 13:37 | |
| *** dkranz has joined #openstack-infra | 13:38 | |
| *** sarob has quit IRC | 13:40 | |
| *** rongze has joined #openstack-infra | 13:41 | |
| sdague | morning folks | 13:42 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add mpcut.sh for milestone-proposed branch cutting https://review.openstack.org/60814 | 13:44 |
| ttx | sdague: morning! | 13:44 |
| *** guohliu has joined #openstack-infra | 13:44 | |
| mordred | morning ttx, sdague | 13:46 |
| *** yaguang has joined #openstack-infra | 13:46 | |
| *** loq_mac has quit IRC | 13:46 | |
| ttx | mordred: still in Israel ? | 13:47 |
| *** flaper87|afk is now known as flaper87 | 13:48 | |
| *** jorisroovers has quit IRC | 13:48 | |
| *** blitzrage has joined #openstack-infra | 13:49 | |
| sdague | man, apparently 2 people nominated me for the board election. I waved off jog0 from doing that, so now I wonder if there is a way to figure out the culprits without accepting the nomination :) | 13:50 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 13:50 |
| sdague | That being said - dhellman only needs 4 more nominations to go to the ballot - http://www.openstack.org/community/members/profile/359 | 13:51 |
| mordred | ttx: I am - although I'm sitting in the airport | 13:51 |
| markmc | sdague, you don't see them here https://www.openstack.org/profile/election/ ? | 13:51 |
| sdague | oh, I do | 13:52 |
| sdague | markmc: thanks | 13:52 |
| markmc | np | 13:52 |
| *** dstanek has joined #openstack-infra | 13:52 | |
| *** nati_ueno has quit IRC | 13:52 | |
| *** zhiyan has joined #openstack-infra | 13:53 | |
| *** thomasbiege has joined #openstack-infra | 13:55 | |
| *** amotoki has quit IRC | 13:56 | |
| *** bpokorny has joined #openstack-infra | 13:57 | |
| * ttx won't be a candidate until the election is not cumulative | 13:58 | |
| ttx | I don't want to steal votes from anyone | 13:58 |
| ttx | well, that's not really true. let's say the people I'm likely to harm are not the ones I want to harm. | 13:59 |
| *** dprince has joined #openstack-infra | 13:59 | |
| *** zoresvit has joined #openstack-infra | 14:02 | |
| *** chandankumar_ has joined #openstack-infra | 14:03 | |
| *** zhiyan has left #openstack-infra | 14:03 | |
| *** zhiyan has joined #openstack-infra | 14:03 | |
| *** weshay has quit IRC | 14:03 | |
| *** mfer has joined #openstack-infra | 14:04 | |
| *** chandankumar__ has joined #openstack-infra | 14:04 | |
| matel | Hi, Is there a way I can reproduce a jenkins job run myself? Is there any documentation on how the test instance is created (base-image -> node in a pool -> running tests)? I would like to come up with a script that creates a XenServer node. | 14:05 |
| *** jaypipes has joined #openstack-infra | 14:05 | |
| SergeyLukjanov | sdague, hi, could you please take a look on the d-g-full job failed on my devstack patch for savanna? (https://review.openstack.org/#/c/60077/) | 14:06 |
| *** chandankumar has quit IRC | 14:06 | |
| SergeyLukjanov | sdague, it looks like just a timeout | 14:06 |
| *** xchu has quit IRC | 14:07 | |
| *** chandankumar_ has quit IRC | 14:08 | |
| *** krtaylor has joined #openstack-infra | 14:09 | |
| *** dizquierdo has joined #openstack-infra | 14:10 | |
| *** sandywalsh has quit IRC | 14:10 | |
| *** thomasbiege has quit IRC | 14:13 | |
| *** senk has quit IRC | 14:13 | |
| *** senk1 has joined #openstack-infra | 14:13 | |
| *** salv-orlando has quit IRC | 14:14 | |
| *** blitzrage is now known as leifmadsen | 14:14 | |
| *** salv-orlando has joined #openstack-infra | 14:14 | |
| *** herndon_ has joined #openstack-infra | 14:16 | |
| *** rvyalov has joined #openstack-infra | 14:16 | |
| *** weshay has joined #openstack-infra | 14:19 | |
| *** sarob has joined #openstack-infra | 14:20 | |
| *** zoresvit has quit IRC | 14:21 | |
| *** zoresvit has joined #openstack-infra | 14:21 | |
| *** zoresvit has quit IRC | 14:21 | |
| *** zoresvit has joined #openstack-infra | 14:21 | |
| *** thomasbiege has joined #openstack-infra | 14:22 | |
| *** zoresvit has quit IRC | 14:22 | |
| *** zoresvit has joined #openstack-infra | 14:22 | |
| *** sandywalsh has joined #openstack-infra | 14:24 | |
| *** mrodden has quit IRC | 14:24 | |
| *** thomasbiege has quit IRC | 14:25 | |
| *** sarob has quit IRC | 14:25 | |
| *** thomasbiege has joined #openstack-infra | 14:29 | |
| *** mriedem has joined #openstack-infra | 14:35 | |
| SergeyLukjanov | are there any guidelines about setting catalog type name? | 14:35 |
| SergeyLukjanov | with space in the name | 14:35 |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Add a check for file with only comments https://review.openstack.org/59090 | 14:35 |
| SergeyLukjanov | for example, for savanna we'd like to have a "data processing" catalog type | 14:35 |
| *** zhiyan has left #openstack-infra | 14:36 | |
| *** loq_mac has joined #openstack-infra | 14:36 | |
| *** changbl has quit IRC | 14:37 | |
| *** loq_mac has quit IRC | 14:44 | |
| *** yamahata_ has joined #openstack-infra | 14:45 | |
| mattymo | mordred, oops I called you Mordred instead of Monty( | 14:48 |
| *** jorisroovers has joined #openstack-infra | 14:50 | |
| *** beagles has joined #openstack-infra | 14:50 | |
| *** changbl has joined #openstack-infra | 14:50 | |
| *** mrodden has joined #openstack-infra | 14:54 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/release-tools: Initial setup of python project layout https://review.openstack.org/60840 | 14:55 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/release-tools: Enable flake8/hacking checks https://review.openstack.org/60841 | 14:55 |
| fungi | matel: for a devstack-gate job, you probably want the instructions at https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n100 | 14:59 |
| matel | fungi: Thanks! | 15:00 |
| mattymo | fungi, ping | 15:00 |
| *** dkliban has joined #openstack-infra | 15:01 | |
| mattymo | dkliban, hey! | 15:01 |
| dkliban | mattymo, hey! | 15:01 |
| *** alcabrera has joined #openstack-infra | 15:01 | |
| *** mika has quit IRC | 15:01 | |
| *** jkyle has quit IRC | 15:01 | |
| *** fbo has quit IRC | 15:01 | |
| *** echohead has quit IRC | 15:01 | |
| *** mtreinish has quit IRC | 15:01 | |
| *** LinuxJedi has quit IRC | 15:01 | |
| *** basic` has quit IRC | 15:01 | |
| *** soren has quit IRC | 15:01 | |
| *** fungi has quit IRC | 15:01 | |
| *** alcabrera has quit IRC | 15:01 | |
| mattymo | dkliban, hi from Moscow | 15:02 |
| *** dkranz has quit IRC | 15:02 | |
| *** rongze has quit IRC | 15:02 | |
| *** rongze has joined #openstack-infra | 15:03 | |
| *** jkyle has joined #openstack-infra | 15:03 | |
| *** echohead has joined #openstack-infra | 15:03 | |
| *** mtreinish has joined #openstack-infra | 15:04 | |
| mordred | http://mroth.github.io/lolcommits/ | 15:04 |
| *** soren has joined #openstack-infra | 15:05 | |
| *** mika has joined #openstack-infra | 15:05 | |
| *** dkliban has quit IRC | 15:06 | |
| *** ryanpetrello has joined #openstack-infra | 15:06 | |
| *** dkliban has joined #openstack-infra | 15:06 | |
| *** alcabrera has joined #openstack-infra | 15:07 | |
| *** fbo has joined #openstack-infra | 15:07 | |
| *** LinuxJedi has joined #openstack-infra | 15:07 | |
| *** fungi has joined #openstack-infra | 15:07 | |
| *** sdake has joined #openstack-infra | 15:07 | |
| *** sdake has quit IRC | 15:07 | |
| *** sdake has joined #openstack-infra | 15:07 | |
| *** LinuxJedi has quit IRC | 15:08 | |
| *** LinuxJedi has joined #openstack-infra | 15:08 | |
| fungi | netsplic central up in here | 15:08 |
| fungi | er, netsplit | 15:08 |
| mordred | fungi: http://mroth.github.io/lolcommits/ | 15:09 |
| *** loq_mac has joined #openstack-infra | 15:09 | |
| *** basic` has joined #openstack-infra | 15:09 | |
| fungi | zomg | 15:10 |
| fungi | the samples are great too | 15:10 |
| fungi | i particularly like "whitespace rage" | 15:10 |
| mattymo | fungi, I finally sent out a note about Fuel and stackforge. I would like to know if I should create a pull request myself for this, or if this is something taken care of on your side? | 15:12 |
| *** nati_ueno has joined #openstack-infra | 15:12 | |
| mattymo | also, it looks like fuel-docs are already underway with a review request | 15:13 |
| fungi | mattymo: yeah, you would just do something similar to that one. when the request gets approved, the repository listed as "upstream" gets imported into gerrit at that point in time | 15:14 |
| fungi | automatically (unless something breaks, at least) | 15:14 |
| *** dcramer_ has joined #openstack-infra | 15:14 | |
| *** zehicle_at_dell has joined #openstack-infra | 15:14 | |
| mattymo | unfortunately my colleague made this request and we weren't in such close contact these past couple days. We're not all located in one office | 15:14 |
| fungi | nobody is these days | 15:15 |
| * fungi is presently in his basement with a cat and lots of coffee | 15:15 | |
| *** dcramer_ has quit IRC | 15:15 | |
| *** zehicle has quit IRC | 15:15 | |
| mattymo | actually most of Fuel is developed in Moscow with 20 people in 2 adjoining rooms | 15:16 |
| mattymo | I can't work near my two cats. they would never let me | 15:16 |
| jeblair | lifeless: no, i use it because the python setup/dist machinery of the day frequently breaks, and it's nice to still be able to run the code i'm writing when that happens. | 15:17 |
| *** nati_ueno has quit IRC | 15:17 | |
| *** DennyZhang has joined #openstack-infra | 15:17 | |
| *** dcramer_ has joined #openstack-infra | 15:17 | |
| *** rustlebee is now known as russellb | 15:17 | |
| *** dkranz has joined #openstack-infra | 15:18 | |
| *** jgrimm has joined #openstack-infra | 15:18 | |
| anteaya | dang missed nominating dhellman, that's what happens when I sleep late | 15:23 |
| *** thomasbiege has quit IRC | 15:23 | |
| ryanpetrello | anteaya: haha, same | 15:25 |
| fungi | anteaya: he got enough nominations. what matters is that you don't miss *voting* | 15:25 |
| jeblair | i wish i could vote for him | 15:25 |
| jeblair | but i can't because of the election system | 15:25 |
| *** AaronG_Zzz is now known as AaronGr | 15:25 | |
| *** herndon_ has quit IRC | 15:25 | |
| fungi | right. one vote, which you have to apply 8 times to make it stick | 15:26 |
| jeblair | all of my votes have to go to mordred because he's the only one that actually voted to work to change the election system. | 15:26 |
| *** senk1 has quit IRC | 15:26 | |
| *** zehicle_at_dell has quit IRC | 15:26 | |
| jeblair | i would hope dhellman would vote similarly in the future, but i can't take the chance now. :( | 15:26 |
| fungi | definitely qualifies as a bootstrapping issue | 15:26 |
| *** NikitaKonovalov has joined #openstack-infra | 15:28 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/release-tools: Enable flake8/hacking checks https://review.openstack.org/60841 | 15:30 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/release-tools: Initial setup of python project layout https://review.openstack.org/60840 | 15:30 |
| *** rongze has quit IRC | 15:30 | |
| *** yamahata_ has quit IRC | 15:30 | |
| *** rongze has joined #openstack-infra | 15:30 | |
| *** damnsmith is now known as dansmith | 15:31 | |
| *** oubiwan__ has joined #openstack-infra | 15:31 | |
| *** datsun180b has joined #openstack-infra | 15:32 | |
| *** oubiwan__ is now known as oubiwann-lambda | 15:32 | |
| anteaya | ryanpetrello: :D | 15:33 |
| anteaya | fungi: very true | 15:33 |
| * anteaya warms up her voting pencil | 15:33 | |
| * anteaya feels queasy about the current voting system | 15:34 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Run tests and add the ability to release for tooz https://review.openstack.org/60504 | 15:34 |
| *** SushilKM__ has quit IRC | 15:38 | |
| *** dims has joined #openstack-infra | 15:40 | |
| *** nati_ueno has joined #openstack-infra | 15:40 | |
| *** jergerber has joined #openstack-infra | 15:41 | |
| *** senk has joined #openstack-infra | 15:41 | |
| *** lana has joined #openstack-infra | 15:43 | |
| *** senk has quit IRC | 15:44 | |
| *** lana has quit IRC | 15:45 | |
| *** lana has joined #openstack-infra | 15:45 | |
| *** loq_mac has quit IRC | 15:46 | |
| *** Ryan_Lane has joined #openstack-infra | 15:48 | |
| *** elasticio has joined #openstack-infra | 15:49 | |
| *** DennyZhang has quit IRC | 15:49 | |
| *** thomasbiege has joined #openstack-infra | 15:50 | |
| *** randallburt has joined #openstack-infra | 15:50 | |
| *** NikitaKonovalov has quit IRC | 15:51 | |
| *** guohliu has quit IRC | 15:52 | |
| *** SergeyLukjanov has quit IRC | 15:54 | |
| *** DennyZhang has joined #openstack-infra | 15:56 | |
| *** beagles has quit IRC | 15:58 | |
| *** beagles has joined #openstack-infra | 15:58 | |
| *** lana is now known as loq_mac | 16:01 | |
| *** herndon has joined #openstack-infra | 16:03 | |
| *** rcleere has joined #openstack-infra | 16:03 | |
| *** guohliu has joined #openstack-infra | 16:04 | |
| *** SushilKM has joined #openstack-infra | 16:04 | |
| *** loq_mac has quit IRC | 16:05 | |
| *** olkonami1 has joined #openstack-infra | 16:06 | |
| *** olkonami1 has left #openstack-infra | 16:06 | |
| *** galstrom has joined #openstack-infra | 16:07 | |
| *** dkranz has quit IRC | 16:08 | |
| *** mgagne has joined #openstack-infra | 16:08 | |
| *** jpeeler has quit IRC | 16:11 | |
| *** smarcet has joined #openstack-infra | 16:11 | |
| *** danger_fo_away is now known as dangers | 16:12 | |
| *** jpeeler has joined #openstack-infra | 16:12 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Refactor reconfiguration into a management event https://review.openstack.org/60623 | 16:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add a promote client command https://review.openstack.org/60644 | 16:13 |
| *** jasond has left #openstack-infra | 16:14 | |
| *** talluri has joined #openstack-infra | 16:14 | |
| *** rvyalov has quit IRC | 16:17 | |
| *** tma996 has quit IRC | 16:19 | |
| *** weshay has quit IRC | 16:20 | |
| *** nprivalova has quit IRC | 16:20 | |
| *** vkozhukalov has quit IRC | 16:20 | |
| *** dkranz has joined #openstack-infra | 16:20 | |
| zaro | mgagne: you've summoned? | 16:24 |
| mgagne | zaro: worst summon lag ever =) | 16:25 |
| *** jamesmcarthur has joined #openstack-infra | 16:26 | |
| zaro | mgagne: sorry, i was detained. | 16:26 |
| mgagne | zaro: clarkb and I were talking about using a jenkins plugin to inject env vars to devstack-gate jobs instead of relying on variables in the shell builder in a aim to reduce code duplication. There is 3+ jenkins plugins adding the ability to inject env vars. clarkb said there was some key differences between each of them but couldn't remember which. He said you could help in that aspect. | 16:27 |
| zaro | mgagne: you had a question about envinject? | 16:27 |
| mgagne | ^ | 16:27 |
| *** SergeyLukjanov has joined #openstack-infra | 16:27 | |
| zaro | mgagne: ohh yah, i remember something about that. | 16:28 |
| *** pcrews has joined #openstack-infra | 16:28 | |
| * zaro looks into memory via review | 16:28 | |
| jamesmcarthur | jeblair: fungi: Hey guys, just a ping on https://review.openstack.org/#/c/53644/ We're blocked on this server config, pending review. Is there someone that could take a look and let us know next steps? Thanks! | 16:29 |
| zaro | mgagne: well i remember coming up against this issue if you want to setup maven project -> https://issues.jenkins-ci.org/browse/JENKINS-14193 | 16:30 |
| fungi | jamesmcarthur: yep, it's at the top of my pile for today. hoping to try out your latest one here in a bit | 16:30 |
| jamesmcarthur | great news! thanks sir :) | 16:30 |
| jamesmcarthur | let smarcet or myself know if we can provide any further info | 16:31 |
| zaro | mgagne: there's was another imortant thingy.. | 16:31 |
| fungi | jamesmcarthur: i'll update the review with comments/failure logs shortly as appropriate | 16:31 |
| *** weshay has joined #openstack-infra | 16:32 | |
| *** jaustinpage has joined #openstack-infra | 16:33 | |
| *** jaustinpage has quit IRC | 16:33 | |
| mgagne | zaro: so there's limitations when used with maven projects | 16:34 |
| *** SushilKM has quit IRC | 16:34 | |
| zaro | mgagne: ohh yes, when using envinject at builder section of jenkins the inject will read a properties file before running the script. | 16:35 |
| zaro | mgagne: that is the opposite of what happens when setup in the properties section. | 16:35 |
| *** hartsocks has joined #openstack-infra | 16:35 | |
| sdague | fungi: I never realized you had a fun gravatar on that address :) | 16:36 |
| * sdague just added a thunderbird addon that pulls gravatars | 16:37 | |
| fungi | sdague: i think i snagged that years ago from the gnome mushrooms game | 16:37 |
| sdague | nice | 16:37 |
| zaro | mgagne: i think that's about it. | 16:37 |
| fungi | gnomush or something along those lines | 16:37 |
| *** mattray has joined #openstack-infra | 16:38 | |
| *** romcheg has joined #openstack-infra | 16:38 | |
| mgagne | zaro: thanks! | 16:38 |
| romcheg | Morning/evening folks! | 16:38 |
| romcheg | I have added an Ironic devstack job to devstack-jobs. Now JJB generates this job for grizzly branch. | 16:40 |
| romcheg | How can I configure it to generate this job only for master? | 16:40 |
| mgagne | zaro: do you receive email notifications on changes/gerrit events? | 16:41 |
| mattymo | fungi, I will propose a change for fuel-devops and fuel tomorrow morning and allow commentary tomorrow and you can merge tomorrow if you like | 16:42 |
| *** SushilKM has joined #openstack-infra | 16:42 | |
| mgagne | zaro: I got an email for that one and I'm not on the list of reviewers nor watching this project: https://review.openstack.org/#/c/60780/ | 16:42 |
| fungi | mattymo: worst case, if your github versions of those accumulate a couple of additional patches after the repositories get imported to gerrit, you can simply re-propose those commits for review (using something like git am to preserve the original author details) | 16:43 |
| zaro | mgagne: i did not get an email for that change. | 16:44 |
| zaro | mgagne: did you check your watch list? | 16:44 |
| mgagne | zaro: yes and the project isn't in the list. your name is in the email | 16:44 |
| *** yaguang has quit IRC | 16:44 | |
| mattymo | fungi, thanks | 16:45 |
| zaro | mgagne: wha? let me check again. | 16:45 |
| *** dstanek has quit IRC | 16:45 | |
| mgagne | zaro: http://paste.openstack.org/show/KY60ggNZcGfLAG3Ii3DV/ | 16:45 |
| *** jaustinpage has joined #openstack-infra | 16:45 | |
| *** jaustinpage has quit IRC | 16:46 | |
| *** zoresvit has quit IRC | 16:46 | |
| *** rongze has quit IRC | 16:46 | |
| *** jaustinpage has joined #openstack-infra | 16:47 | |
| zaro | mgagne: oohh actually found it but it doesn't look like an email sent by gerrit. it looks like forwarded from patch owner? | 16:48 |
| *** geekinutah has left #openstack-infra | 16:48 | |
| *** zoresvit has joined #openstack-infra | 16:48 | |
| zaro | mgagne: the title doesn't match the commit message. | 16:49 |
| mgagne | zaro: I don't believe they went to the length of adding our names to the greetings at the beginning just to forward something... | 16:50 |
| mgagne | zaro: also, there is an hidden patch set 1. | 16:50 |
| *** acabrera has joined #openstack-infra | 16:51 | |
| *** jgriffit2 has quit IRC | 16:53 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Install libtidy on slaves https://review.openstack.org/60872 | 16:53 |
| mgagne | can someone take a look at https://review.openstack.org/#/c/60140/? thanks | 16:53 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 16:54 | |
| *** alcabrera has quit IRC | 16:54 | |
| *** _SergeyLukjanov has quit IRC | 16:54 | |
| romcheg | clarkb: are you around? | 16:56 |
| zaro | mgagne: actually there's a 'Fuel Ci' reviewer there. the email address for that is 'Fuel CI <fuel-ci-bot@mirantis.com>' | 16:56 |
| *** acabrera is now known as alcabrera | 16:57 | |
| *** jcooley_ has joined #openstack-infra | 16:57 | |
| mgagne | zaro: Gerrit wouldn't know about the subscribers of the list. (and add them to the greetings) | 16:57 |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Adding chef-manager-core to stable maintainers acl https://review.openstack.org/60875 | 16:57 |
| *** gyee has joined #openstack-infra | 16:58 | |
| zaro | mgagne: i don't see that group in gerrit but somehow it made it onto the reviewers list. maybe manually added? | 16:59 |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Adding chef-manager-core to stable maintainers acl https://review.openstack.org/60875 | 16:59 |
| mgagne | zaro: can you add reviewers at submit time using git review? | 16:59 |
| zaro | mgagne: fuel-ci-bot might be doing the fowarding? | 16:59 |
| *** sarob has joined #openstack-infra | 16:59 | |
| zaro | mgagne: yes you can. | 16:59 |
| mgagne | zaro: could be it I guess | 17:00 |
| *** jcoufal has quit IRC | 17:00 | |
| zaro | mgagne: i believe you can add reviews at submit time and thru the gui. | 17:00 |
| fungi | mgagne: zaro: comparing the message and the review, i think they uploaded that change as a draft and added us as individual reviewers, then deleted us from the reviewers list and pushed a non-draft patchset #2 | 17:01 |
| mgagne | fungi: damn draft | 17:01 |
| zaro | how did 'Fuel Ci' get on reviewers list? | 17:01 |
| zaro | it's not a group in gerrit | 17:02 |
| *** SergeyLukjanov has joined #openstack-infra | 17:02 | |
| fungi | zaro: it's an account | 17:02 |
| fungi | not a group | 17:02 |
| fungi | it's the account used by their third-party test rig | 17:02 |
| *** johnthetubaguy has quit IRC | 17:02 | |
| zaro | ohh, ok. | 17:02 |
| *** dstanek has joined #openstack-infra | 17:03 | |
| *** johnthetubaguy has joined #openstack-infra | 17:03 | |
| fungi | zaro: you can see the list of third-party testing accounts at https://review.openstack.org/#/admin/groups/91,members | 17:03 |
| fungi | since it's an account, it can be added as a requested reviewer just like any other account | 17:03 |
| *** reed has joined #openstack-infra | 17:04 | |
| fungi | as to *why* the committer added that account as a reviewer, maybe it was so that their ci bot would be able to see the patch and run tests on it | 17:04 |
| mgagne | fungi: the question was: why were we added in the first place | 17:05 |
| fungi | but since you can't vote on drafts, it didn't report (probably tried and gave up after its attempt to vote was refused, rather than leave a non-voting comment) | 17:05 |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 17:05 |
| *** thomasbiege has quit IRC | 17:05 | |
| fungi | mgagne: you probably have to ask Alexandra Stepanchuk <astepanchuk@mirantis.com> | 17:06 |
| fungi | i'm going to guess it was a mistake | 17:06 |
| fungi | /misunderstanding | 17:07 |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 17:07 |
| *** zoresvit has quit IRC | 17:07 | |
| *** jgriffit1 has joined #openstack-infra | 17:08 | |
| *** ^d has joined #openstack-infra | 17:09 | |
| *** jgriffit1 is now known as jgriffith | 17:11 | |
| *** loq_mac has joined #openstack-infra | 17:12 | |
| *** lana has joined #openstack-infra | 17:13 | |
| *** yolanda has quit IRC | 17:13 | |
| *** lana has quit IRC | 17:14 | |
| *** sarob_ has joined #openstack-infra | 17:14 | |
| *** lana has joined #openstack-infra | 17:14 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: list jobs in config and on target Jenkins https://review.openstack.org/60881 | 17:15 |
| *** hashar has quit IRC | 17:15 | |
| *** yolanda has joined #openstack-infra | 17:16 | |
| *** sarob has quit IRC | 17:16 | |
| *** loq_mac has quit IRC | 17:16 | |
| jeblair | mordred: ping | 17:17 |
| *** rongze has joined #openstack-infra | 17:17 | |
| jeblair | fungi, clarkb: fyi, nodepool is a little sluggish because the gearman-based bursting is broken.... | 17:17 |
| jeblair | fungi, clarkb: putting a non-existent job on the queue (the ironic job from fri) exposed a bug in the gear lib; and even though that's since corrected, nodepool's client is still stuck in a bad state. | 17:18 |
| jeblair | fungi, clarkb: a restart will fix it though | 17:18 |
| *** sarob_ has quit IRC | 17:18 | |
| *** NikitaKonovalov has joined #openstack-infra | 17:19 | |
| jeblair | (until then, it's operating only using the replacement algorithm) | 17:19 |
| fungi | jeblair: okay, makes sense | 17:19 |
| *** jorisroovers has quit IRC | 17:19 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Restructure periodic cleanup https://review.openstack.org/60585 | 17:19 |
| *** cody-somerville has joined #openstack-infra | 17:20 | |
| *** cody-somerville has quit IRC | 17:20 | |
| *** cody-somerville has joined #openstack-infra | 17:20 | |
| fungi | also, freenode servers seem to be in a bad way. i keep getting >2-minute delays and am seeing repeated netsplits, so if i don't answer it may be because i'm blind | 17:20 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add job runtime stats https://review.openstack.org/60557 | 17:22 |
| *** jamesmcarthur has quit IRC | 17:24 | |
| *** lana has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:25 | |
| *** nati_ueno has quit IRC | 17:26 | |
| *** jcoufal has joined #openstack-infra | 17:28 | |
| *** rongze has quit IRC | 17:28 | |
| clarkb | morning | 17:28 |
| jeblair | clarkb: good morning | 17:28 |
| clarkb | romcheg: I am now | 17:28 |
| clarkb | jeblair: should we go ahead and restart nodepool? | 17:29 |
| fungi | it looks like it's got about half the pool available, so presumably an interruption in nodepool running shouldn't slow down gating too much | 17:29 |
| pleia2 | morning clarkb | 17:30 |
| jeblair | clarkb: yeah, i wanted to go ahead and merge those changes so they got deployed in the restart ^ | 17:30 |
| fungi | clarkb: are you cool with zaro's answer to your question in https://review.openstack.org/60043 ? i didn't want to merge without giving you a chance to follow up | 17:30 |
| fungi | jeblair: moving them up the priority list in that case | 17:30 |
| clarkb | jeblair: makes sense | 17:30 |
| *** NikitaKonovalov has quit IRC | 17:30 | |
| fungi | oh, nm. already merged them | 17:31 |
| clarkb | fungi: looking | 17:31 |
| clarkb | fungi: fine with me, feel free to approve | 17:32 |
| jeblair | clarkb, fungi: i will restart now. | 17:32 |
| fungi | awesome | 17:32 |
| *** jcoufal has quit IRC | 17:32 | |
| jeblair | done | 17:33 |
| *** NikitaKonovalov has joined #openstack-infra | 17:33 | |
| *** yolanda has quit IRC | 17:33 | |
| jeblair | i kind of think nodepool should have an exit handler that marks all BUILDING nodes as DELETE. | 17:33 |
| fungi | that's not a half bad idea | 17:34 |
| jeblair | it does have a graceful exit handler, but i'm always too impatient to use it. :/ | 17:34 |
| fungi | and way faster than waiting for them to finish building | 17:34 |
| *** luisg has quit IRC | 17:34 | |
| fungi | right, that | 17:34 |
| openstackgerrit | A change was merged to openstack-infra/config: upgrade review-dev from gerrit ver 2.4.x to ver 2.8 https://review.openstack.org/60043 | 17:34 |
| jeblair | ooh exciting | 17:35 |
| fungi | i'm mostly expecting that to break spectacularly, but at least it's a starting point | 17:36 |
| fungi | zaro: you'll presumably want to start watching review-dev here shortly | 17:36 |
| clarkb | mordred: https://jenkins02.openstack.org/job/check-tempest-dsvm-postgres-full/buildTimeTrend bigger region-b flavor got the test to run in just under the timeout value | 17:37 |
| *** fbo is now known as fbo_away | 17:38 | |
| jeblair | clarkb: so that's a moderate improvement but not nearly what was hoped for, right? | 17:38 |
| clarkb | jeblair: correct, improvement of about ~5 minutes but nothing dramatic | 17:38 |
| jeblair | i'm on reviews now, but when done, i'll resume work on the d-g changes for rax-performance | 17:39 |
| clarkb | jeblair: cool, any chance we can get https://review.openstack.org/#/c/48064/ rereviewed? | 17:40 |
| clarkb | jeblair: I did not approve it due to the other d-g changes and your -2 in the past (I wanted you to get a chance to rereview) | 17:40 |
| jeblair | clarkb: i'm planning on reviewing everything; do i need to bump it to the top of the list? | 17:40 |
| clarkb | jeblair: no, I don't think it is that urgent | 17:40 |
| jeblair | clarkb: oh, it is at the top of the list :) | 17:41 |
| *** jamesmcarthur has quit IRC | 17:41 | |
| zaro | fungi: watching intently. about 30 mins right? | 17:41 |
| fungi | zaro: i expect review-dev to notice it in about 3 minutes | 17:42 |
| jeblair | pleia2, anteaya: what's the puppet dashboard project status? | 17:42 |
| fungi | er, no, any time after that for the next 15 minutes | 17:42 |
| fungi | zaro: by 1800utc it should have tried to do something | 17:42 |
| *** dkranz has quit IRC | 17:43 | |
| *** jamesmcarthur has joined #openstack-infra | 17:44 | |
| *** NikitaKonovalov has quit IRC | 17:44 | |
| pleia2 | jeblair: I'll doublecheck, I think Hunner has an action item on it | 17:44 |
| *** jamesmcarthur has quit IRC | 17:45 | |
| *** yolanda has joined #openstack-infra | 17:45 | |
| *** dizquierdo has quit IRC | 17:46 | |
| *** NikitaKonovalov has joined #openstack-infra | 17:46 | |
| *** dkranz has joined #openstack-infra | 17:46 | |
| clarkb | lifeless: https://review.openstack.org/#/c/53917/ lgtm now btw | 17:47 |
| *** jamesmcarthur has joined #openstack-infra | 17:47 | |
| *** cody-somerville has quit IRC | 17:47 | |
| pleia2 | jeblair: yeah, Hunner was going to finish some apache manifests, said "I have code, but still working out the apache manifest stuff (it's an older module version that my prototype used)" | 17:47 |
| pleia2 | that was from meeting on the 26th | 17:48 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Updates to static.o.o volume management docs https://review.openstack.org/59615 | 17:48 |
| jeblair | pleia2: ok, thx. i think zaro could really use it now. :| | 17:50 |
| pleia2 | yeah | 17:50 |
| clarkb | sdague: I see in email that https://review.openstack.org/#/c/57066/ still needs doing. But that change fails tests. Anything I can do to help that get through? | 17:52 |
| *** sdake_ has joined #openstack-infra | 17:52 | |
| *** sdake_ has quit IRC | 17:52 | |
| *** sdake_ has joined #openstack-infra | 17:52 | |
| fungi | zaro: i'm able to start watching and troubleshooting it with you now as needed | 17:53 |
| *** rongze has joined #openstack-infra | 17:54 | |
| sdague | clarkb: I don't know, maurosr where do those patches stand right now? i.e. what's next to get that landed? | 17:54 |
| *** jcooley_ has quit IRC | 17:54 | |
| fungi | jeblair: clarkb: mordred: https://review.openstack.org/53644 is in a working state now, so as long as you're okay with the solutions there, we could dislodge some hold-up on tipit's (i can go ahead and stick this last test into dns if so) | 17:54 |
| clarkb | sdague: maurosr: I am happy to make that a priority on my end if CI involvement is a missing piece | 17:54 |
| fungi | er, on tipit's subsequent work | 17:54 |
| zaro | fungi: interesting, when i visit review-dev it still reports ver 2.4.4 ? | 17:55 |
| jeblair | fungi: will review and let you know | 17:55 |
| fungi | zaro: there's a number of potential reasons why. i'm logging in now to check it out | 17:55 |
| clarkb | fungi: jeblair: I reviewed it a few patches back, I don't need to rereview if you guys want to push it through | 17:56 |
| fungi | zaro: so, reason the first... puppet agent isn't running. gives me a chance to run it manually anyway and paste the errors if any | 17:57 |
| *** galstrom is now known as galstrom_zzz | 17:57 | |
| *** NikitaKonovalov has quit IRC | 17:57 | |
| *** cody-somerville has joined #openstack-infra | 17:57 | |
| *** jerryz has joined #openstack-infra | 17:57 | |
| zaro | jeblair: what can i really use now? | 17:59 |
| *** NikitaKonovalov has joined #openstack-infra | 17:59 | |
| *** nati_ueno has joined #openstack-infra | 17:59 | |
| *** weshay has quit IRC | 17:59 | |
| jeblair | zaro: the puppet dashboard so you can see the puppet output. | 17:59 |
| Hunner | pleia2: Hmm. Haven't made any progress since then :( | 17:59 |
| zaro | jeblair: ohh yea. /me checks it out. | 18:00 |
| Hunner | pleia2: Did you hear anything about the okay-ness of postgresql? | 18:00 |
| fungi | zaro: !!! Powered by Gerrit Code Review (2.8) | 18:01 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Allow encoded path separators in URLs https://review.openstack.org/60893 | 18:01 |
| *** WarrenUsui has joined #openstack-infra | 18:01 | |
| pleia2 | fungi: quick licensing question - in https://review.openstack.org/#/c/60375 I have some modified copies of files from cgit (which is gplv2), pabelanger correctly pointed out that I need some license headers, should I just do short header with link to full gplv2, or do we need to "ship" the full gpl text with config/ repo and reference that? | 18:01 |
| pleia2 | Hunner: I think we're still at "we can live with it" re: postgres | 18:01 |
| *** ArxCruz has quit IRC | 18:02 | |
| fungi | pleia2: i'll review | 18:02 |
| clarkb | fungi: zaro: woot (but I can't open any changes :( ) | 18:03 |
| fungi | "The page you requested was not found, or you do not have permission to view this page." | 18:04 |
| Hunner | pleia2: Okay, thanks | 18:04 |
| clarkb | I am willing to bet that some of _david_'s changes will help here | 18:04 |
| *** aardvark has quit IRC | 18:04 | |
| *** derekh has quit IRC | 18:05 | |
| jeblair | fungi, clarkb: what's the story with the php ppa? | 18:06 |
| *** cody-somerville has quit IRC | 18:06 | |
| zaro | clarkb: was that your own change or someone else's change? | 18:06 |
| jeblair | zaro: i can't see my own change | 18:06 |
| clarkb | zaro: on any change I tried, my own, jeblair's and merged changes | 18:07 |
| zaro | ok. let me checkout the acls to see if there's anything else that's needed. | 18:07 |
| *** NikitaKonovalov has quit IRC | 18:07 | |
| jeblair | fungi, clarkb: i'm _really_ concerned about running non-distro php. that's pretty much been a recipe for compromise for like 15 years now. | 18:08 |
| *** sdake has quit IRC | 18:08 | |
| fungi | jeblair: according to smarcet, supporting older php (5.4) wasn't possible, and needed the backport from saucy | 18:08 |
| *** ArxCruz has joined #openstack-infra | 18:08 | |
| *** galstrom_zzz is now known as galstrom | 18:08 | |
| clarkb | right precise's version is too old | 18:08 |
| fungi | presumably once trusty is out, it should be fine (it's releasing with 5.5) | 18:08 |
| *** sdake has joined #openstack-infra | 18:09 | |
| *** sdake has quit IRC | 18:09 | |
| *** sdake has joined #openstack-infra | 18:09 | |
| anteaya | jeblair zaro pleia2 I took a look at where my sodabrew dashboard was running: http://15.185.153.222/ and the workers fell over some time ago, I have one running again but it can't do any updates: Class#create_from_yaml failed with ActiveRecord::RecordInvalid: Validation failed: Host already has a report for time and kind - 0 failed attempts | 18:09 |
| *** NikitaKonovalov has joined #openstack-infra | 18:09 | |
| fungi | jeblair: so this is enough to bootstrap until lts has a usable version in a few months | 18:09 |
| anteaya | I do belive this was do to using the same cert everywhere and the dashboard not knowing the difference | 18:09 |
| anteaya | it seems to have gummed up all attempts to update in this configuration | 18:10 |
| clarkb | jog0: (and others) is https://review.openstack.org/#/c/57347/ run tempest tests twice something we want to do? | 18:10 |
| *** dprince has quit IRC | 18:10 | |
| fungi | jeblair: and it's a ppa maintained by the php package maintainer, so falls into the same category as the python3 and pypy backport ppas we use | 18:10 |
| jeblair | fungi: we don't run public services based on those | 18:10 |
| anteaya | mostly I mention it to see if we are going to hit the same obstacle with puppet-board | 18:10 |
| anteaya | Hunner: ^^ | 18:10 |
| clarkb | anteaya: I don't think puppet board has any workers | 18:10 |
| fungi | jeblair: fair point, but we do run public services based on things pulled from upstream git repositories (devil's advocate) | 18:10 |
| clarkb | anteaya: this is why it relies on puppetdb, all of the data is preprocessed for it | 18:11 |
| jeblair | fungi: right, so that we get timely security updates for them. | 18:11 |
| anteaya | clarkb: okay good | 18:11 |
| anteaya | was hoping that zaro could at least use this for the moment but it doesn't appear to be the case | 18:11 |
| anteaya | so +1 for puppet-board | 18:11 |
| *** salv-orlando has quit IRC | 18:12 | |
| pleia2 | anteaya: yeah, I think puppetboard just uses the database | 18:12 |
| Hunner | anteaya: That would be an obsticle for puppetdb and I don't know what happens. It should be noted that certname is the uniqueness constraint for nodes, so two nodes with the same certname will each submit reports for the same name, and if one fails and one succeeds there won't be a way to see which | 18:13 |
| jeblair | smarcet: why isn't php 5.3 possible? | 18:13 |
| clarkb | Hunner: which is wrong btw | 18:14 |
| Hunner | anteaya: Since there are no workers, puppetdb *should* be able to receive any number of reports for a given cert... it will just be represented as a single node | 18:14 |
| pleia2 | sdake: can you add the openstackinfra account to #heat re: https://review.openstack.org/#/c/60298/ | 18:14 |
| smarcet | jeblair: bc we used laravel 4 fwk to develop openstackid project, and its a laravel requirement | 18:15 |
| Hunner | clarkb: No other piece of information coming from the agent is as secure as the certname, which is why that's used for identification | 18:15 |
| clarkb | Hunner: I think it should be used as part of the key, but not the only bit of the key | 18:15 |
| jeblair | fungi: are you comfortable with that ppa? i admit that since it is maintained by the php mantainers, it is probably okay to use. | 18:16 |
| clarkb | regardless, 5 runs from the same certname should show up as 5 runs. not at 1 run and a bunch of confusion | 18:16 |
| jeblair | smarcet: will we be able to upgrade to 5.5 in 5-6 months? | 18:16 |
| fungi | jeblair: in this case, i'm comfortable with it mainly as a stop-gap to keep development moving. i'd like to see the production server on 14.04 lts as soon as possible though | 18:16 |
| Hunner | clarkb: It will show up as 5 runs... but only the last run submitted will be the "best run" even though the 5 runs could be from 5 nodes | 18:16 |
| *** smarcet has left #openstack-infra | 18:17 | |
| *** jaustinpage has quit IRC | 18:17 | |
| clarkb | Hunner: oh gotcha, old puppet dashboard just completely failed when it got reports from the same certname in the same chunk of time | 18:17 |
| *** smarcet has joined #openstack-infra | 18:17 | |
| smarcet | jblair: i think so, actually my dev env is php 5.5 | 18:17 |
| clarkb | mtreinish: https://review.openstack.org/#/c/58605/2 is that something we should try to get through as well? I like the idea but getting it in may be tricky | 18:18 |
| jeblair | smarcet: okay, i can go along with fungi on this one. i'm not excited about the ppa, but i think we'll be okay until 14.04. | 18:18 |
| pabelanger | FWIW: After talking to clarkb last week, I managed to get puppetboard up and running. After I understood all the moving parts it wasn't that bad. | 18:18 |
| smarcet | jeblair: roger that, and thx | 18:19 |
| jeblair | fungi: +2d | 18:19 |
| fungi | jeblair: smarcet: thanks | 18:19 |
| openstackgerrit | Max Rydahl Andersen proposed a change to openstack-infra/jenkins-job-builder: Add support for sidebar-links plugin via properties. https://review.openstack.org/60710 | 18:20 |
| *** vkozhukalov has joined #openstack-infra | 18:20 | |
| mtreinish | clarkb: yeah we should try to get it in, I just haven't had a chance to debug the failure it hit yet | 18:20 |
| mtreinish | and I felt bad no bugging it | 18:20 |
| jeblair | fungi, clarkb, mtreinish, sdague: speaking of which; should i send an announcement to the ml saying we'll remove 'reverify no bug' on, say, wednesday? | 18:22 |
| *** NikitaKonovalov has quit IRC | 18:22 | |
| fungi | zaro: if you think there might be something worth checking for in the logs, let me know. i'm also able to manage the all-projects acl through the webui, so it at least thinks i'm an admin still | 18:22 |
| clarkb | jeblair: ++ please do mention core reviwers can reapprove | 18:22 |
| *** jcooley_ has joined #openstack-infra | 18:22 | |
| sdague | jeblair: +1 | 18:23 |
| fungi | clarkb: jeblair: right. so far it's been mostly core reviewers who have voiced objections, and i think that's mitigated by their ability to just reapprove a change which failed the gate as appropriate | 18:23 |
| *** NikitaKonovalov has joined #openstack-infra | 18:23 | |
| jeblair | clarkb: so i was looking at https://review.openstack.org/#/c/53917/12 and wondered, when do you want bitrot jobs defined? | 18:23 |
| jeblair | clarkb: just when we add new globally-applicable devstack-gate jobs to the master branch of all projects? | 18:24 |
| fungi | wednesday sounds good | 18:24 |
| *** krotscheck has joined #openstack-infra | 18:24 | |
| *** talluri has quit IRC | 18:25 | |
| fungi | though later in the week might be better, as it would avoid changing things up on the stable branch maintainers trying to get 2013.2.1 out | 18:25 |
| *** randallburt has quit IRC | 18:25 | |
| fungi | but i guess a couple days heads up on the ml is fair warning nonetheless | 18:26 |
| sdague | can I convince anyone to run through these quick - https://review.openstack.org/#/q/status:open+project:openstack-infra/elastic-recheck+branch:master+topic:fix_height,n,z should provide a few good tweaks on top of the existing er page | 18:26 |
| jeblair | sdague: i'm doing reviews now; i should get to those in a bit. | 18:26 |
| sdague | cool | 18:26 |
| *** NikitaKonovalov has quit IRC | 18:27 | |
| clarkb | jeblair: initially I wanted it for everything, but that has been a losing battle. I don't think we can have that without writing all of the jobs ourselves | 18:27 |
| clarkb | jeblair: so yes, I have instead been shooting for globally gating | 18:27 |
| jeblair | clarkb: well, i mean, we can -1 changes unless they have both | 18:28 |
| sdake | pleia2 moment | 18:28 |
| clarkb | jeblair: that is an option, I don't want to impede progress because of future theoretical needs. However the whole point behind that refactor was to avoid both problems | 18:29 |
| sdake | pleia2 I got a successful response from chanserv, can you try now ? | 18:30 |
| pleia2 | sdake: perfect! thank you | 18:30 |
| *** jaustinpage has joined #openstack-infra | 18:31 | |
| *** ^demon has joined #openstack-infra | 18:31 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:31 | |
| jeblair | clarkb: so i think the most sensible test is: do we expect those jobs to need to be run on any stable branches in the future. My guess is that regardless of integration status, ironic will probably want to run those jobs on its stable branches. | 18:33 |
| jeblair | clarkb: is that your understanding? | 18:33 |
| *** NikitaKonovalov has quit IRC | 18:34 | |
| zaro | fungi: yes. any errors from the log would be helpful. could you paste somewhere? | 18:34 |
| *** jpich has quit IRC | 18:34 | |
| *** ^d has quit IRC | 18:34 | |
| clarkb | jeblair: pretty much. The goal was to make gating on future stable branches as simple as possible. So if jobs are expected to run against stable branches they should follow the template | 18:34 |
| fungi | zaro: sure, i'll go on a loghunt | 18:35 |
| *** che-arne has quit IRC | 18:36 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:36 | |
| *** rnirmal has joined #openstack-infra | 18:37 | |
| *** rongze has quit IRC | 18:37 | |
| anteaya | pleia2: just for kicks I also added openstackinfra to #openstack-opw | 18:37 |
| pleia2 | anteaya: thank you :) | 18:39 |
| *** nicedice has joined #openstack-infra | 18:40 | |
| *** Ryan_Lane has quit IRC | 18:41 | |
| *** Ryan_Lane has joined #openstack-infra | 18:42 | |
| *** Ryan_Lane has joined #openstack-infra | 18:42 | |
| jeblair | fungi, clarkb, sdague: https://etherpad.openstack.org/p/s6hBn91cZK | 18:42 |
| zaro | fungi: i noticed that all the links to cgit from review-dev do not work. are those the replication links? https://review-dev.openstack.org/#/admin/projects/ | 18:42 |
| clarkb | zaro: they are the 'gitweb' links and that is a known problem | 18:43 |
| clarkb | zaro: we need to fix how apache handles escaped slashes on cgit.o.o I think | 18:43 |
| fungi | jeblair: message text lgtm | 18:43 |
| clarkb | jeblair: lgtm | 18:44 |
| *** prad has joined #openstack-infra | 18:44 | |
| fungi | jeblair: oh, maybe "Core reviewers can still REsubmit"? | 18:45 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix the issue which raise FloatingIpPoolNotFound https://review.openstack.org/54331 | 18:45 |
| jeblair | ArxCruz: hey, i think you might have missed part of my comment on https://review.openstack.org/#/c/55829/ | 18:45 |
| jeblair | ArxCruz: the docs for the parameter need to go in devstack-vm-gate-wrap.sh | 18:45 |
| jeblair | ArxCruz: let me know if you have questions | 18:45 |
| ArxCruz | jeblair: oh... | 18:45 |
| ArxCruz | jeblair: okay | 18:46 |
| Hunner | pleia2: For what reason is infra using one cert for many nodes? Ease of use, or some catalog reason? | 18:46 |
| *** senk has joined #openstack-infra | 18:46 | |
| jeblair | ArxCruz: basically, there are a bunch of env-variable "parameters" that the script takes, and they are all defined, documented, and defaults set in the -wrap file | 18:46 |
| jeblair | ArxCruz: (even though they are mostly used in the -gate.sh script) | 18:46 |
| fungi | Hunner: historically it was for simplicity, but we've since seen that it creates other problems and are moving away from that | 18:47 |
| ArxCruz | jeblair: got it | 18:47 |
| Hunner | fungi: Ooo. Also, have you seen node_name_fact? It allows you to use one cert for simplicity, but give a fact as the "identity" of the box https://gist.github.com/ahpook/1182243 | 18:47 |
| *** ryanpetrello has quit IRC | 18:47 | |
| jesusaurus | Hunner: im pretty sure its to make it easy to manage a large pool of vms that are destroyed often | 18:47 |
| Hunner | fungi: This would avoid the issues of "one name for many boxes" but not increase the cert overhead | 18:48 |
| *** ryanpetrello has joined #openstack-infra | 18:48 | |
| jeblair | fungi: ++resubmit | 18:48 |
| jesusaurus | Hunner: that gist looks promising | 18:49 |
| Hunner | jesusaurus: It would also allow VM images to be pre-seeded with a single cert, and use any random hostname as their identity :) | 18:49 |
| fungi | clarkb: the main case for shared certs was that we didn't want to constantly create and delete them for ephemeral devstack slaves, back before we eventually disconnected them from the master entirely right? | 18:49 |
| *** senk has quit IRC | 18:50 | |
| jeblair | mordred: your attention on https://review.openstack.org/#/c/58770/ would be greatly appreciated | 18:50 |
| *** luisg has joined #openstack-infra | 18:50 | |
| *** weshay has joined #openstack-infra | 18:51 | |
| *** fbo_away is now known as fbo | 18:52 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add gate-climate-devstack-dsvm https://review.openstack.org/58702 | 18:52 |
| *** ^demon has quit IRC | 18:52 | |
| clarkb | fungi: it was before my time but simplicity for managing herd of cattle makes sense | 18:53 |
| *** ^d has joined #openstack-infra | 18:53 | |
| *** ^d has joined #openstack-infra | 18:53 | |
| jeblair | fungi: yes, that was the thought | 18:53 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Add script to post a gerrit welcome message https://review.openstack.org/58900 | 18:53 |
| fungi | jeblair: clarkb: how safe is it to make the gerrit error_log from review-dev public? i want to paste a copy for zaro to review, and don't immediately see any obvious sensitive/credential information but hard to be certain | 18:53 |
| zaro | fungi: ok. i think i see one problem. verified label has been removed in new gerrit. | 18:53 |
| *** ^d has quit IRC | 18:53 | |
| clarkb | mysql creds would bw my biggest worry | 18:54 |
| jeblair | fungi, clarkb: should be completely obsolete by moving to nodepool for all slaves | 18:54 |
| *** ^d has joined #openstack-infra | 18:54 | |
| *** ^d has joined #openstack-infra | 18:54 | |
| *** ^d has quit IRC | 18:54 | |
| jeblair | fungi: i can't _think_ of anything, but then i haven't seen 2.8's error log; i think you should scan it manually | 18:54 |
| fungi | clarkb: and github presumably | 18:54 |
| zaro | fungi: ohh wait. i see it in all-projects so should be ok. | 18:54 |
| fungi | jeblair: yeah, i am scanning it manually, just making sure i'm not missing anything obvious to look for | 18:55 |
| *** ^d has joined #openstack-infra | 18:55 | |
| jeblair | clarkb: what's the diff between {pipeline}-grenade-dsvm-neutron and gate-grenade-dsvm-neutron-stable ? | 18:56 |
| *** randallburt has joined #openstack-infra | 18:56 | |
| jeblair | clarkb: https://review.openstack.org/#/c/59039/5/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml | 18:57 |
| lifeless | jeblair: fair enough; I haven't had that experience, but - your mileage has varied :) | 18:57 |
| *** senk has joined #openstack-infra | 18:59 | |
| jeblair | clarkb: nm; i think i understand. | 18:59 |
| clarkb | jeblair: the -stable job is stable/havana to master | 18:59 |
| clarkb | *changss to stable/havana to current head of master | 19:00 |
| clarkb | jeblair: there is a comment (from sdague iirc) to remove that job when the other job is capable of handling it all itself | 19:01 |
| jeblair | clarkb: that is not a forward-testing job | 19:01 |
| jeblair | clarkb: that is the regular job, but non-voting on a specific branch | 19:02 |
| fungi | zaro: http://paste.openstack.org/show/54720 (starts after the last message from 2.4.4 shutting down) | 19:03 |
| jeblair | clarkb: (so i think that will be testing grizzly->havana upgrades to changes on havana) | 19:04 |
| clarkb | I see. Still old to new but handling grizzly to havana? | 19:04 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use select-mirror in devstack-gate https://review.openstack.org/48064 | 19:04 |
| clarkb | jeblair: gotcha | 19:04 |
| fungi | zaro: guessing it's... ERROR com.google.gerrit.server.query.change.RevWalkPredicate : Repository "gtest-org/gerrit" unknown. | 19:04 |
| jeblair | clarkb: honestly, i wonder why they're bothering. | 19:04 |
| fungi | org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found | 19:04 |
| *** sdake has quit IRC | 19:04 | |
| *** gothicmindfood has joined #openstack-infra | 19:04 | |
| *** markmcclain has joined #openstack-infra | 19:06 | |
| *** markmcclain has quit IRC | 19:06 | |
| *** DennyZhang has quit IRC | 19:06 | |
| *** markmcclain has joined #openstack-infra | 19:06 | |
| sdague | jeblair: so the neutron versions of these jobs are probably bad copies before we had this right | 19:07 |
| *** lcheng has joined #openstack-infra | 19:07 | |
| jeblair | sdague: had what right? do we have forward testing yet? | 19:08 |
| gothicmindfood | hi guys, this is Colette, btw. | 19:08 |
| clarkb | gothicmindfood: hi there | 19:08 |
| jeblair | gothicmindfood: hi! | 19:08 |
| *** loq_mac has joined #openstack-infra | 19:09 | |
| lcheng | hello, I am having an issue with django_openstack_auth. The jenkins build keeps failing on sphinx build. | 19:09 |
| gothicmindfood | It's day #1 at HP so I'm gonna lurk everywhere and say very little - but I wanted to let you know who I was :) | 19:09 |
| lcheng | anyone have an idea? | 19:09 |
| jeblair | gothicmindfood: congrats! | 19:10 |
| zaro | gothicmindfood: welcome! | 19:10 |
| jeblair | lcheng: do you have a link to a failed job? | 19:10 |
| lcheng | here is the link: http://logs.openstack.org/74/60574/2/check/gate-django_openstack_auth-docs/a8618cb/console.html | 19:10 |
| *** jcooley_ has quit IRC | 19:11 | |
| fungi | gothicmindfood: are you in monty's group with clarkb then? planning to work on infra things with us? | 19:11 |
| fungi | also, welcome, of course! | 19:11 |
| lcheng | jeblair: thanks for looking | 19:12 |
| gothicmindfood | fungi: I am on Monty's group. Probably I will be hopping onto storyboard things. | 19:12 |
| sdague | jeblair: no yet - https://review.openstack.org/#/c/53879/ got to cycle back around to it | 19:12 |
| gothicmindfood | but I loved infra sessions in HK, so I will also be here. This is where the cool kids are, anyways, right ;) | 19:13 |
| *** rossella_s has joined #openstack-infra | 19:13 | |
| fungi | gothicmindfood: excellent! we need that working yesterday, so all the momentum you can bring it will be most welcome | 19:13 |
| jeblair | sdague: ok; if you spot something wrong with those jobs, can you leave a review? | 19:13 |
| sdague | yeh | 19:13 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Adding an option to use qpid instead of rabbit or zeromq https://review.openstack.org/55829 | 19:13 |
| ArxCruz | jeblair: done ^ | 19:13 |
| fungi | gothicmindfood: and yes, i tend to agree that infra is where the coolest things happen ;) | 19:13 |
| sdague | actually, a pointer on what's wrong with this one - https://review.openstack.org/#/c/53879/7 about where else I need to definite it would be good | 19:14 |
| zaro | fungi: interesting. gtest-org/gerrit is indeed not a repo. not sure why gerrit attempts to access it. could it be that it was created then deleted or something? | 19:14 |
| zaro | fungi: 'gerrit' project doesn't appear on the list of projects on review-dev | 19:15 |
| fungi | zaro: good question... if it's not in the review_dev.projects.yaml.erb then manage-projects shouldn't have tried to add it... | 19:16 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1253482 https://review.openstack.org/60917 | 19:16 |
| uvirtbot | Launchpad bug 1253482 in keystone "Keystone default port in linux local ephemeral port range. Devstack should shift range." [Undecided,In progress] https://launchpad.net/bugs/1253482 | 19:16 |
| *** sdake has joined #openstack-infra | 19:17 | |
| *** jcooley_ has joined #openstack-infra | 19:17 | |
| zaro | fungi: is there some way we can check the existence of 'gtest-org/gerrit'? gerrit seems to think it's here: /home/gerrit2/review_site/git/gtest-org/gerrit | 19:19 |
| jeblair | zaro, fungi: i don't have a perfect recollection, but i would not be surprised if it was deleted | 19:20 |
| *** johnthetubaguy has quit IRC | 19:21 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template https://review.openstack.org/53879 | 19:21 |
| fungi | zaro: jeblair: right, gerrit thinks it's there, but that git repo is not present on the filesystem | 19:21 |
| sdague | jeblair: ok, that should be the forward job | 19:21 |
| fungi | zaro: i also noticed that while i can get to the all-projects project, i can't view branches for https://review-dev.openstack.org/#/admin/projects/gtest-org/gtest,branches | 19:21 |
| fungi | or even just https://review-dev.openstack.org/#/admin/projects/gtest-org/gtest | 19:22 |
| fungi | checking to see if it's logged any new errors from tryin | 19:22 |
| fungi | g | 19:22 |
| zaro | yes, not viewable from UI, but i can clone it so it's there and i (registered user) have access. | 19:23 |
| fungi | nope, no log entries at all from that | 19:23 |
| mattray | Any way we can nudge this one? https://review.openstack.org/#/c/60875/ We're forking the Chef cookbooks for grizzly and want to keep the same reviewers | 19:23 |
| fungi | zaro: did you try setting up a 2.4.4 with a project on it and then restarting with the 2.8 warfile? | 19:23 |
| mattray | s/fork/branching/ | 19:24 |
| zaro | no. i haven't but can give that a try as a baseline | 19:24 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add e-r query for bug 1253482 https://review.openstack.org/60917 | 19:25 |
| uvirtbot | Launchpad bug 1253482 in keystone "Keystone default port in linux local ephemeral port range. Devstack should shift range." [Undecided,In progress] https://launchpad.net/bugs/1253482 | 19:25 |
| *** jorisroovers has joined #openstack-infra | 19:25 | |
| *** gothicmindfood has quit IRC | 19:25 | |
| jeblair | mattray: you should expect a 2-business-day turnaround for stackforge changes | 19:25 |
| mattray | jeblair: thanks! | 19:26 |
| mattray | expectations are set properly now | 19:27 |
| *** gothicmindfood has joined #openstack-infra | 19:27 | |
| *** gothicmindfood has quit IRC | 19:28 | |
| *** gothicmindfood has joined #openstack-infra | 19:29 | |
| fungi | zaro: it would be helpful to know whether we should expect that to "just work" so we don't go looking in the wrong places | 19:30 |
| jeblair | lcheng: think i found it | 19:31 |
| * gothicmindfood is running irssi for the first time so excuse weirdness. | 19:31 | |
| zaro | fungi: ok. will let you know a little later. | 19:31 |
| lcheng | jeblair: cool! | 19:31 |
| jeblair | lcheng: testing; should have a patch in a sec if it works | 19:32 |
| *** sdake has quit IRC | 19:32 | |
| fungi | zaro: interestingly, the acl ("access") page for those projects is accessible without error, but it gives me an empty acl | 19:32 |
| *** denis_makogon has quit IRC | 19:32 | |
| fungi | nothing but " Rights Inherit From: | 19:32 |
| fungi | All-Projects | 19:32 |
| fungi | and then the history cgit link | 19:33 |
| *** DennyZhang has joined #openstack-infra | 19:33 | |
| *** senk has quit IRC | 19:33 | |
| *** sdake has joined #openstack-infra | 19:34 | |
| zaro | fungi: i noticed that a few are empty. i guess that just means that project inherts all acls from all-projects | 19:34 |
| fungi | zaro: that's how it looks to me | 19:34 |
| zaro | fungi: the openstack-ci project does have acls set though. | 19:34 |
| clarkb | one of the wookiee projects should ahve acls too, mordred used that for testing manage-proejcts | 19:35 |
| openstackgerrit | James E. Blair proposed a change to openstack/requirements: Add django_openstack_auth to projects https://review.openstack.org/60921 | 19:35 |
| jeblair | lcheng: ^ i think that should fix it | 19:35 |
| fungi | zaro: so it does. good eye | 19:35 |
| *** AaronGr is now known as AaronGr_Lunch | 19:35 | |
| jeblair | lcheng: recently we made a change to the mirror selection script that uses that file to determine which projects use the openstack pypi mirror | 19:35 |
| jeblair | lcheng: it looks like mordred missed adding django_openstack_auth to that list, so it stopped using the mirror | 19:36 |
| fungi | clarkb: zaro: actually https://review-dev.openstack.org/#/admin/projects/gtest-org/test-manage-project,access | 19:36 |
| jeblair | lcheng: (it used to use it because its name started with 'openstack/') | 19:36 |
| jeblair | lcheng: so the doc build job changed because it started using a different version of sphinx | 19:36 |
| jeblair | lcheng: after that change merges, it should switch back to using the mirror and start working again. | 19:37 |
| *** loq_mac has quit IRC | 19:37 | |
| jeblair | clarkb, fungi: speedy review of https://review.openstack.org/#/c/60921/ will help unblock django_openstack_auth (i hope) | 19:37 |
| *** DennyZhang has quit IRC | 19:37 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template https://review.openstack.org/53879 | 19:37 |
| *** SushilKM has quit IRC | 19:37 | |
| *** dstanek has quit IRC | 19:37 | |
| fungi | clarkb: jeblair: mordred: any objections to adding zaro to https://review-dev.openstack.org/#/admin/groups/1,members so he can poke around? | 19:37 |
| jeblair | fungi: no objections | 19:38 |
| clarkb | go for it | 19:38 |
| clarkb | jeblair: looking | 19:38 |
| *** dstanek has joined #openstack-infra | 19:38 | |
| fungi | zaro: you have admin in the review-dev webui now, so you should be able to see the same things we see there (gerrit has an annoying habit of hiding acl entries when you're not in the groups allowed by them) | 19:39 |
| lcheng | jeblair: that was fast! thanks a lot | 19:39 |
| lcheng | jeblair: going to try to re-run the build | 19:40 |
| zaro | what a privilege! thanks. | 19:40 |
| *** jcooley_ has quit IRC | 19:40 | |
| clarkb | jeblair: I am going to let the check tests run before approving, slight worry that the syncing that happens in the tempest tests may break it | 19:40 |
| fungi | zaro: don't spend it all in one place | 19:40 |
| mgagne | jeblair: in nodepool repo, there is 2 README: README and README.rst (empty). Github only shows README.rst (and therefore no content). Should I move the content of README to README.rst or delete README.rst? | 19:40 |
| jeblair | mgagne: https://review.openstack.org/#/c/60579/ | 19:41 |
| *** gyee has quit IRC | 19:41 | |
| mgagne | jeblair: :O (okay) | 19:41 |
| krotscheck | Since we're doing introductions, I'll be joining up on Storyboard with gothicmindfood this next monday. I like long walks on the beach, romantic movies, and sacrificing small vertebrates to the elder gods. | 19:41 |
| fungi | krotscheck: what, not the outer gods? heathen! | 19:42 |
| krotscheck | What can I say: I like calamari. | 19:42 |
| anteaya | krotscheck: thank the elder gods the invertebrates are safe | 19:42 |
| zaro | clarkb: too cold to leave the apartment today? | 19:42 |
| clarkb | zaro: that and I am cooking dinner all day | 19:43 |
| jeblair | krotscheck: do you go by any other names? | 19:43 |
| mordred | krotscheck, gothicmindfood: o hai | 19:43 |
| clarkb | picked up ribs at the store last night and they need babysitting | 19:43 |
| gothicmindfood | krotscheck: sooooooo excited. | 19:43 |
| fungi | krotscheck: what do you have against chordates anyway? | 19:43 |
| gothicmindfood | anteaya: hi hi hi hi! | 19:43 |
| *** randallburt has quit IRC | 19:43 | |
| anteaya | gothicmindfood: what do you like to sacrifice to the gods | 19:43 |
| gothicmindfood | mordred: hi back! | 19:43 |
| anteaya | since we are getting that out of the way first | 19:44 |
| zaro | got a slow cooker? | 19:44 |
| krotscheck | mordred: lol hai | 19:44 |
| fungi | (note: most of our infrastructure is propped up with animal sacrifice) | 19:44 |
| mordred | so many dead pigs | 19:44 |
| krotscheck | fungi: Nothing? They're tasty too. | 19:44 |
| clarkb | zaro: I do | 19:44 |
| * anteaya swings rubber chicken | 19:44 | |
| gothicmindfood | anteaya: erm. Do the gods like fancy wine? | 19:45 |
| zaro | clarkb: a smoker? | 19:45 |
| clarkb | zaro: no smoker | 19:45 |
| anteaya | gothicmindfood: some do | 19:46 |
| *** sdake has quit IRC | 19:46 | |
| anteaya | I offer up the dust and cat hair in my dust pan with irregular frequency | 19:46 |
| anteaya | some got has got to like that | 19:46 |
| anteaya | *god | 19:46 |
| clarkb | mordred: we are now testing the correct requirements when requirements changes are proposed. Isn't that exciting? | 19:46 |
| * anteaya finds that exciting | 19:47 | |
| fungi | krotscheck: gothicmindfood: in case nobody's warned you yet, weekly infrastructure team irc meeting is tomorrow (tuesdays) at 19:00 utc in #openstack-meeting. agenda is updated at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting | 19:47 |
| clarkb | we will also be sorting through our bug list and cleaning it up as appropriate | 19:48 |
| * gothicmindfood offers up the best Morgon for the gods | 19:48 | |
| fungi | right, tomorrow == bugday. excellent reminder | 19:48 |
| mordred | clarkb: I'm VERY excited! | 19:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Review and create Puppet configuration for dev server https://review.openstack.org/53644 | 19:48 |
| mordred | clarkb: do we know that because something didn't work but was properly caught by the gate? | 19:48 |
| fungi | smarcet: ^ | 19:48 |
| fungi | (i'll have it in dns momentarily) | 19:49 |
| jeblair | heh, we probably should have updated the git subject in one of those 50 revisions :) | 19:49 |
| clarkb | mordred: no, but changes to requirements will hit pypi.o.o and use the proper git revision | 19:49 |
| gothicmindfood | fungi: yeah, mordred sent me a welcome-to-the-jungle message with a meeting invite for Tuesdays in it yesterday. :) | 19:49 |
| fungi | jeblair: aww crap. i kept meaning to suggest that and never remembered when i was writing comments. bad me | 19:49 |
| clarkb | mordred: what we can do to test is propose the django1.6 change back torequirements | 19:49 |
| clarkb | mordred: that should fail | 19:49 |
| clarkb | I will do that | 19:49 |
| jamesmcarthur | fungi: jeblair: smart: thanks for the approval guys! looking forward to next steps :) | 19:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack/requirements: Allow django >=1.6 https://review.openstack.org/60923 | 19:51 |
| mordred | clarkb: ++ | 19:51 |
| *** mrodden has quit IRC | 19:53 | |
| *** mrodden has joined #openstack-infra | 19:54 | |
| *** jamesmcarthur has quit IRC | 19:54 | |
| *** dstanek has quit IRC | 19:56 | |
| *** DinaBelova has joined #openstack-infra | 19:57 | |
| *** jerryz has quit IRC | 19:59 | |
| *** melwitt has joined #openstack-infra | 19:59 | |
| *** nati_uen_ has joined #openstack-infra | 20:00 | |
| *** ^demon has joined #openstack-infra | 20:00 | |
| krotscheck | Is there a channel out there for random chatter? | 20:01 |
| *** ^demon has quit IRC | 20:01 | |
| fungi | krotscheck: there's an entire irc network full of them. but agreed, it's nice to keep the random chatter here to a minimum | 20:02 |
| *** ^demon has joined #openstack-infra | 20:02 | |
| *** ^demon has joined #openstack-infra | 20:02 | |
| *** galstrom has left #openstack-infra | 20:02 | |
| fungi | (and i'm one of the biggest offenders where that's concerned, i acknowledge) | 20:02 |
| anteaya | krotscheck: #openstack-dev for general openstack stuff | 20:03 |
| *** nati_ueno has quit IRC | 20:03 | |
| krotscheck | anteaya: Aweomse, thanks. | 20:03 |
| *** ^d has quit IRC | 20:04 | |
| *** rongze has joined #openstack-infra | 20:04 | |
| anteaya | krotscheck: np | 20:05 |
| *** rfolco has quit IRC | 20:06 | |
| *** ^demon has quit IRC | 20:06 | |
| *** jamesmcarthur has joined #openstack-infra | 20:07 | |
| *** ^d has joined #openstack-infra | 20:07 | |
| devananda | krotscheck: hi there! | 20:09 |
| *** AaronGr_Lunch is now known as AaronGr | 20:09 | |
| krotscheck | devananda: oh shit you found me! | 20:09 |
| devananda | :) | 20:09 |
| *** sdake_ has quit IRC | 20:12 | |
| *** sdake_ has joined #openstack-infra | 20:13 | |
| *** dstanek has joined #openstack-infra | 20:13 | |
| *** NikitaKonovalov has quit IRC | 20:14 | |
| * anteaya wags her finger at krotscheck | 20:14 | |
| anteaya | language | 20:14 |
| *** loq_mac has joined #openstack-infra | 20:16 | |
| *** galstrom has joined #openstack-infra | 20:18 | |
| *** ^d has quit IRC | 20:20 | |
| *** ^demon has joined #openstack-infra | 20:20 | |
| *** markmc has quit IRC | 20:21 | |
| *** ^demon has quit IRC | 20:21 | |
| *** ^d has joined #openstack-infra | 20:22 | |
| *** rcleere has quit IRC | 20:22 | |
| *** rcleere_ has joined #openstack-infra | 20:22 | |
| *** gyee has joined #openstack-infra | 20:23 | |
| *** DinaBelova has quit IRC | 20:25 | |
| *** yolanda has quit IRC | 20:25 | |
| *** bpokorny has quit IRC | 20:26 | |
| *** ^d has quit IRC | 20:26 | |
| *** ^d has joined #openstack-infra | 20:27 | |
| *** thedodd has joined #openstack-infra | 20:28 | |
| *** vkozhukalov has quit IRC | 20:28 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 20:29 |
| fungi | smarcet: you should be able to ssh into openstackid-dev.openstack.org at your convenience | 20:30 |
| notmyname | can someone explain how the DocImpact tag works? what are the technical guts of it? and the end question is, "can I have other tags?" | 20:30 |
| EmilienM | sdague: I took your patch in consideration and did a rebase: https://review.openstack.org/#/c/59039/ | 20:31 |
| smarcet | fungi: cool will check on it | 20:31 |
| clarkb | notmyname: gerrit has hooks similar to git hooks. We have a script that runs in the new patchset hook and change merged hook that scans git commit messages for known strings to take actions on them | 20:31 |
| fungi | notmyname: it's a gerrit hook script at http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/notify_impact.py | 20:31 |
| mgagne | notmyname: AFAIK, tags are handled by jeepyb: http://ci.openstack.org/jeepyb.html | 20:31 |
| notmyname | heh | 20:31 |
| fungi | it's a chorus | 20:31 |
| mgagne | notmyname: http://ci.openstack.org/jeepyb.html#impact-notification | 20:31 |
| clarkb | notmyname: long story short is yes you can have other tags through the modification of where we call the hooks | 20:31 |
| clarkb | fungi: :) | 20:31 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 20:32 |
| *** nati_uen_ has quit IRC | 20:32 | |
| fungi | notmyname: security impact is already another example of one in use | 20:32 |
| *** thomasbiege has joined #openstack-infra | 20:32 | |
| fungi | and i think the i18n team was talking about one for translation impact maybe soon? | 20:32 |
| EmilienM | clarkb: also i applied your suggestion, let me know if I'm wrong | 20:32 |
| clarkb | EmilienM: looking | 20:32 |
| *** ^d has quit IRC | 20:32 | |
| *** sdake has joined #openstack-infra | 20:32 | |
| *** sdake has quit IRC | 20:32 | |
| *** sdake has joined #openstack-infra | 20:32 | |
| *** ^d has joined #openstack-infra | 20:33 | |
| *** nati_ueno has joined #openstack-infra | 20:33 | |
| EmilienM | clarkb: i forget {pipeline} in job template i think | 20:33 |
| smarcet | fungi: confirmed, i was able to access through ssh, will discuss with Jimmy next steps, thx a lot :) | 20:34 |
| clarkb | EmilienM: yup I think jenkins caught that | 20:34 |
| clarkb | EmilienM: otehrwise it lgtm | 20:34 |
| EmilienM | :) | 20:34 |
| *** sarob has joined #openstack-infra | 20:34 | |
| *** jcooley_ has joined #openstack-infra | 20:34 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 20:34 |
| EmilienM | clarkb: ok, cool, i wonder what think sdague and others | 20:34 |
| notmyname | fungi: clarkb: is there any functionality to have a "Tags: foo, bar, baz" commit message line that would then allow searching/notification on tags? ie more freeform than something hard-coded | 20:34 |
| *** mrodden has quit IRC | 20:34 | |
| fungi | smarcet: glad to help, sorry it took so long to get going, but it should be a lot easier to iterate on quickly now that there's a dev server to try out ideas on | 20:34 |
| notmyname | http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/notify_impact.py#n155 seems to be where docimpact is defined. where is the security impact? | 20:35 |
| *** alcabrera has quit IRC | 20:36 | |
| clarkb | notmyname: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/gerrit/change-merged and http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/gerrit_patchset-created.erb where the script is called | 20:36 |
| clarkb | docimpact is currently hard coded to do the bug instead of email thing | 20:36 |
| *** ^d has quit IRC | 20:36 | |
| clarkb | notmyname: I would be open to that. Would be even more useful when we have gerrit indexing things for us | 20:37 |
| *** ^d has joined #openstack-infra | 20:37 | |
| *** jaustinpage has quit IRC | 20:38 | |
| *** gothicmindfood has quit IRC | 20:39 | |
| notmyname | clarkb: so the http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/gerrit_patchset-created.erb is called for every patchset? and the "--impact SecurityImpact" is a filter? or is that script only called if SecurityImpact is in the commit (ie filtered by something else) | 20:40 |
| clarkb | notmyname: it is the filter | 20:41 |
| clarkb | so it is called for everything looking for that string | 20:41 |
| *** gothicmindfood has joined #openstack-infra | 20:41 | |
| *** mrodden has joined #openstack-infra | 20:41 | |
| zaro | fungi: i've testing going from basic 2.4 to 2.8 and everthing worked fine for me. | 20:42 |
| zaro | fungi: had a few groups, accounts, and reviews setup in 2.4 and everthing migrated to 2.8 just fine. | 20:42 |
| notmyname | clarkb: ah I see. so change-merged is called for every changeset (or commit merged) and it in turn calls the jeepyb stuff to take the appropriate actions | 20:42 |
| zaro | fungi: when i do the upgrade i get some reporting of the process. it looks something like this.. http://paste.openstack.org/show/54724 | 20:43 |
| clarkb | notmyname: yup exactly | 20:43 |
| clarkb | notmyname: so simple way of adding a new string to filter for is call that script with different args | 20:43 |
| zaro | fungi: can we get that log from the job that did the upgrade? it will tell us if anything crapped out during the process. | 20:43 |
| notmyname | clarkb: which means that today, for every tag we would want to filter on, we'd have to have a new line in change-merged or gerrit_patchset_created | 20:44 |
| clarkb | notmyname: correct | 20:44 |
| zaro | fungi: also i think when you do an upgrade of gerrit you must first stop the 2.4 instance before running the upgrade process. do you know if that happened? | 20:44 |
| notmyname | clarkb: ok, thanks | 20:44 |
| *** dprince has joined #openstack-infra | 20:45 | |
| zaro | fungi: i'm concerned that since the puppet agent wasn't running maybe the upgrade process didn't work smoothly? | 20:46 |
| fungi | zaro: i believe puppet swallows the stdout. also, what in our infrastructure ran the 'java -jar gerrit-2.8.war init --batch -d my_site' command? was there a separate review to add that exec? all 60043 did was replace the warfile... | 20:46 |
| *** jcooley_ has quit IRC | 20:46 | |
| smarcet | fungi: so, we already have in place a public git repo at github | 20:46 |
| *** cody-somerville has joined #openstack-infra | 20:47 | |
| *** cody-somerville has quit IRC | 20:47 | |
| *** cody-somerville has joined #openstack-infra | 20:47 | |
| smarcet | do u guys need the address to set up gerrit/jenkins? | 20:47 |
| fungi | smarcet: get me the url and i'll put together a change to import it into openstack-infra if you're ready for that to happen | 20:47 |
| *** jcooley_ has joined #openstack-infra | 20:47 | |
| smarcet | sure | 20:47 |
| smarcet | on it | 20:47 |
| smarcet | fungi: here is it , https://github.com/smarcet/openstackid | 20:48 |
| fungi | zaro: i suspect we're going to need a transition script added to puppet which stops gerrit, runs the upgrade command (logging it to a file for later inspection) and then starts gerrit again | 20:49 |
| *** fbo is now known as fbo_away | 20:49 | |
| clarkb | did the db get corrupted? | 20:49 |
| clarkb | fungi: zaro: or are we still trying to sort out what went wrong | 20:50 |
| fungi | clarkb: possible, but i suspect it's not too corrupted (and we can always grab the mysqldump from bup and retry that) | 20:50 |
| *** dkliban has quit IRC | 20:50 | |
| fungi | clarkb: more likely is that we never ran the schema update | 20:50 |
| *** randallburt has joined #openstack-infra | 20:51 | |
| * fungi assumed, i suppose wrongly, that the proposed change was all that was needed and that gerrit would autodetect if it needed to do db schema migrations | 20:51 | |
| *** thomasbiege has quit IRC | 20:51 | |
| *** lcheng has quit IRC | 20:52 | |
| sdague | EmilienM: so https://review.openstack.org/#/c/59039/ is close, but I think you changed a job name in there by accident | 20:52 |
| sdague | comments inline | 20:52 |
| *** Ajaeger has joined #openstack-infra | 20:52 | |
| clarkb | fungi: it should autodetect and do schema migrations iirc | 20:52 |
| fungi | clarkb: zaro: so it sounds like we need slightly more puppet/scripting to pull this off | 20:52 |
| clarkb | fungi: the install script does that | 20:53 |
| clarkb | which we run in batch mode with a preexisting config | 20:53 |
| *** mattray has left #openstack-infra | 20:53 | |
| fungi | clarkb: oh, that runs automatically, subscribed to the warfile change... right | 20:53 |
| fungi | i hope it logs somewhere in that case | 20:53 |
| clarkb | fungi: perhaps our config is insufficnet? | 20:53 |
| clarkb | zaro: did you run the upgrade interactively? | 20:54 |
| *** jcooley_ has quit IRC | 20:54 | |
| *** loq_mac has quit IRC | 20:54 | |
| *** datsun180b has quit IRC | 20:55 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/gerritlib: Add connection attempt limit https://review.openstack.org/52986 | 20:55 |
| *** marun has joined #openstack-infra | 20:55 | |
| *** jcooley_ has joined #openstack-infra | 20:55 | |
| EmilienM | sdague: a typo indeed :( | 20:56 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: Neutron support in Grenade jobs https://review.openstack.org/59039 | 20:57 |
| *** jcooley_ has quit IRC | 20:57 | |
| *** datsun180b has joined #openstack-infra | 20:58 | |
| Ajaeger | Hi! A question for the experts: When I filed this review: https://review.openstack.org/#/c/60931/, the referenced bug - via closes-bug - was not updated at all. Did I miss some configuration when setting up the new openstack-doc-tools repo? | 20:58 |
| clarkb | I bet that project doesn't know which org to find bugs under | 20:59 |
| mgagne | Ajaeger: ^ | 20:59 |
| *** yamahata_ has joined #openstack-infra | 20:59 | |
| Ajaeger | clarkb: do you have a reference to a config file that I should check? | 21:00 |
| mgagne | Ajaeger: jeepyb is doing this mapping | 21:00 |
| mgagne | unless it has been moved somewhere else recently | 21:00 |
| *** amotoki has joined #openstack-infra | 21:01 | |
| clarkb | mgagne: I think it may haev moved and I haven't kept up | 21:01 |
| clarkb | fungi: ^ | 21:01 |
| mgagne | Ajaeger: should be the launchpad property here: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/templates/review.projects.yaml.erb | 21:01 |
| mgagne | clarkb: https://review.openstack.org/#/c/60329/2 | 21:01 |
| *** mrodden has quit IRC | 21:01 | |
| *** herndon has quit IRC | 21:02 | |
| fungi | clarkb: mgagne: Ajaeger: right now it works from either place (the hardcoded entry in the script is overridden by any entry in the projects.yaml file) | 21:02 |
| *** ^d has quit IRC | 21:02 | |
| *** herndon has joined #openstack-infra | 21:03 | |
| *** fbo_away is now known as fbo | 21:03 | |
| Ajaeger | fungi: I guess review.projects.yaml.erb is preferred? I'll send a patch.. | 21:04 |
| notmyname | so if I know a gate is going to fail (based on the console, can I preemptively reenque it? | 21:04 |
| *** gothicmindfood has quit IRC | 21:04 | |
| notmyname | https://jenkins01.openstack.org/job/gate-tempest-dsvm-full/1029/ | 21:05 |
| clarkb | notmyname: currently the only way to do that is to push a new patchset (to kick it out of the pipeline, then approve that). I know jeblair wants zuul to do that on a -2 as well | 21:05 |
| fungi | Ajaeger: so, your change was a commit to openstack/openstack-doc-tools which means the hook looks for a launchpad project named openstack-doc-tools. instead the bug you referenced is for openstack-manuals on launchpad, which is why it was not updated | 21:05 |
| notmyname | clarkb: but because of the zuul pipelining, it would just get in line and then get restarted when the current one fails | 21:05 |
| notmyname | so it probably won't work | 21:06 |
| fungi | Ajaeger: yes, you can add a "launchpad" entry for your project in review.projects.yaml.erb in openstack-infra/config (there should be some examples there already) telling it which launchpad project houses the bugs for that repo | 21:06 |
| clarkb | notmyname: kicking it out of the pipeline will force a reset | 21:06 |
| clarkb | notmyname: then you will reenqueue it | 21:06 |
| zaro | clarkb: i ran the upgrade on my laptop, so yes interactively. | 21:07 |
| fungi | clarkb: though will kicking it out actually cause a reset now that the gate is already rebasing the rest of the queue on nnfi? | 21:07 |
| clarkb | if the nnfi has already been found then no | 21:08 |
| Ajaeger | fungi: thanks! | 21:08 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add launchpad entries for openstack-manuals https://review.openstack.org/60943 | 21:09 |
| notmyname | what is the current timeout? it used to be 60 minutes, but it's been running for over 70 now | 21:10 |
| clarkb | notmyname: it depends on the job, For that job I believe it is 90 minutes, I can try killing the job now instead of doing a dance with zuul | 21:10 |
| clarkb | notmyname: should I go ahead and do that? | 21:11 |
| notmyname | clarkb: well, actually.... | 21:11 |
| notmyname | clarkb: when this was happening last week, I filed https://bugs.launchpad.net/tempest/+bug/1258682 | 21:11 |
| uvirtbot | Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,New] | 21:11 |
| notmyname | clarkb: but there is pretty much 0 info there, because the timeout kills the logs | 21:12 |
| notmyname | clarkb: so there are a ton of "FAIL" lines in the currently running job, but I have no idea why or even what to point people to. but it seems to be a common error case | 21:12 |
| clarkb | notmyname: so maybe better to save that slave, then kill the job. That will allow us to look at logs | 21:12 |
| notmyname | clarkb: https://jenkins01.openstack.org/job/gate-tempest-dsvm-full/1029/ | 21:13 |
| clarkb | I will hold the slave in a minute | 21:13 |
| zaro | clarkb, fungi : the gerrit upgrade happens but it doesn't stop an existing gerrit before running. | 21:13 |
| clarkb | zaro: but why is that a problem? | 21:14 |
| clarkb | I am guessing DB related things? | 21:14 |
| *** emagana has joined #openstack-infra | 21:15 | |
| fungi | zaro: you mean the one in puppet? if that presents an issue, then we can probably add invoke-rc.d calls to it at the beginning and end fairly easily i would think | 21:15 |
| zaro | clarkb: i'm guessing as well. that's what is recommened. | 21:15 |
| *** gothicmindfood has joined #openstack-infra | 21:15 | |
| jeblair | zaro: it used to :( | 21:15 |
| zaro | let me take another look for sure. | 21:16 |
| clarkb | notmyname: I have put that slave in a hold state. Do you know who I should be wrangling to hop on that node and debug? iirc these were cinder problem? | 21:17 |
| notmyname | clarkb: I have no idea, honestly. all I have to go on is the FAIL lines in the console log | 21:17 |
| jeblair | fungi: ok to create new projects? | 21:18 |
| fungi | jeblair: on review-dev? | 21:18 |
| jeblair | mtreinish: ^ see convo between clarkb and notmyname | 21:18 |
| jeblair | fungi: in prod | 21:18 |
| clarkb | notmyname: ok, I will try QA first | 21:18 |
| clarkb | mtreinish: you have been summoned | 21:18 |
| notmyname | clarkb: thanks | 21:18 |
| fungi | jeblair: should be--seemed to be working recently | 21:18 |
| jeblair | fungi: k, i thought i saw something about some lingering races | 21:18 |
| mtreinish | jeblair, clarkb: ok reading | 21:18 |
| fungi | jeblair: the couple i cleaned up were, i think, from before clarkb's group sleep merged, as you suggested | 21:18 |
| jeblair | fungi: ack. aprving then. | 21:19 |
| fungi | jeblair: but i'll definitely do a sanity check behind it once it hits gerrit | 21:19 |
| fungi | since i'm curious | 21:19 |
| *** fifieldt has joined #openstack-infra | 21:20 | |
| clarkb | I am curious too | 21:20 |
| clarkb | it is a super hack :) | 21:20 |
| pleia2 | clarkb: does this just need a rebase? https://review.openstack.org/#/c/58871/ | 21:20 |
| clarkb | mtreinish: tl;dr is https://jenkins01.openstack.org/job/gate-tempest-dsvm-full/1029/ is going to timeout meaning we can't collect any of the logs. I have held that slave so that someone can log into it and collect the necessary logs to figure out what is going on and possibly do local testing with a borked machine | 21:20 |
| clarkb | mtreinish: any chance you would be interested in doing that? | 21:20 |
| mtreinish | clarkb: I wouldn't say I'm interested, but I can take a quick look | 21:21 |
| mtreinish | hopefully it'll be something obvious | 21:21 |
| *** galstrom has left #openstack-infra | 21:21 | |
| clarkb | mtreinish: can you point me at a public ssh key I can put on the slave? | 21:21 |
| jeblair | mtreinish: you should have a passing familiarity with this problem as i think you looked at it briefly last week | 21:21 |
| clarkb | pleia2: no it should be abandoned. We aren't doing fedora nodepool slaves until nodepool can sudo on them | 21:21 |
| *** mrodden has joined #openstack-infra | 21:21 | |
| clarkb | pleia2: I will abandon it | 21:21 |
| pleia2 | clarkb: ok, thanks | 21:21 |
| mtreinish | jeblair: yeah I remember looking at the timed-out console logs last week | 21:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 21:21 |
| notmyname | clarkb: I can't see any pattern to what tests FAIL | 21:22 |
| mtreinish | notmyname: my guess is a service crashed or something | 21:22 |
| mtreinish | this used to happen when something related ceilometer would hang nova compute | 21:23 |
| *** _david_ has joined #openstack-infra | 21:24 | |
| *** ^d has joined #openstack-infra | 21:25 | |
| zaro | fungi: there's two puppet entries that run the gerrit.war. I'm assuming that 'gerrit-init' one gets run on a change to gerrit.war file. -> http://git.openstack.org/cgit/openstack-infra/config/tree/modules/gerrit/manifests/init.pp#n484 | 21:25 |
| * anteaya shops for upvotes for EmilienM Neutron Grenade patch https://review.openstack.org/#/c/59039/ | 21:26 | |
| _david_ | clarkb, have you noticed: https://gerrit-review.googlesource.com/#/c/52748 ? | 21:26 |
| anteaya | we are in the -neutron meeting, we might get some reviews out of it | 21:26 |
| *** sdake has quit IRC | 21:26 | |
| EmilienM | anteaya: o/ | 21:26 |
| anteaya | :D | 21:26 |
| clarkb | _david_: I did see it on the ml I am excited about it | 21:26 |
| anteaya | EmilienM: looking good | 21:26 |
| fungi | zaro: i have the log of the puppet apply run, though it looks like the gerrit-initial-init exec doesn't log the output anywhere helpful | 21:26 |
| clarkb | _david_: glad that someone else saw it as useful and decided to do it correclty :) | 21:27 |
| zaro | fungi: not sure how we could save the output from the upgrade. | 21:27 |
| *** sdake has joined #openstack-infra | 21:27 | |
| *** sdake has quit IRC | 21:27 | |
| *** sdake has joined #openstack-infra | 21:27 | |
| mtreinish | clarkb: won't the job finish to completion now that you've held the slave? (which will copy the logs) | 21:27 |
| fungi | zaro: >> /home/gerrit2/review_site/logs/upgrade_log | 21:27 |
| zaro | fungi: gerrit-initial-init is the wrong one, that one does not have the stop gerrit | 21:27 |
| zaro | fungi: i think it needs to run 'gerrit-init'. can you double check. | 21:27 |
| clarkb | mtreinish: no the jenkins timeout is independent of nodepool deleting the slave | 21:28 |
| zaro | ? | 21:28 |
| clarkb | mtreinish: the problem is that when jenkins times out a build the scriptage that copies log files around doesn't run so no logs get copied | 21:28 |
| _david_ | clarkb, yeah good ideas always win ;-) And as you tried to solve it, the gerrit plugin API wasn't mature enough to solve that kind of problems | 21:28 |
| fungi | zaro: oh, good point. gerrit-init right (initial is for the first ever install of gerrit) | 21:28 |
| zaro | fungi: any errors? | 21:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Add workInProgress Gerrit ACL for Documentation https://review.openstack.org/60171 | 21:29 |
| *** sdake_ has quit IRC | 21:30 | |
| fungi | zaro: no, just says notice: /Stage[main]/Gerrit/Exec[gerrit-initial-init]: Triggered 'refresh' from 1 events | 21:30 |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Mark puppet-unit-3.2/3 jobs as voting https://review.openstack.org/60140 | 21:30 |
| *** loq_mac has joined #openstack-infra | 21:30 | |
| fungi | zaro: like i said, puppet agent swallows the stdout from execs, doesn't regurgitate or log them | 21:31 |
| zaro | fungi: does that mean puppet ran the wrong gerrit init? | 21:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Use force when purging https://review.openstack.org/60313 | 21:32 |
| fungi | zaro: no, it looks like that exec stops gerrit and then runs the new warfile's init, and then starts gerrit again after | 21:32 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Remove openstack-infra projects mappings to LP https://review.openstack.org/60332 | 21:32 |
| *** loq_mac has quit IRC | 21:33 | |
| mgagne | fungi: no sure what it's worth: http://docs.puppetlabs.com/references/latest/type.html#exec-attribute-logoutput | 21:33 |
| zaro | fungi: is there no possibility to make puppet save stdout? | 21:33 |
| fungi | zaro: mgagne just read your mind | 21:33 |
| fungi | so my >> hack suggestion in the exec appears to be unnecessary, just need to set that option i guess | 21:34 |
| zaro | ohh nice. thanks mgagne | 21:34 |
| openstackgerrit | A change was merged to openstack-infra/config: Mark puppet-unit-3.2/3 jobs as voting https://review.openstack.org/60140 | 21:34 |
| zaro | fungi: so can we redo? | 21:34 |
| *** smarcet has left #openstack-infra | 21:34 | |
| zaro | fungi: add capture stdout this time? | 21:35 |
| fungi | zaro: probably after i go grab dinner. though we may want to restore the db first before we try that | 21:35 |
| *** openstackgerrit has quit IRC | 21:37 | |
| jeblair | fungi, clarkb: i'm continuing to single-approve trivial stackforge config changes | 21:37 |
| zaro | fungi: cool. i think that should be the next step. maybe we do without puppet as a trial? | 21:37 |
| *** openstackgerrit has joined #openstack-infra | 21:37 | |
| openstackgerrit | A change was merged to openstack-infra/config: config changes to create the stackforge project milk https://review.openstack.org/60394 | 21:37 |
| fungi | zaro: well, i thought the point of this exercise was to work out how to get puppet to upgrade gerrit safely | 21:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Add manila repos to PROJECTS in d-g job https://review.openstack.org/60442 | 21:37 |
| clarkb | EmilienM: https://review.openstack.org/#/c/59039/ lgtm | 21:37 |
| zaro | no. i thought it was to see if upgrade works. | 21:37 |
| zaro | fungi: with out review-dev | 21:38 |
| zaro | sp out/our | 21:38 |
| mtreinish | clarkb, notmyname: it's got a lot of fun stack traces in the logs: http://paste.openstack.org/show/54727/ | 21:38 |
| fungi | zaro: sure, i can give that a shot too then, later this evening | 21:38 |
| EmilienM | clarkb: thank you for your help on this one | 21:38 |
| mgagne | why is there no logs for #openstack-glance? :O | 21:39 |
| mtreinish | it looks like we're hitting the ram limit though | 21:39 |
| *** denis_makogon_ has joined #openstack-infra | 21:39 | |
| zaro | fungi: excellent. then we can figure out the puppet part after. | 21:39 |
| mtreinish | and cpu limit too | 21:39 |
| openstackgerrit | A change was merged to openstack-infra/config: Run tests and add the ability to release for tooz https://review.openstack.org/60504 | 21:39 |
| *** jhesketh_ has joined #openstack-infra | 21:39 | |
| *** denis_makogon_ is now known as denis_makogon | 21:39 | |
| *** loq_mac has joined #openstack-infra | 21:39 | |
| notmyname | mtreinish: I'll add that paste to the bug | 21:40 |
| mtreinish | there are over quota traces in the nova logs | 21:40 |
| zaro | clarkb, jeblair : do we need to let anyone know that review-dev is in a broken state? | 21:40 |
| clarkb | zaro: no | 21:40 |
| mtreinish | sdague: ^^^ I think we've hit the guest size limit | 21:40 |
| mtreinish | clarkb: can we get bigger guests :) | 21:40 |
| clarkb | zaro: if previous testing and breaking things is any indications the -dev servers are fair game :) | 21:40 |
| jeblair | mtreinish: that first exception definitely looks like a cinder bug, even if it's not the thing causing this. | 21:40 |
| jeblair | mtreinish: what limit? | 21:40 |
| *** ArxCruz has quit IRC | 21:41 | |
| mtreinish | there are several log messages like: TooManyInstances: Quota exceeded for cores: Requested 20, but already used 2 of 20 cores | 21:41 |
| mtreinish | and TooManyInstances: Quota exceeded for ram: Requested 51137, but already used 128 of 51200 ram | 21:41 |
| clarkb | why would we be requesting that much ram and cpu all at once? | 21:42 |
| mtreinish | clarkb: I'm not aware of a test that does | 21:43 |
| *** boris-42 has quit IRC | 21:43 | |
| lifeless | what about the ones with the fake virt driver? | 21:44 |
| mtreinish | let me check the corresponding error message and get the req-id to see what's generating the traces | 21:44 |
| jeblair | mtreinish: and were you asking clarkb for bigger devstack-gate nodes? that seems sort of orthogonal to whatever limits devstack and tempest configure in openstack. | 21:44 |
| *** Ajaeger has left #openstack-infra | 21:45 | |
| jeblair | lifeless: that gets its own jenkins job; these failures are in one that does not run the large-ops test. | 21:45 |
| jeblair | (so if it's being run, it's accidental) | 21:45 |
| fifieldt | hiya - just wondering what the best place is to put that welcome message patchset-created hook for testing on review-dev ... | 21:45 |
| *** loq_mac has quit IRC | 21:46 | |
| jeblair | fifieldt: review-dev is out of commission at the moment (it's dedicated to testing a new gerrit) | 21:46 |
| jeblair | fifieldt: but you'll want to add it to modules/openstack_project/templates/gerrit_patchset-created.erb | 21:46 |
| mtreinish | jeblair: yeah I was but you're right. Also it looks like a false error, it looks like it was caused by: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/admin/test_servers_negative.py#n59 | 21:47 |
| fifieldt | argh | 21:47 |
| fifieldt | any idea when it'll be back jeblair? | 21:47 |
| *** sandywalsh has quit IRC | 21:47 | |
| clarkb | fuel-docs seems to have landed properly (though no new groups may have been created for that since fuel has a bunch of other repos) | 21:47 |
| clarkb | fifieldt: hopefully after fungi has dinner | 21:48 |
| *** hashar has joined #openstack-infra | 21:48 | |
| fifieldt | that's fast, cool :) | 21:48 |
| jeblair | fifieldt: could be a while. i'm not sure it's set up to have a different configuration of that anyway. i'd recommend either testing it locally with your own gerrit, or testing it in production. :) | 21:48 |
| jeblair | clarkb: i wasn't really expecting review-dev to work for a while. | 21:48 |
| fungi | it seems like something we could land slightly neutered in production if we wanted and then expand its scope once confirmed working there | 21:49 |
| clarkb | out of curiousity is tehre a general sense that the how to contribute and other new contributor documentation is lacking or that this is the best way to get this particular information in front of people? | 21:49 |
| clarkb | jeblair: gotcha | 21:49 |
| *** dkliban has joined #openstack-infra | 21:49 | |
| fungi | right, testing it on review-dev in parallel with us testing multiple upgrades of gerrit would probably be messy, but also would probably result in testing it against gerrit 2.8 rather than current production version | 21:49 |
| jeblair | clarkb: i personally feel that is not the main benefit of fifieldt's change... | 21:49 |
| *** loq_mac has joined #openstack-infra | 21:50 | |
| jeblair | clarkb: i believe the actual benefit is a side-effect -- reviews are told that this is this person's first patch; behavior adjustments can be made accordingly. | 21:50 |
| clarkb | jeblair: is the benefit having some sort of feedback rather than silence? | 21:50 |
| clarkb | oh interesting | 21:50 |
| jeblair | clarkb: that expections of the author are set, and info is provided, are also benifits. | 21:50 |
| jeblair | i can not spell today. | 21:50 |
| clarkb | I can't spell most days :) | 21:51 |
| clarkb | jeblair: that is a good point that I hadn't considered. I didn't manage to completely follow the ml thraed | 21:51 |
| *** rongze has quit IRC | 21:52 | |
| jeblair | clarkb, sdague: are multi-line elasticsearches possible, or does e-r need to do two queries? https://review.openstack.org/#/c/60508/3 | 21:52 |
| mtreinish | clarkb: so I think it's probably one of these: http://paste.openstack.org/show/54730/ or http://paste.openstack.org/show/54731/ | 21:52 |
| clarkb | jeblair: multiline are not currently possible. mriedem ran into the same problem | 21:52 |
| clarkb | jeblair: we need to teach e-r to do the union of two searches | 21:53 |
| jeblair | clarkb: ack, thx. | 21:53 |
| mriedem | yeah, i'm kind of stuck on my e-r query due to that limitation | 21:53 |
| openstackgerrit | Max Rydahl Andersen proposed a change to openstack-infra/jenkins-job-builder: add private-repository, java-opts, settings and global-settings to maven-target https://review.openstack.org/60161 | 21:53 |
| *** dstanek has quit IRC | 21:54 | |
| *** loq_mac has quit IRC | 21:55 | |
| zaro | do you know what authentication scheme we are using on review-dev? from puppet it looks like opensso, but wasn't real sure. | 21:56 |
| zaro | clarkb: ^ | 21:56 |
| jeblair | zaro: yes | 21:56 |
| *** loq_mac has joined #openstack-infra | 21:56 | |
| *** openstackgerrit has quit IRC | 21:57 | |
| *** openstackgerrit has joined #openstack-infra | 21:57 | |
| mtreinish | jeblair: so that cinder trace I think is caused by a negative test, but I'm not sure | 21:57 |
| *** salv-orlando has joined #openstack-infra | 21:58 | |
| *** sandywalsh has joined #openstack-infra | 21:58 | |
| zaro | fungi: new message from dcramer_ about getting rights to maven central -> https://review.openstack.org/#/c/46099 | 21:59 |
| *** loq_mac has quit IRC | 21:59 | |
| zaro | fungi: i assume you should probably do that with the openstackinfra account. | 21:59 |
| mtreinish | jog0: have you seen http://paste.openstack.org/show/54730/ before? | 21:59 |
| jeblair | mtreinish: i'm not positive, but i think jog0 may be in transit today | 22:01 |
| mtreinish | jeblair: that would explain why he's been quiet today | 22:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: make launchpad integration optional https://review.openstack.org/60952 | 22:01 |
| mtreinish | notmyname: there were swiftclient tracebacks in the glance logs in that pastebin | 22:01 |
| mtreinish | I don't think they were the cause of the failure though | 22:02 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/nodepool: Fix README file https://review.openstack.org/60579 | 22:02 |
| fungi | zaro: i'll have a look after dinner--heading out now but should be back in an hour or so | 22:03 |
| *** emagana has quit IRC | 22:04 | |
| *** markmcclain has quit IRC | 22:04 | |
| *** elasticio has quit IRC | 22:07 | |
| jeblair | sdague: are e-r facets groups of hits with matching values for the specified facet? | 22:07 |
| *** SergeyLukjanov has quit IRC | 22:07 | |
| *** mfer has quit IRC | 22:11 | |
| *** cody-somerville has quit IRC | 22:13 | |
| *** datsun_F40PH has joined #openstack-infra | 22:13 | |
| *** datsun180b has quit IRC | 22:17 | |
| *** datsun_F40PH is now known as datsun180b | 22:17 | |
| *** rongze has joined #openstack-infra | 22:18 | |
| mtreinish | clarkb: you can kill the slave if you want. I copied all the openstack log files off of it. There were lots of traces and errors, but I couldn't find an obvious cause for all of the failures. (like n-cpu hanging up) | 22:18 |
| *** dstanek has joined #openstack-infra | 22:20 | |
| *** dstanek has quit IRC | 22:25 | |
| *** reed has quit IRC | 22:25 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/nodepool: Improve README.rst formatting https://review.openstack.org/60955 | 22:25 |
| sdague | jeblair: yes | 22:26 |
| *** jamesmcarthur has quit IRC | 22:26 | |
| sdague | jeblair: and per previous question, we can not currently do multi line matches | 22:26 |
| sdague | we're going to have to build some more logic into er for that | 22:26 |
| *** yassine has quit IRC | 22:27 | |
| *** cody-somerville has joined #openstack-infra | 22:28 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: log output when installing and upgrading gerrit https://review.openstack.org/60959 | 22:28 |
| jeblair | clarkb: https://review.openstack.org/#/c/60921/ is ready for your consideration | 22:32 |
| *** jergerber has quit IRC | 22:34 | |
| openstackgerrit | Justin Shepherd proposed a change to openstack-infra/config: Adding chef-manager-core to stable maintainers acl https://review.openstack.org/60875 | 22:34 |
| *** vipuls is now known as vipuls-away | 22:35 | |
| *** dangers is now known as danger_fo_away | 22:35 | |
| *** amotoki has quit IRC | 22:36 | |
| *** thomasem has quit IRC | 22:37 | |
| jgriffith | mtreinish: which one? | 22:37 |
| jgriffith | jeblair: ^^ | 22:37 |
| *** dkranz has quit IRC | 22:37 | |
| *** mriedem has quit IRC | 22:37 | |
| mtreinish | jgriffith: at the top of: paste.openstack.org/show/54727/ | 22:38 |
| mtreinish | jgriffith: but I think it's caused by one of these tests: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/volume/test_volumes_negative.py#n181 | 22:38 |
| jgriffith | mtreinish: yeah... but I *thought* I fixed this once already? | 22:39 |
| * jgriffith looking... | 22:39 | |
| jgriffith | nope... but I can fix it now at any rate | 22:41 |
| mtreinish | clarkb: is there a place we can copy the service logs to for the bug that notmyname opened? | 22:41 |
| mtreinish | jgriffith: cool | 22:41 |
| jgriffith | mtreinish: is there a bug already? | 22:41 |
| notmyname | mtreinish: I put a comment on https://bugs.launchpad.net/tempest/+bug/1258682 | 22:42 |
| uvirtbot | Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,New] | 22:42 |
| mtreinish | for that I dunno, I was looking into https://bugs.launchpad.net/tempest/+bug/1258682 and it was in the logs | 22:42 |
| *** paul-- has joined #openstack-infra | 22:43 | |
| jgriffith | mtreinish: k... I'll poke around and file one if not found | 22:43 |
| mtreinish | notmyname: the pastebin wasn't the full logs, I just grepped for TRACE in the logs directory and copied that into pastebin | 22:43 |
| jeblair | mtreinish: you can probably upload some of them to launchpad; i don't know what its size limit is | 22:44 |
| mtreinish | jeblair: ok I'll give that a try | 22:45 |
| *** dprince has quit IRC | 22:46 | |
| *** vipuls-away is now known as vipuls | 22:47 | |
| mtreinish | jeblair: it seems to work. I probably should have gzipped the logs before I uploaded them though, it's kinda slow. | 22:48 |
| jeblair | mtreinish: :) | 22:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 22:50 |
| *** vipuls is now known as vipuls-away | 22:50 | |
| *** rongze has quit IRC | 22:51 | |
| *** JpMaxMan has joined #openstack-infra | 22:51 | |
| *** JpMaxMan has left #openstack-infra | 22:54 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the ephemeral disk on rackspace performance nodes https://review.openstack.org/60347 | 22:55 |
| *** hashar has quit IRC | 22:55 | |
| jeblair | clarkb, fungi: excellent -- the new d-g check jobs worked! https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full/2/console | 22:56 |
| *** vipuls-away is now known as vipuls | 22:58 | |
| *** moted has joined #openstack-infra | 22:58 | |
| *** rossella_s has quit IRC | 22:58 | |
| *** denis_makogon has quit IRC | 22:59 | |
| jgriffith | mtreinish: https://bugs.launchpad.net/cinder/+bug/1259336 | 22:59 |
| uvirtbot | Launchpad bug 1259336 in cinder "Unhandled exception when "None" is passed in as new_size to extend" [Undecided,Triaged] | 22:59 |
| jgriffith | mtreinish: give me 2 minutes for patch | 22:59 |
| *** yamahata_ has quit IRC | 23:00 | |
| mtreinish | jgriffith, the bug fixing machine | 23:01 |
| jgriffith | mtreinish: well.. this is a pretty easy one :) | 23:01 |
| jgriffith | mtreinish: glad I saw ya'll talking about it | 23:02 |
| mtreinish | jgriffith: I'm guessing this is on the error log whitelist too. let me check | 23:02 |
| jgriffith | ahhaa | 23:02 |
| mtreinish | jgriffith: yeah: http://git.openstack.org/cgit/openstack/tempest/tree/etc/whitelist.yaml#n149 | 23:03 |
| jgriffith | ha! | 23:03 |
| jgriffith | well... I have some more to look at there it seems :) | 23:03 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 23:05 |
| *** sarob_ has joined #openstack-infra | 23:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 23:07 |
| *** sarob_ has quit IRC | 23:07 | |
| *** gothicmindfood has quit IRC | 23:08 | |
| *** sarob has quit IRC | 23:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 23:10 |
| *** _david_ has quit IRC | 23:11 | |
| *** MarkAtwood has joined #openstack-infra | 23:11 | |
| *** MarkAtwood has quit IRC | 23:13 | |
| *** dims has quit IRC | 23:14 | |
| clarkb | ok I was AFK back now | 23:15 |
| *** dcramer_ has quit IRC | 23:16 | |
| clarkb | mtreinish: jgriffith did you guys get everything you need from that slave? | 23:16 |
| clarkb | jgriffith: I can give you access to it if you want to do additional triage | 23:16 |
| clarkb | jeblair: woot, looking at the change now | 23:16 |
| jeblair | clarkb: oh, the change is fubar -- but the tests are catching it. :) | 23:16 |
| clarkb | oh gotcha | 23:16 |
| clarkb | success in seeing failure before approval | 23:17 |
| clarkb | not success in configuring disks | 23:17 |
| *** randallburt has quit IRC | 23:17 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 23:17 |
| *** paul-- has quit IRC | 23:19 | |
| jgriffith | clarkb: no need, patch is up | 23:20 |
| clarkb | jeblair: mordred fungi my requirements change to bump django was also caught by the tests | 23:20 |
| jeblair | clarkb: yay! | 23:20 |
| jgriffith | clarkb: pretty sure what the issue is/was | 23:20 |
| jgriffith | of course it'll still fail, but no nasty trace | 23:21 |
| clarkb | jgriffith: great news | 23:21 |
| clarkb | jeblair: yes, I am happy about that. We should prevent such horribleness from getting in again | 23:21 |
| jgriffith | clarkb: if I'm wrong I'll go hide in a hole ;) | 23:21 |
| jeblair | jgriffith: er, just to be clear, do you believe you're fixing the problem that caused the job to timeout, or just reporting that error better? | 23:21 |
| jgriffith | jeblair: ha! That explains it | 23:21 |
| jgriffith | I believe we're talking two different bugs | 23:22 |
| jgriffith | I was just referring to the unhandled exception for extend with no size | 23:22 |
| clarkb | Oh I was referring to whatever made the job run in the swift gate to hang | 23:22 |
| jeblair | clarkb: yes, the exception that jgriffith is fixing was exposed in that, but i don't know that anyone believes it is the cause | 23:23 |
| mtreinish | clarkb: I just scp'd the screen-logs dir off of it, I don't think I'll need anything else from it | 23:23 |
| mtreinish | but I'm not sure what was causing the hangup there were lot's of scary looking tracebacks and errors | 23:24 |
| mtreinish | but most of log messages looked normal and nothing hung | 23:24 |
| clarkb | jeblair: gotcha | 23:24 |
| jgriffith | Is that the same swift one that's been around for a bit? Time out getting the image from glance to cinder? | 23:25 |
| jgriffith | that shows up in the Glance logs as a swift error? | 23:25 |
| mtreinish | jgriffith: this?: http://paste.openstack.org/show/54737/ | 23:26 |
| mtreinish | that was in the logs along with a bunch of glanceclient errors in nova | 23:27 |
| clarkb | jgriffith: I don't think so, I think swift just ran into the problem | 23:27 |
| jgriffith | mtreinish: yeah, that looks like the one I was thinking of | 23:27 |
| clarkb | I don't know if swift is doing anything to break us. that said notmyname has apparently noticed the problem frequently | 23:27 |
| mtreinish | plus 2 nova tracebacks one from libvirt, and one from an rpc | 23:27 |
| jgriffith | clarkb: so I don't mean to imply swift is broken | 23:28 |
| jgriffith | clarkb: but... | 23:28 |
| *** gothicmindfood has joined #openstack-infra | 23:28 | |
| mtreinish | jgriffith: this is the bug notmyname opened for the issue: https://bugs.launchpad.net/nova/+bug/1258682 | 23:28 |
| uvirtbot | Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,New] | 23:28 |
| *** jerryz has joined #openstack-infra | 23:28 | |
| mtreinish | it's kinda sparse on detail though | 23:28 |
| jgriffith | it seems clear that it's a communication between glance and swift IMO | 23:29 |
| notmyname | anyone else have a really hard time getting paste.openstack.org to load? really slow | 23:29 |
| *** dims has joined #openstack-infra | 23:29 | |
| jgriffith | Cinder sent the req to glance succesfully and was evidently received without an issue | 23:29 |
| jgriffith | but then glance puked getting the bits from the object store, or it's auth is jacked or something? | 23:29 |
| *** gothicmindfood has quit IRC | 23:30 | |
| jgriffith | unless I'm not following correctly here | 23:30 |
| clarkb | notmyname: yeah it keeps running into swap. We briefly talked about bumping the node size, should probably talk about that in the meeting tomorrow and just do it | 23:30 |
| jeblair | +justdoit | 23:30 |
| mtreinish | jgriffith: sounds reasonable to me. I haven't had a chance to dive in full yet. I just logged into the node that hit the 1.5hr timeout and did some grepping | 23:31 |
| clarkb | jeblair: should I kick off a node resize now? is that what +justdoit implies? | 23:31 |
| mtreinish | and then copied the service logs off of it | 23:31 |
| fungi | mmmscrollback | 23:32 |
| jeblair | clarkb: that wfm. | 23:32 |
| *** gothicmindfood has joined #openstack-infra | 23:32 | |
| mtreinish | oh, wow I didn't realize it was after 6:30. I'm going home | 23:32 |
| clarkb | fungi: opinions on resizing paste.o.o right now? | 23:32 |
| jgriffith | gotta run... bbl | 23:32 |
| clarkb | mtreinish: :) | 23:32 |
| * mtreinish wishes his office had a window | 23:32 | |
| *** sarob has joined #openstack-infra | 23:33 | |
| *** guohliu has quit IRC | 23:35 | |
| *** prad has quit IRC | 23:36 | |
| *** branen__ has quit IRC | 23:36 | |
| *** branen has joined #openstack-infra | 23:36 | |
| clarkb | fungi: if I don't hear from you shortly I am going to kick off the resize | 23:37 |
| fungi | zaro: i've added the sonatype request for org.openstack.cloud.api to my list of things to do tomorrow | 23:38 |
| mrodden | so, i'm looking into integration tests / deployment tests for the chef cookbooks in stackforge and am wondering if its possible to run virtualbox or LXC on the jenkins job runner nodes | 23:40 |
| *** flaper87 is now known as flaper87|afk | 23:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: log output when installing and upgrading gerrit https://review.openstack.org/60959 | 23:41 |
| *** gothicmindfood has left #openstack-infra | 23:43 | |
| clarkb | mrodden: I think the tl;dr answer to both is yes. The long answer is virtualbox probably won't perform well, but lxc may be worth looking into | 23:43 |
| mrodden | clarkb: i sort of have it working on ubuntu precise | 23:43 |
| mrodden | need to install lxc and vagrant | 23:44 |
| mrodden | but it works... | 23:44 |
| mgagne | zaro: ping | 23:44 |
| fungi | clarkb: caught up, double plus good on node resize for paste.o.o | 23:44 |
| mrodden | downloading the base container probably isn't very good though | 23:44 |
| clarkb | fungi: jeblair: kicking off a resize to 2GB node now | 23:44 |
| *** gothicmindfood has joined #openstack-infra | 23:45 | |
| *** gothicmindfood has quit IRC | 23:45 | |
| fungi | clarkb: also, i figured out a very, very easy resource consumption based denial of service attack against lodgeit a while back. this might help reduce the incidence | 23:45 |
| clarkb | mrodden: I would avoid vagrant... can we not get away with libvirt? | 23:45 |
| clarkb | mrodden: simply because libvirt should be on all of our slaves | 23:45 |
| zaro | mgagne: sup? | 23:46 |
| mgagne | zaro: hi! how well would this plugin play with JJB? https://review.openstack.org/#/c/60674/ | 23:46 |
| mrodden | vagrant is mainly for the ruby interfacing with lxc | 23:46 |
| *** rcleere_ has quit IRC | 23:46 | |
| mgagne | zaro: especially the hidden HTML comment we add to deal with no longer managed jobs | 23:46 |
| mrodden | most of the other test kitchen plugins seem to be pretty poorly maintained | 23:46 |
| mrodden | i haven't looked to see if there was a libvirt one | 23:46 |
| clarkb | fungi: the longer we run lodgeit the more i think we should point people at third party pastes... I hate saying it but they are simple services too easy to abuse | 23:47 |
| * zaro looking | 23:48 | |
| clarkb | many people already use ubuntu/fedora/gist and so on. But removing services seems blah | 23:48 |
| zaro | mgagne: we probably won't use. | 23:48 |
| mordred | ola | 23:49 |
| * mordred is in barcelona now | 23:49 | |
| fungi | someone collapsed the mordred wave function | 23:50 |
| *** fbo is now known as fbo_away | 23:50 | |
| *** rongze has joined #openstack-infra | 23:51 | |
| * mordred collapsed | 23:52 | |
| clarkb | mordred: fungi: https://review.openstack.org/#/c/60923/ it failed :) | 23:53 |
| bknudson | I thought barcelona was covered in trash? | 23:54 |
| fungi | clarkb: hooray for intentional failure! | 23:54 |
| mordred | clarkb: yay! | 23:54 |
| mordred | bknudson: is it? | 23:54 |
| bknudson | http://www.freerepublic.com/focus/f-news/3091750/posts -- maybe it's over by now | 23:54 |
| bknudson | oh, that's madrid! | 23:54 |
| fungi | bknudson: it's covered in catalonians. totally different | 23:54 |
| zaro | fungi: log output merge from about. do you know where the output goes to? | 23:55 |
| zaro | sp about/above | 23:55 |
| bknudson | http://www.zuzafun.com/el-prat-airport-barcelona-covered-in-garbage | 23:55 |
| bknudson | I think you would notice if it looked like that. | 23:56 |
| mordred | bknudson: weird. I did not experience that | 23:56 |
| fungi | zaro: i'm assuming i'm going to find it in the logs the next time puppet attempts to upgrade gerrit. first i need to stop puppet and put things back the way they were | 23:56 |
| *** samuelbercovici has joined #openstack-infra | 23:56 | |
| zaro | fungi: ok. just wondering is all. | 23:57 |
| *** samuelbercovici has left #openstack-infra | 23:57 | |
| fungi | zaro: in the future, when there's a working puppet(dash)board, i expect its output to be probably visible there from the puppet report | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!