| *** openstack has joined #openstack-infra | 00:03 | |
| *** yassine has quit IRC | 00:04 | |
| *** oubiwan__ has quit IRC | 00:05 | |
| *** matsuhashi has joined #openstack-infra | 00:09 | |
| *** Hunner has quit IRC | 00:12 | |
| *** Hunner has joined #openstack-infra | 00:20 | |
| *** milki_ is now known as milki | 00:23 | |
| *** jcooley_ has quit IRC | 00:31 | |
| *** fifieldt has joined #openstack-infra | 00:32 | |
| *** gokrokve has joined #openstack-infra | 00:34 | |
| *** gokrokve has quit IRC | 00:39 | |
| morganfainberg | lifeless, fungi, mordred, clarkb, https://review.openstack.org/#/c/75284 this review seems wedged somehow. it's not hitting zuul, do i need to un-approve it then recheck? | 00:45 |
|---|---|---|
| morganfainberg | it looks like the verify job was completely lost at one point...and never requeued | 00:45 |
| morganfainberg | s/verify/gate | 00:45 |
| *** mriedem has joined #openstack-infra | 00:46 | |
| *** davidhadas_ has joined #openstack-infra | 00:46 | |
| morganfainberg | i can do a new patchset that changes something to force it back through check if needed. | 00:46 |
| morganfainberg | (trivial) | 00:46 |
| morganfainberg | but not sure if you guys want to look at it first | 00:47 |
| lifeless | I would recheck no bug; not that that might be case sensitive :) | 00:47 |
| morganfainberg | lifeless, lol let me try it w/ case corrected | 00:47 |
| morganfainberg | lifeless, but if thats not working i'll goahead and do a trivial change. | 00:48 |
| *** davidlenwell_ has joined #openstack-infra | 00:48 | |
| morganfainberg | lifeless *grumble* typo in casesensitivity :P | 00:48 |
| *** juice- has joined #openstack-infra | 00:48 | |
| morganfainberg | lifeless, thanks | 00:48 |
| morganfainberg | jenkins seems to be chewing on it again | 00:49 |
| *** fifieldt has quit IRC | 00:49 | |
| *** flaper87 has quit IRC | 00:49 | |
| *** bodepd_ has quit IRC | 00:49 | |
| *** plomakin_ has quit IRC | 00:49 | |
| *** andre__ has quit IRC | 00:49 | |
| *** dosaboy has quit IRC | 00:49 | |
| *** davidhadas has quit IRC | 00:49 | |
| *** dachary has quit IRC | 00:49 | |
| *** ogelbukh has quit IRC | 00:49 | |
| *** bookwar has quit IRC | 00:49 | |
| *** lyxus has quit IRC | 00:49 | |
| *** vipuls has quit IRC | 00:49 | |
| *** juice has quit IRC | 00:49 | |
| *** jang1 has quit IRC | 00:49 | |
| *** davidlenwell has quit IRC | 00:49 | |
| *** juice- is now known as juice | 00:51 | |
| *** fifieldt has joined #openstack-infra | 00:52 | |
| *** flaper87 has joined #openstack-infra | 00:52 | |
| *** bodepd_ has joined #openstack-infra | 00:52 | |
| *** plomakin_ has joined #openstack-infra | 00:52 | |
| *** andre__ has joined #openstack-infra | 00:52 | |
| *** dosaboy has joined #openstack-infra | 00:52 | |
| *** dachary has joined #openstack-infra | 00:52 | |
| *** ogelbukh has joined #openstack-infra | 00:52 | |
| *** bookwar has joined #openstack-infra | 00:52 | |
| *** lyxus has joined #openstack-infra | 00:52 | |
| *** vipuls has joined #openstack-infra | 00:52 | |
| *** jang1 has joined #openstack-infra | 00:52 | |
| *** dolphm_503 is now known as dolphm | 00:52 | |
| *** alexpilotti has quit IRC | 00:54 | |
| *** oubiwan__ has joined #openstack-infra | 01:01 | |
| *** oubiwan__ has quit IRC | 01:06 | |
| *** nosnos has joined #openstack-infra | 01:07 | |
| *** sdake_ has joined #openstack-infra | 01:15 | |
| *** sdake_ has quit IRC | 01:23 | |
| *** zhiwei has joined #openstack-infra | 01:25 | |
| *** Guest633 has quit IRC | 01:30 | |
| zhiwei | clarkb: hi | 01:33 |
| *** gokrokve has joined #openstack-infra | 01:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Rename Openstack to OpenStack https://review.openstack.org/72900 | 01:34 |
| *** gokrokve has quit IRC | 01:38 | |
| *** nati_ueno has joined #openstack-infra | 01:42 | |
| *** thomasbiege has joined #openstack-infra | 01:45 | |
| *** nati_ueno has quit IRC | 01:47 | |
| *** unicell has joined #openstack-infra | 01:48 | |
| *** morganfainberg is now known as morganfainberg_Z | 01:56 | |
| *** morganfainberg_Z is now known as morganfainberg | 01:57 | |
| *** oubiwan__ has joined #openstack-infra | 02:02 | |
| *** oubiwan__ has quit IRC | 02:07 | |
| *** masayukig has joined #openstack-infra | 02:08 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Install libffi-dev needed for python-glanceclient https://review.openstack.org/75738 | 02:10 |
| *** yamahata has joined #openstack-infra | 02:10 | |
| lifeless | clarkb: around ? ^ is that sensible? Or does jenkins have sudo access? | 02:11 |
| Alex_Gaynor | lifeless: did that come up via pyOpenSSL? | 02:11 |
| lifeless | Alex_Gaynor: probably - https://jenkins01.openstack.org/job/check-tripleo-seed-precise/59/console | 02:11 |
| lifeless | Alex_Gaynor: | 02:11 |
| lifeless | 2014-02-24 01:17:25.540 | Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->python-glanceclient) | 02:11 |
| Alex_Gaynor | Looks like yeah. | 02:11 |
| lifeless | StevenK: ah, we do have sudo | 02:12 |
| StevenK | Ah, nice. | 02:13 |
| lifeless | local tripleo-ci change proposed | 02:16 |
| lifeless | that should fail in the image | 02:16 |
| lifeless | if it does then we can land it and go to the image one and recheck | 02:16 |
| *** yaguang has joined #openstack-infra | 02:20 | |
| *** CaptTofu has quit IRC | 02:26 | |
| openstackgerrit | Zang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP https://review.openstack.org/67858 | 02:30 |
| *** Ryan_Lane1 has joined #openstack-infra | 02:31 | |
| *** gokrokve has joined #openstack-infra | 02:34 | |
| *** thomasbiege has quit IRC | 02:34 | |
| *** vkozhukalov has joined #openstack-infra | 02:35 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 02:38 |
| *** gokrokve has quit IRC | 02:39 | |
| *** sarob has quit IRC | 02:49 | |
| *** sarob has joined #openstack-infra | 02:50 | |
| *** CaptTofu has joined #openstack-infra | 02:53 | |
| *** sarob has quit IRC | 02:54 | |
| *** dolphm is now known as dolphm_503 | 02:57 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Add oslo.vmware https://review.openstack.org/75555 | 02:58 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Rename slave_template to single_use_slave https://review.openstack.org/71450 | 02:58 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Support filtering by review id(s) https://review.openstack.org/72446 | 02:58 |
| *** dolphm_503 is now known as dolphm | 02:59 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 03:00 |
| *** pcrews has joined #openstack-infra | 03:01 | |
| *** oubiwan__ has joined #openstack-infra | 03:03 | |
| *** davidhadas has joined #openstack-infra | 03:05 | |
| *** davidhadas_ has quit IRC | 03:08 | |
| *** oubiwan__ has quit IRC | 03:08 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Start compressing config files too https://review.openstack.org/74838 | 03:10 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add support for running tempest serially without tenant isolation https://review.openstack.org/72365 | 03:11 |
| *** dolphm is now known as dolphm_503 | 03:14 | |
| *** pcrews has quit IRC | 03:15 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Support grenade-svm-rolling rename to *-partial-ncpu https://review.openstack.org/73460 | 03:17 |
| *** sarob has joined #openstack-infra | 03:17 | |
| *** coolsvap has joined #openstack-infra | 03:20 | |
| *** CaptTofu has quit IRC | 03:22 | |
| *** dolphm_503 is now known as dolphm | 03:23 | |
| *** mwagner_lap has joined #openstack-infra | 03:26 | |
| *** SpamapS_ is now known as SpamapS | 03:30 | |
| *** SpamapS has quit IRC | 03:30 | |
| *** SpamapS has joined #openstack-infra | 03:30 | |
| *** pmathews has joined #openstack-infra | 03:33 | |
| *** talluri has joined #openstack-infra | 03:33 | |
| *** gokrokve has joined #openstack-infra | 03:34 | |
| *** pballand has joined #openstack-infra | 03:35 | |
| *** thomasbiege has joined #openstack-infra | 03:37 | |
| *** matsuhashi has quit IRC | 03:38 | |
| *** gokrokve has quit IRC | 03:39 | |
| *** dstanek has joined #openstack-infra | 03:40 | |
| *** david_lyle_ has joined #openstack-infra | 03:40 | |
| *** david_lyle_ is now known as david_lyle | 03:43 | |
| *** pballand has quit IRC | 03:49 | |
| *** pballand has joined #openstack-infra | 03:50 | |
| *** pballand has quit IRC | 03:54 | |
| *** miqui_ has quit IRC | 04:01 | |
| *** mriedem has quit IRC | 04:02 | |
| *** talluri has quit IRC | 04:02 | |
| *** lnxnut has joined #openstack-infra | 04:05 | |
| *** Hunner has quit IRC | 04:06 | |
| *** Hunner has joined #openstack-infra | 04:12 | |
| *** nati_ueno has joined #openstack-infra | 04:14 | |
| *** matsuhashi has joined #openstack-infra | 04:22 | |
| *** nati_ueno has quit IRC | 04:23 | |
| *** thomasbiege has quit IRC | 04:29 | |
| *** sirushti is now known as shortstop | 04:45 | |
| *** sarob has quit IRC | 04:49 | |
| *** zhiyan_ is now known as zhiyan | 04:49 | |
| *** sarob has joined #openstack-infra | 04:50 | |
| *** afazekas has joined #openstack-infra | 04:50 | |
| *** sarob_ has joined #openstack-infra | 04:51 | |
| *** sarob has quit IRC | 04:54 | |
| *** sarob_ has quit IRC | 05:00 | |
| *** amotoki has joined #openstack-infra | 05:01 | |
| *** chandan_kumar has joined #openstack-infra | 05:08 | |
| *** sarob has joined #openstack-infra | 05:09 | |
| *** chandan_kumar has quit IRC | 05:12 | |
| *** matsuhas_ has joined #openstack-infra | 05:14 | |
| *** lnxnut has quit IRC | 05:14 | |
| *** matsuhashi has quit IRC | 05:16 | |
| *** lnxnut has joined #openstack-infra | 05:22 | |
| *** CaptTofu has joined #openstack-infra | 05:23 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 05:24 |
| *** lnxnut has quit IRC | 05:26 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 05:26 |
| *** matsuhashi has joined #openstack-infra | 05:27 | |
| *** CaptTofu has quit IRC | 05:27 | |
| *** matsuhas_ has quit IRC | 05:30 | |
| *** nicedice_ has quit IRC | 05:30 | |
| openstackgerrit | mark mcclain proposed a change to openstack-infra/reviewstats: Add Oleg Bondarev to Neutron core https://review.openstack.org/75764 | 05:33 |
| *** matsuhas_ has joined #openstack-infra | 05:33 | |
| *** matsuhashi has quit IRC | 05:36 | |
| *** dolphm is now known as dolphm_503 | 05:43 | |
| *** nosnos_ has joined #openstack-infra | 05:43 | |
| *** nosnos has quit IRC | 05:43 | |
| *** jcooley_ has joined #openstack-infra | 05:44 | |
| *** jcooley_ has quit IRC | 05:47 | |
| *** jcooley_ has joined #openstack-infra | 05:49 | |
| *** jcooley_ has quit IRC | 05:49 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: remove flaky gerrit tests https://review.openstack.org/75769 | 05:53 |
| *** kashyap has joined #openstack-infra | 05:55 | |
| *** chandan_kumar has joined #openstack-infra | 05:55 | |
| *** matsuhas_ has quit IRC | 06:00 | |
| *** matsuhashi has joined #openstack-infra | 06:01 | |
| *** mrda is now known as mrda_away | 06:03 | |
| *** lnxnut has joined #openstack-infra | 06:05 | |
| *** matsuhas_ has joined #openstack-infra | 06:05 | |
| *** matsuhashi has quit IRC | 06:05 | |
| *** lnxnut has quit IRC | 06:06 | |
| *** lnxnut has joined #openstack-infra | 06:07 | |
| *** lnxnut has quit IRC | 06:11 | |
| *** sarob has quit IRC | 06:17 | |
| *** pmathews has quit IRC | 06:17 | |
| *** sarob has joined #openstack-infra | 06:18 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 06:21 |
| *** matsuhas_ has quit IRC | 06:21 | |
| *** matsuhashi has joined #openstack-infra | 06:22 | |
| *** matsuhashi has quit IRC | 06:22 | |
| *** sarob has quit IRC | 06:22 | |
| *** matsuhashi has joined #openstack-infra | 06:23 | |
| *** jcooley_ has joined #openstack-infra | 06:25 | |
| *** jishaom has joined #openstack-infra | 06:28 | |
| *** jcoufal has joined #openstack-infra | 06:30 | |
| *** nosnos_ has quit IRC | 06:32 | |
| *** nosnos has joined #openstack-infra | 06:32 | |
| *** matsuhashi has quit IRC | 06:33 | |
| *** ArxCruz has quit IRC | 06:36 | |
| *** matsuhashi has joined #openstack-infra | 06:38 | |
| *** jcooley_ has quit IRC | 06:42 | |
| *** dolphm_503 is now known as dolphm | 06:44 | |
| *** pblaho has joined #openstack-infra | 06:49 | |
| *** vkozhukalov has quit IRC | 06:51 | |
| *** gokrokve has joined #openstack-infra | 06:51 | |
| *** gokrokve_ has joined #openstack-infra | 06:52 | |
| *** dolphm is now known as dolphm_503 | 06:53 | |
| *** gokrokve has quit IRC | 06:55 | |
| *** saju_m has joined #openstack-infra | 06:58 | |
| *** Hunner has quit IRC | 07:01 | |
| *** Hunner has joined #openstack-infra | 07:02 | |
| *** nati_ueno has joined #openstack-infra | 07:03 | |
| tchaypo | I'm making a change to openstack_project::slave and want to test it on both precise and fedora19 to make sure it works on both supported platforms, so I find myself writing a multi-box vagrant file. It's not a lot of work, but I'm wondering if someone has done something like this for testing changes already.. | 07:04 |
| *** lnxnut has joined #openstack-infra | 07:05 | |
| *** lnxnut has quit IRC | 07:06 | |
| *** lnxnut has joined #openstack-infra | 07:06 | |
| *** nati_ueno has quit IRC | 07:10 | |
| *** lnxnut has quit IRC | 07:11 | |
| *** gokrokve_ has quit IRC | 07:12 | |
| *** CaptTofu has joined #openstack-infra | 07:24 | |
| *** CaptTofu has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-infra | 07:31 | |
| *** DinaBelova has joined #openstack-infra | 07:34 | |
| *** nati_ueno has joined #openstack-infra | 07:36 | |
| *** davidhadas has quit IRC | 07:38 | |
| *** e0ne_ has joined #openstack-infra | 07:41 | |
| jishaom | I'm testing my jobs using zuul+gearman, but I meet an error, error info "zuul.Scheduler: Run handler sleeping ,DEBUG zuul.Gearman: Looking for lost builds". It always shows 'Looking for lost builds' and zuul can't work. Did anyone meet this error and how to fix it? | 07:42 |
| *** flaper87 has quit IRC | 07:42 | |
| *** flaper87 has joined #openstack-infra | 07:42 | |
| *** e0ne has quit IRC | 07:43 | |
| *** luqas has joined #openstack-infra | 07:46 | |
| *** luqas has quit IRC | 07:49 | |
| *** matsuhashi has quit IRC | 07:52 | |
| *** matsuhashi has joined #openstack-infra | 07:52 | |
| *** gokrokve has joined #openstack-infra | 07:53 | |
| *** jamielennox is now known as jamielennox|away | 07:54 | |
| *** dstanek_afk has joined #openstack-infra | 07:54 | |
| *** plomakin has joined #openstack-infra | 07:58 | |
| *** bodepd has joined #openstack-infra | 07:59 | |
| openstackgerrit | james Polley proposed a change to openstack-infra/config: Make sure gawk is installed https://review.openstack.org/75785 | 07:59 |
| *** jang has joined #openstack-infra | 07:59 | |
| *** fesp has joined #openstack-infra | 08:00 | |
| *** dizquierdo has joined #openstack-infra | 08:00 | |
| *** dachary_ has joined #openstack-infra | 08:02 | |
| *** vipul has joined #openstack-infra | 08:02 | |
| *** dstanek has quit IRC | 08:03 | |
| *** fifieldt has quit IRC | 08:03 | |
| *** flaper87 has quit IRC | 08:03 | |
| *** bodepd_ has quit IRC | 08:03 | |
| *** plomakin_ has quit IRC | 08:03 | |
| *** andre__ has quit IRC | 08:03 | |
| *** dosaboy has quit IRC | 08:03 | |
| *** dachary has quit IRC | 08:03 | |
| *** ogelbukh has quit IRC | 08:03 | |
| *** bookwar has quit IRC | 08:03 | |
| *** lyxus has quit IRC | 08:03 | |
| *** vipuls has quit IRC | 08:03 | |
| *** jang1 has quit IRC | 08:03 | |
| *** dachary_ is now known as dachary | 08:04 | |
| *** fesp is now known as flaper87 | 08:04 | |
| *** dosaboy has joined #openstack-infra | 08:04 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:06 | |
| *** rlandy has joined #openstack-infra | 08:06 | |
| *** lnxnut has joined #openstack-infra | 08:06 | |
| *** oubiwan__ has joined #openstack-infra | 08:08 | |
| *** gcb has joined #openstack-infra | 08:08 | |
| *** lyxus has joined #openstack-infra | 08:08 | |
| *** andre__ has joined #openstack-infra | 08:09 | |
| *** andre__ has quit IRC | 08:09 | |
| *** oubiwan__ has quit IRC | 08:09 | |
| *** ogelbukh has joined #openstack-infra | 08:10 | |
| *** lnxnut has quit IRC | 08:11 | |
| *** matrohon has joined #openstack-infra | 08:12 | |
| *** bookwar has joined #openstack-infra | 08:15 | |
| *** fifieldt has joined #openstack-infra | 08:15 | |
| *** andre__ has joined #openstack-infra | 08:15 | |
| *** oubiwan__ has joined #openstack-infra | 08:15 | |
| *** oubiwan__ has quit IRC | 08:16 | |
| *** luqas has joined #openstack-infra | 08:17 | |
| *** sarob has joined #openstack-infra | 08:19 | |
| *** jcooley_ has joined #openstack-infra | 08:21 | |
| *** jgallard has joined #openstack-infra | 08:23 | |
| *** sarob has quit IRC | 08:23 | |
| *** jcooley_ has quit IRC | 08:28 | |
| *** ociuhandu has joined #openstack-infra | 08:37 | |
| *** vkozhukalov has joined #openstack-infra | 08:37 | |
| *** garyk has joined #openstack-infra | 08:37 | |
| *** boris-42_ has quit IRC | 08:37 | |
| *** hashar has joined #openstack-infra | 08:40 | |
| *** mattymo has joined #openstack-infra | 08:42 | |
| *** rossella-s has joined #openstack-infra | 08:42 | |
| *** DinaBelova is now known as DinaBelova_ | 08:43 | |
| *** davidhadas has joined #openstack-infra | 08:45 | |
| *** davidhadas_ has joined #openstack-infra | 08:46 | |
| *** davidhadas__ has joined #openstack-infra | 08:46 | |
| *** persia has joined #openstack-infra | 08:50 | |
| *** persia is now known as Guest88271 | 08:51 | |
| *** sahid has joined #openstack-infra | 08:51 | |
| *** Guest88271 has quit IRC | 08:51 | |
| *** Guest88271 has joined #openstack-infra | 08:51 | |
| *** Guest88271 is now known as persia | 08:51 | |
| *** tteggel_ is now known as tteggel | 08:53 | |
| *** dizquierdo has quit IRC | 08:54 | |
| *** nati_ueno has quit IRC | 08:57 | |
| *** yassine has joined #openstack-infra | 09:01 | |
| *** yassine_ has joined #openstack-infra | 09:01 | |
| *** nati_ueno has joined #openstack-infra | 09:04 | |
| *** fbo_away is now known as fbo | 09:04 | |
| *** lnxnut has joined #openstack-infra | 09:05 | |
| *** zhiwei has quit IRC | 09:06 | |
| *** Guest86727 is now known as bobba | 09:09 | |
| *** bobba is now known as BobBall | 09:10 | |
| *** lnxnut has quit IRC | 09:10 | |
| *** derekh has joined #openstack-infra | 09:17 | |
| *** sarob has joined #openstack-infra | 09:20 | |
| *** CaptTofu has joined #openstack-infra | 09:25 | |
| *** sarob has quit IRC | 09:26 | |
| *** CaptTofu has quit IRC | 09:29 | |
| *** gcb has quit IRC | 09:31 | |
| *** zhiwei has joined #openstack-infra | 09:34 | |
| *** matel has quit IRC | 09:35 | |
| *** luqas has quit IRC | 09:35 | |
| *** nati_ueno has quit IRC | 09:36 | |
| *** afrittoli has joined #openstack-infra | 09:37 | |
| *** afrittoli has quit IRC | 09:38 | |
| *** andreaf has joined #openstack-infra | 09:38 | |
| *** johnthetubaguy has joined #openstack-infra | 09:39 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:41 | |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-devstack job https://review.openstack.org/75078 | 09:42 |
| *** johnthetubaguy has quit IRC | 09:42 | |
| ihrachys | hub_cap: ping | 09:43 |
| *** nati_ueno has joined #openstack-infra | 09:43 | |
| *** saju_m has quit IRC | 09:46 | |
| *** e0ne_ has quit IRC | 09:50 | |
| *** saju_m has joined #openstack-infra | 09:50 | |
| *** e0ne has joined #openstack-infra | 09:51 | |
| *** gcb has joined #openstack-infra | 09:54 | |
| openstackgerrit | ChangBo Guo(gcb) proposed a change to openstack/requirements: Remove vim header https://review.openstack.org/75821 | 09:56 |
| *** jp_at_hp has joined #openstack-infra | 09:58 | |
| *** mkerrin has joined #openstack-infra | 10:02 | |
| *** lnxnut has joined #openstack-infra | 10:05 | |
| *** luqas has joined #openstack-infra | 10:05 | |
| *** rlandy has quit IRC | 10:08 | |
| *** lnxnut has quit IRC | 10:10 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 10:10 |
| *** protux has joined #openstack-infra | 10:12 | |
| *** rlandy has joined #openstack-infra | 10:19 | |
| *** jhesketh has quit IRC | 10:20 | |
| *** ildikov_ has joined #openstack-infra | 10:21 | |
| *** sarob has joined #openstack-infra | 10:23 | |
| *** jhesketh_ has quit IRC | 10:24 | |
| *** e0ne_ has joined #openstack-infra | 10:26 | |
| *** masayukig has quit IRC | 10:27 | |
| *** jishaom has quit IRC | 10:27 | |
| *** matsuhashi has quit IRC | 10:27 | |
| *** sarob has quit IRC | 10:27 | |
| *** CaptTofu has joined #openstack-infra | 10:28 | |
| *** e0ne has quit IRC | 10:29 | |
| *** markmc has joined #openstack-infra | 10:29 | |
| *** dizquierdo has joined #openstack-infra | 10:35 | |
| *** talluri has joined #openstack-infra | 10:43 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 10:43 |
| *** nati_ueno has quit IRC | 10:49 | |
| *** lcostantino has joined #openstack-infra | 10:50 | |
| *** alexpilotti has joined #openstack-infra | 10:51 | |
| *** jlibosva has joined #openstack-infra | 10:52 | |
| *** zenoway has joined #openstack-infra | 10:53 | |
| *** alexpilotti has quit IRC | 10:57 | |
| *** matsuhashi has joined #openstack-infra | 10:59 | |
| *** homeless_ has joined #openstack-infra | 11:00 | |
| *** cyeoh has quit IRC | 11:01 | |
| *** markmc has quit IRC | 11:01 | |
| *** homeless has quit IRC | 11:02 | |
| *** markmc has joined #openstack-infra | 11:02 | |
| *** vipul has quit IRC | 11:03 | |
| *** vipul has joined #openstack-infra | 11:04 | |
| *** cyeoh has joined #openstack-infra | 11:05 | |
| *** yamahata has quit IRC | 11:07 | |
| *** vkozhukalov has quit IRC | 11:12 | |
| *** saju_m has quit IRC | 11:12 | |
| *** bodepd_ has joined #openstack-infra | 11:12 | |
| *** vkozhukalov_ has joined #openstack-infra | 11:12 | |
| *** jcooley_ has joined #openstack-infra | 11:12 | |
| *** matsuhashi has quit IRC | 11:13 | |
| *** SpamapS_ has joined #openstack-infra | 11:14 | |
| *** zigo_ is now known as zigo | 11:14 | |
| *** plomakin_ has joined #openstack-infra | 11:14 | |
| *** matsuhashi has joined #openstack-infra | 11:14 | |
| *** jcooley_ has quit IRC | 11:14 | |
| *** ociuhandu has quit IRC | 11:14 | |
| *** nibalizer has joined #openstack-infra | 11:14 | |
| *** jaypipes has quit IRC | 11:14 | |
| *** jp_at_hp has quit IRC | 11:14 | |
| *** andreaf has quit IRC | 11:15 | |
| *** coolsvap has quit IRC | 11:15 | |
| *** matrohon has quit IRC | 11:16 | |
| *** SpamapS has quit IRC | 11:17 | |
| *** plomakin has quit IRC | 11:17 | |
| *** bodepd has quit IRC | 11:17 | |
| *** julienvey has quit IRC | 11:17 | |
| *** dhellmann has quit IRC | 11:17 | |
| *** jp_at_hp1 has joined #openstack-infra | 11:17 | |
| *** andreaf has joined #openstack-infra | 11:17 | |
| *** samalba has quit IRC | 11:19 | |
| *** max_lobur has quit IRC | 11:19 | |
| *** jaypipes has joined #openstack-infra | 11:20 | |
| *** nibalize1 has quit IRC | 11:20 | |
| *** coolsvap has joined #openstack-infra | 11:20 | |
| *** jp_at_hp1 has quit IRC | 11:20 | |
| *** jp_at_hp1 has joined #openstack-infra | 11:20 | |
| *** andreaf has quit IRC | 11:20 | |
| *** andreaf has joined #openstack-infra | 11:20 | |
| *** dhellmann has joined #openstack-infra | 11:20 | |
| *** samalba has joined #openstack-infra | 11:20 | |
| *** sarob has joined #openstack-infra | 11:23 | |
| *** fifieldt has quit IRC | 11:25 | |
| *** matrohon has joined #openstack-infra | 11:26 | |
| *** saju_m has joined #openstack-infra | 11:27 | |
| *** sarob has quit IRC | 11:31 | |
| *** dizquierdo has quit IRC | 11:35 | |
| *** nosnos has quit IRC | 11:41 | |
| *** coolsvap has quit IRC | 11:45 | |
| *** dizquierdo has joined #openstack-infra | 11:48 | |
| *** alexpilotti has joined #openstack-infra | 11:48 | |
| *** ociuhandu has joined #openstack-infra | 11:51 | |
| *** yaguang has quit IRC | 11:54 | |
| *** zenoway has quit IRC | 11:55 | |
| *** rfolco has joined #openstack-infra | 11:56 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 11:58 |
| *** jcooley_ has joined #openstack-infra | 11:59 | |
| *** lcostantino has quit IRC | 12:01 | |
| *** lcostantino has joined #openstack-infra | 12:02 | |
| *** saju_m has quit IRC | 12:02 | |
| *** jcooley_ has quit IRC | 12:04 | |
| *** boris-42_ has joined #openstack-infra | 12:05 | |
| *** mattymo has quit IRC | 12:06 | |
| *** mattymo has joined #openstack-infra | 12:06 | |
| *** lcostantino has quit IRC | 12:06 | |
| *** lcostantino has joined #openstack-infra | 12:07 | |
| *** CaptTofu has quit IRC | 12:07 | |
| *** lcostantino has quit IRC | 12:08 | |
| *** DinaBelova_ is now known as DinaBelova | 12:10 | |
| *** matsuhashi has quit IRC | 12:14 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 12:15 |
| *** ArxCruz has joined #openstack-infra | 12:17 | |
| *** zhiwei has quit IRC | 12:18 | |
| *** dolphm_503 is now known as dolphm | 12:20 | |
| *** lcostantino has joined #openstack-infra | 12:24 | |
| *** zhiwei has joined #openstack-infra | 12:27 | |
| *** eharney has joined #openstack-infra | 12:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 12:31 |
| openstackgerrit | Alexander Jones proposed a change to openstack-infra/git-review: Fix parsing of SCP-style URLs, as these are valid in Git itself https://review.openstack.org/72751 | 12:33 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Improved timestamp parsing https://review.openstack.org/73741 | 12:34 |
| *** CaptTofu has joined #openstack-infra | 12:35 | |
| *** luqas has quit IRC | 12:38 | |
| *** zhiwei has quit IRC | 12:38 | |
| *** talluri has quit IRC | 12:39 | |
| *** weshay has joined #openstack-infra | 12:39 | |
| *** talluri has joined #openstack-infra | 12:40 | |
| *** hashar has quit IRC | 12:40 | |
| *** heyongli has joined #openstack-infra | 12:41 | |
| *** talluri has quit IRC | 12:44 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 12:45 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 12:45 |
| *** dolphm is now known as dolphm_503 | 12:45 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Add cryptography needed by python-openstackclient https://review.openstack.org/75857 | 12:47 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 12:48 |
| *** Adri2000_ is now known as Adri2000 | 12:49 | |
| ihrachys | fungi: ping | 12:52 |
| ihrachys | fungi: regarding unmaintained stable/havana trove branch and its removal | 12:53 |
| *** che-arne has joined #openstack-infra | 12:54 | |
| *** smarcet has joined #openstack-infra | 12:54 | |
| sdague | hmmmm - http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9zdGF0dXM6U1VDQ0VTUyBBTkQgbG9nbGV2ZWw6RVJST1IiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzI0NjM5MTMxMH0= | 12:55 |
| *** saju_m has joined #openstack-infra | 12:55 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 12:55 |
| sdague | 500k ERROR log lines on success builds in last 24 hrs | 12:55 |
| *** jcooley_ has joined #openstack-infra | 12:57 | |
| *** lcostantino has quit IRC | 12:57 | |
| *** dims has joined #openstack-infra | 12:58 | |
| *** ArxCruz has quit IRC | 12:59 | |
| *** dkranz has quit IRC | 13:01 | |
| *** sandywalsh has quit IRC | 13:02 | |
| *** ArxCruz has joined #openstack-infra | 13:04 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 13:05 |
| *** jcooley_ has quit IRC | 13:05 | |
| *** CaptTofu has quit IRC | 13:10 | |
| *** heyongli has quit IRC | 13:11 | |
| *** gcb has quit IRC | 13:12 | |
| *** lcostantino has joined #openstack-infra | 13:14 | |
| *** dolphm_503 is now known as dolphm | 13:14 | |
| *** david_lyle has quit IRC | 13:14 | |
| *** hashar has joined #openstack-infra | 13:14 | |
| *** sandywalsh has joined #openstack-infra | 13:14 | |
| *** pdmars has joined #openstack-infra | 13:14 | |
| *** zul has quit IRC | 13:26 | |
| *** zul has joined #openstack-infra | 13:27 | |
| *** dims has quit IRC | 13:28 | |
| *** thomasbiege has joined #openstack-infra | 13:31 | |
| *** lcostantino has quit IRC | 13:31 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 13:31 | |
| *** enikanorov_ has joined #openstack-infra | 13:31 | |
| *** locke105 has joined #openstack-infra | 13:32 | |
| *** bnemec_ has joined #openstack-infra | 13:33 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:33 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 13:35 |
| *** jpeeler has joined #openstack-infra | 13:35 | |
| *** jpeeler has quit IRC | 13:35 | |
| *** jpeeler has joined #openstack-infra | 13:35 | |
| *** fbo is now known as fbo_away | 13:36 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 13:37 |
| *** zhiyan- has joined #openstack-infra | 13:38 | |
| *** wayneseguin has joined #openstack-infra | 13:38 | |
| *** jd__` has joined #openstack-infra | 13:38 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-dev/pbr: Declare support for Python versions in setup.cfg https://review.openstack.org/75868 | 13:39 |
| *** rahmu_ has joined #openstack-infra | 13:39 | |
| *** StevenK_ has joined #openstack-infra | 13:39 | |
| *** sbadia_ has joined #openstack-infra | 13:39 | |
| *** enikanorov has quit IRC | 13:40 | |
| *** EntropyWorks has quit IRC | 13:40 | |
| *** wayneeseguin has quit IRC | 13:40 | |
| *** jd__ has quit IRC | 13:40 | |
| *** changbl has quit IRC | 13:40 | |
| *** jasond`` has quit IRC | 13:40 | |
| *** bnemec has quit IRC | 13:40 | |
| *** sbadia has quit IRC | 13:40 | |
| *** locke1051 has quit IRC | 13:40 | |
| *** dansmith has quit IRC | 13:40 | |
| *** zhiyan has quit IRC | 13:40 | |
| *** pabelanger has quit IRC | 13:40 | |
| *** stevebaker has quit IRC | 13:40 | |
| *** greghaynes has quit IRC | 13:40 | |
| *** mancdaz has quit IRC | 13:40 | |
| *** rahmu has quit IRC | 13:40 | |
| *** mrda_away has quit IRC | 13:40 | |
| *** wayneseguin is now known as wayneeseguin | 13:40 | |
| *** jd__` is now known as jd__ | 13:42 | |
| *** rahmu_ is now known as rahmu | 13:42 | |
| *** Mithrand1r has joined #openstack-infra | 13:42 | |
| *** ilyashakhat has joined #openstack-infra | 13:42 | |
| *** dims has joined #openstack-infra | 13:42 | |
| *** EntropyWorks has joined #openstack-infra | 13:42 | |
| *** sbadia_ is now known as sbadia | 13:42 | |
| *** dansmith has joined #openstack-infra | 13:43 | |
| *** sbadia has quit IRC | 13:43 | |
| *** Guest50062 has joined #openstack-infra | 13:43 | |
| *** saju_m has quit IRC | 13:43 | |
| *** loquacities has quit IRC | 13:43 | |
| *** shortstop has quit IRC | 13:43 | |
| *** skraynev has quit IRC | 13:43 | |
| *** sirushti has joined #openstack-infra | 13:43 | |
| *** skraynev has joined #openstack-infra | 13:44 | |
| *** saju_m has joined #openstack-infra | 13:44 | |
| *** che-arne has quit IRC | 13:44 | |
| *** DinaBelova is now known as DinaBelova_ | 13:44 | |
| *** lcostantino has joined #openstack-infra | 13:44 | |
| *** psedlak has quit IRC | 13:44 | |
| *** psedlak has joined #openstack-infra | 13:45 | |
| *** loquacities has joined #openstack-infra | 13:45 | |
| *** Vivek_ has joined #openstack-infra | 13:45 | |
| *** jasond`` has joined #openstack-infra | 13:45 | |
| *** pdmars has quit IRC | 13:45 | |
| *** mbacchi has joined #openstack-infra | 13:46 | |
| *** sileht_ has joined #openstack-infra | 13:48 | |
| *** pabelanger has joined #openstack-infra | 13:50 | |
| *** stevebaker has joined #openstack-infra | 13:50 | |
| *** greghaynes has joined #openstack-infra | 13:50 | |
| *** mancdaz has joined #openstack-infra | 13:50 | |
| *** mrda_away has joined #openstack-infra | 13:51 | |
| *** dhellmann_ has joined #openstack-infra | 13:51 | |
| *** sirushti is now known as shortstop | 13:51 | |
| *** mbacchi has quit IRC | 13:51 | |
| *** mbacchi has joined #openstack-infra | 13:51 | |
| *** changbl has joined #openstack-infra | 13:51 | |
| *** Mithrandir has quit IRC | 13:51 | |
| *** ilyashakhat_ has quit IRC | 13:51 | |
| *** Vivek has quit IRC | 13:51 | |
| *** StevenK has quit IRC | 13:51 | |
| *** dhellmann has quit IRC | 13:51 | |
| *** dhellmann_ is now known as dhellmann | 13:51 | |
| *** sileht has quit IRC | 13:51 | |
| *** exz has quit IRC | 13:51 | |
| *** julienvey has joined #openstack-infra | 13:51 | |
| *** che-arne has joined #openstack-infra | 13:52 | |
| *** exz has joined #openstack-infra | 13:52 | |
| *** DinaBelova_ is now known as DinaBelova | 13:52 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 13:52 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 13:52 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 13:53 |
| *** jcoufal has quit IRC | 13:53 | |
| *** matrohon has quit IRC | 13:53 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Test for email-ext publisher https://review.openstack.org/75872 | 13:53 |
| *** mrodden has quit IRC | 13:53 | |
| *** jcoufal has joined #openstack-infra | 13:54 | |
| *** dprince has joined #openstack-infra | 13:55 | |
| *** pdmars has joined #openstack-infra | 13:56 | |
| *** jcooley_ has joined #openstack-infra | 13:56 | |
| *** matrohon has joined #openstack-infra | 13:57 | |
| *** CaptTofu has joined #openstack-infra | 13:58 | |
| *** yamahata has joined #openstack-infra | 13:58 | |
| *** mestery has quit IRC | 13:58 | |
| *** boris-42_ has quit IRC | 14:00 | |
| *** mkoderer has joined #openstack-infra | 14:00 | |
| *** mestery has joined #openstack-infra | 14:02 | |
| *** jcooley_ has quit IRC | 14:02 | |
| *** luqas has joined #openstack-infra | 14:03 | |
| *** saju_m has quit IRC | 14:05 | |
| *** kiall has quit IRC | 14:06 | |
| *** thomasem has joined #openstack-infra | 14:06 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Auth Token Middleware https://review.openstack.org/74735 | 14:06 |
| *** kiall has joined #openstack-infra | 14:07 | |
| *** mrodden has joined #openstack-infra | 14:08 | |
| *** dkranz has joined #openstack-infra | 14:08 | |
| fungi | ihrachys: did you contact hub_cap about that? | 14:09 |
| fungi | ihrachys: i'm all for deleting branches from trove if their ptl requests it | 14:09 |
| *** jeckersb_gone is now known as jeckersb | 14:10 | |
| openstackgerrit | ChangBo Guo(gcb) proposed a change to openstack-dev/pbr: Remove copyright from empty files https://review.openstack.org/75876 | 14:12 |
| fungi | sdague: that sounds to me like we should just rebuild openstack from scratch. the current incarnation is nothing but error messages | 14:12 |
| sdague | well, volume is against us :) I don't know the 24hr message catalog is probably hundreds of millions | 14:13 |
| *** lnxnut has joined #openstack-infra | 14:13 | |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add support for TestNG publisher. https://review.openstack.org/70157 | 14:14 |
| *** mayu has joined #openstack-infra | 14:14 | |
| mayu | jaypipes: hi, there ? | 14:15 |
| *** amotoki has quit IRC | 14:15 | |
| mayu | jaypipes: hi, I sent a email to you for help, there are ssh key file error on jenkins master node. | 14:17 |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Fix unused variable in ci-skip wrapper. https://review.openstack.org/75879 | 14:19 |
| ihrachys | fungi: he doesn't respond, but let's wait another day or two | 14:19 |
| *** mfer has joined #openstack-infra | 14:21 | |
| mayu | fungi: hi, there ? | 14:21 |
| fungi | mayu: hi | 14:22 |
| mayu | fungi: I apply a service account last time | 14:22 |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Add attachment pattern expression to email-ext. https://review.openstack.org/70148 | 14:22 |
| *** mriedem has joined #openstack-infra | 14:22 | |
| *** luqas has quit IRC | 14:23 | |
| mayu | fungi: there are problem on my ssh key files, I need to generate a new one | 14:23 |
| mayu | fungi: I need your help to refresh my service account public key info | 14:24 |
| fungi | mayu: are you trying to use /home/jenkins/.ssh/id_rsa.pub from your jenkins master? | 14:25 |
| *** homeless_ has quit IRC | 14:25 | |
| mayu | fungi: right | 14:25 |
| *** miqui has joined #openstack-infra | 14:26 | |
| mayu | fungi: the key file is generated from another vm, I copy it to the jenkins master | 14:26 |
| fungi | mayu: from a shell in that account, did you try to 'ssh -p 29418 huawei-ci@review.openstack.org gerrit ls-files' and received an authentication error? | 14:27 |
| mayu | fungi: no | 14:27 |
| *** homeless has joined #openstack-infra | 14:28 | |
| fungi | if the /home/jenkins/.ssh/id_rsa and /home/jenkins/.ssh/id_rsa.pub files were generated together and the /home/jenkins/.ssh/id_rsa.pub is the one i entered into gerrit got you, it should work | 14:28 |
| *** DinaBelova has quit IRC | 14:28 | |
| *** SergeyLukjanov has quit IRC | 14:28 | |
| *** ruhe has quit IRC | 14:28 | |
| *** NikitaKonovalov has quit IRC | 14:28 | |
| *** julim has joined #openstack-infra | 14:28 | |
| fungi | er, the one i entered into gerrit for you | 14:28 |
| *** DinaBelova_ has joined #openstack-infra | 14:29 | |
| mayu | fungi: http://paste.openstack.org/show/69025/ | 14:29 |
| *** DinaBelova_ is now known as DinaBelova | 14:29 | |
| *** NikitaKonovalov has joined #openstack-infra | 14:29 | |
| *** ruhe has joined #openstack-infra | 14:29 | |
| mayu | fungi: this is the whole steps. | 14:29 |
| *** SergeyLukjanov has joined #openstack-infra | 14:30 | |
| *** roey_ has joined #openstack-infra | 14:30 | |
| *** zehicle_at_dell has quit IRC | 14:30 | |
| *** roey_ has left #openstack-infra | 14:31 | |
| mayu | fungi: ssh -p 29418 huawei-ci@review.openstack.org gerrit stream-events, exec that command, I got the gerrit event | 14:31 |
| *** lcostantino has quit IRC | 14:31 | |
| *** roeyc has joined #openstack-infra | 14:31 | |
| fungi | mayu: that sounds like the keyfiles are correct then. this is with the decrypted private key (after you ran ssh-keygen -f id_rsa -p on it)? | 14:32 |
| *** lcostantino has joined #openstack-infra | 14:32 | |
| *** Mithrand1r is now known as Mithrandir | 14:33 | |
| fungi | looks like step 7 in your paste says that was the case, okay | 14:33 |
| *** ociuhandu has quit IRC | 14:33 | |
| mayu | fungi: you mean, the decrypted process cause the problem | 14:34 |
| fungi | i don't think you need new keys... have you tried restarting the jenkins service on your jenkins masters? they may cache copies of the original keyfiles in memory | 14:34 |
| mayu | yes | 14:34 |
| *** lcostantino has quit IRC | 14:35 | |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Adjust _TRUE_VALUES to be consistent with OpenStack projects. https://review.openstack.org/56318 | 14:35 |
| mayu | there are still errors in /var/log/zuul/debug.log | 14:35 |
| fungi | oh, i see, at the end of your paste you're using zuul anyway, so it's zuul which is connecting to gerrit, not the jenkins masters | 14:35 |
| mayu | fungi: as you see on the paste | 14:35 |
| mayu | right | 14:35 |
| fungi | the paste doesn't seem to include the errors you're getting from zuul, however | 14:36 |
| mayu | wait | 14:36 |
| *** prad_ has joined #openstack-infra | 14:36 | |
| mayu | fungi:http://paste.openstack.org/show/69030/ | 14:37 |
| mayu | fungi: the error | 14:37 |
| *** blamar has joined #openstack-infra | 14:38 | |
| fungi | mayu: did you possibly save your rsa keys onto the zuul server as id_dsa and id_dsa.pub instead if id_rsa and id_rsa.pub? | 14:39 |
| *** markmcclain has joined #openstack-infra | 14:41 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/75668 | 14:41 |
| mayu | fungi: you mean, change the file name ? | 14:41 |
| *** andreaf has quit IRC | 14:41 | |
| *** sandywalsh has quit IRC | 14:41 | |
| fungi | mayu: i'm just trying to figure out what could cause that error. you're trying to use an rsa keypair but paramiko is raising an exception complaining about an invalid dsa key | 14:43 |
| mayu | fungi: or, regenerate the key pairs? | 14:43 |
| mayu | fungi: right | 14:43 |
| *** dstanek_afk is now known as dstanek | 14:44 | |
| fungi | zuul should be able to use keyfiles in the same format as jenkins, and you've confirmed you're able to authenticate using it so i don't think you need to regenerate them | 14:44 |
| mayu | fungi: jaypipes use rsa keypair, I follow his blog to construct. | 14:45 |
| fungi | how did you copy the files? if you did it by cut-and-paste in your terminal, maybe the file is misformatted (stray newlines?) | 14:45 |
| mayu | scp the id_rsa and id_rsa.pub files | 14:45 |
| fungi | okay, that should have been fine | 14:46 |
| fungi | looking around on the internet, it seems like "not a valid dsa private key file" is a misleading error paramiko produces whenever it can't understand your keyfiles (regardless of whether they're rsa, dsa, ecdsa, whatever) | 14:47 |
| mayu | right, paramiko raise the exception | 14:48 |
| *** dkliban has joined #openstack-infra | 14:49 | |
| fungi | i'm still hunting around for any likely causes of that | 14:49 |
| sdague | fungi: can I get your eyes on - https://review.openstack.org/#/c/75683/ ? | 14:49 |
| mayu | It is keyfile problem | 14:49 |
| fungi | mayu: do the md5sums of your id_rsa and id_rsa.pub on your zuul server match the files on your jenkins servers where you're able to connect to gerrit? | 14:50 |
| sdague | mayu: does your keyfile have a password on it? | 14:50 |
| fungi | mayu did an 'ssh -f id_rsa -p' to strip the passphrase. just wondering whether the copy on the zuul server is from before or after doing that | 14:51 |
| mayu | Reseting passphase as empty, id_rsa must be modified. | 14:52 |
| mayu | fungi: there is no password now. | 14:52 |
| fungi | sdague: is 75683 replacing 75534? | 14:52 |
| *** gokrokve has quit IRC | 14:53 | |
| *** gokrokve has joined #openstack-infra | 14:53 | |
| *** jcooley_ has joined #openstack-infra | 14:53 | |
| fungi | mayu: in a shell on your zuul server, as the same user under which your zuul process runs, try running ssh -p 29418 huawei-ci@review.openstack.org gerrit stream-events | 14:53 |
| sdague | fungi: yeh, from what I can tell 75534 doesn't actually work | 14:54 |
| mayu | fungi: after | 14:54 |
| fungi | sdague: right. i would have expected a takeover of the old change, but this is fine | 14:54 |
| sdague | http://logs.openstack.org/83/75683/6/check/check-tempest-dsvm-full/6215674/logs/devstack-gate-setup-host.txt.gz | 14:54 |
| sdague | yeh, I guess I didn't quite realize what jeblair was doing with that change, and I just went roaring through a bunch of fix ups this weekend | 14:55 |
| mayu | fungi: it works, the gerrit event output to the screen | 14:55 |
| fungi | sdague: and we get the doubled timestamps in the console log like with 75534, so that we'll still have them when we're no longer relying on jenkins. lgtm | 14:55 |
| sdague | fungi: yeh, if you go up the patch series, I actually drop outputing the devstack logs to console eventually as well | 14:56 |
| *** talluri has joined #openstack-infra | 14:56 | |
| mayu | fungi: sorry, I got a miss | 14:56 |
| *** luqas has joined #openstack-infra | 14:56 | |
| *** sileht_ is now known as sileht | 14:57 | |
| fungi | sdague: grenade-specific setup logs look to be timestamped too. excellent | 14:57 |
| mayu | fungi: I did not run that as zuul user | 14:57 |
| *** sandywalsh has joined #openstack-infra | 14:57 | |
| *** sileht has joined #openstack-infra | 14:57 | |
| sdague | yep | 14:57 |
| sdague | I think I got all the logs | 14:57 |
| sdague | once that is in, we can get this sorted - https://review.openstack.org/#/c/75719/ | 14:58 |
| fungi | mayu: if your zuul service runs as the user "zuul" then you need to make sure the decrypted keypair gets copied into the ~zuul/.ssh/ directory | 14:58 |
| mayu | fungi: I will try. if it works, how do i do, if it doesn't work, how do i do | 14:58 |
| *** boris-42_ has joined #openstack-infra | 14:58 | |
| *** jcooley_ has quit IRC | 14:58 | |
| mayu | fungi: I think you get the point | 14:59 |
| *** mwagner_lap is now known as mwagner_bbl | 14:59 | |
| SergeyLukjanov | morning folks | 15:00 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Remove neutron-heat-slow job from query for bug 1253896 https://review.openstack.org/75891 | 15:00 |
| *** mrodden has quit IRC | 15:00 | |
| *** simonmcc_ is now known as simonmcc | 15:00 | |
| sdague | morning SergeyLukjanov | 15:01 |
| SergeyLukjanov | fungi, sdague, could you please take a look at https://review.openstack.org/#/c/73431/ ? it's about avoiding pbr's setup.py override while proposing requirements updates | 15:01 |
| fungi | mayu: it looks like we actually configure our zuul process to use a key at /var/lib/zuul/ssh/id_rsa | 15:01 |
| mayu | fungi: 26th,Feb is my deadline for contruction of third-party ci. Thanks for your help. I will go to company lab to finish the work. keep in touch | 15:01 |
| *** jnoller has joined #openstack-infra | 15:02 | |
| sdague | SergeyLukjanov: +A | 15:03 |
| sdague | SergeyLukjanov: then I'll hit you up for this - https://review.openstack.org/#/c/75683/ :) | 15:03 |
| mayu | fungi: you mean, the file in /var/lib/zuul/ssh/id_rsa, I had replaced this file with the passwordless one. | 15:03 |
| sdague | as I'd really like to get those timestamps in so we can adjust the indexer | 15:03 |
| fungi | mayu: correct | 15:03 |
| *** sahid has quit IRC | 15:04 | |
| mayu | I had exec command find / -name id_rsa, and replace the proper ones | 15:05 |
| SergeyLukjanov | sdague, thanks | 15:06 |
| SergeyLukjanov | sdague, looking at your timestamps patch | 15:06 |
| SergeyLukjanov | sdague, my knowledge of awk == google | 15:07 |
| sdague | yeh, it's lifted largely from stack.sh | 15:07 |
| *** markmcclain has quit IRC | 15:07 | |
| sdague | with one minor format change | 15:07 |
| *** oubiwan__ has joined #openstack-infra | 15:08 | |
| SergeyLukjanov | sdague, yup, find it | 15:08 |
| SergeyLukjanov | sdague, so, it'll just add timestamp prefix for each line | 15:08 |
| *** oubiwan__ has quit IRC | 15:09 | |
| sdague | yep | 15:10 |
| sdague | so the sublogs will have timestamps | 15:10 |
| sdague | because they don't get them from jenkins | 15:10 |
| SergeyLukjanov | sdague, ok, +A | 15:10 |
| *** Steap has joined #openstack-infra | 15:11 | |
| mayu | fungi: hi, I will go to company and try again. If it doesn't work, please help to refresh my service account public key. | 15:11 |
| *** mayu has quit IRC | 15:13 | |
| *** eharney has quit IRC | 15:13 | |
| *** thomasbiege has quit IRC | 15:14 | |
| *** oubiwan__ has joined #openstack-infra | 15:14 | |
| *** rwsu has joined #openstack-infra | 15:14 | |
| *** vishy has quit IRC | 15:15 | |
| *** dkliban is now known as dkliban_br | 15:20 | |
| *** dkliban_br is now known as dkliban_brb | 15:20 | |
| *** DuncanT- has joined #openstack-infra | 15:22 | |
| *** avishay has joined #openstack-infra | 15:22 | |
| *** DuncanT- has left #openstack-infra | 15:22 | |
| avishay | hi all, we have a bunch of patches failing, it looks like this is the culprit: https://bugs.launchpad.net/openstack-ci/+bug/1276242 | 15:23 |
| avishay | there's a patch with one +2 already, would appreciate it if someone could take a look | 15:23 |
| *** amir2 is now known as asadoughi | 15:23 | |
| *** jgrimm has joined #openstack-infra | 15:25 | |
| fungi | avishay: can you point me to a recent failure? i just want to make sure we're on the right track with that (though i approved the --no-use-wheels patch to pypi-mirror just now anyway because it's sane for the moment) | 15:25 |
| *** DuncanT- has joined #openstack-infra | 15:26 | |
| *** david_lyle_ has joined #openstack-infra | 15:27 | |
| *** rcleere has joined #openstack-infra | 15:28 | |
| mordred | fungi: ++ to --no-use-wheels | 15:28 |
| ildikov_ | fungi: here is one console output for this failure: https://jenkins03.openstack.org/job/check-tempest-dsvm-full/3738/console | 15:29 |
| *** andre__ has quit IRC | 15:30 | |
| *** dkliban_brb is now known as dkliban | 15:31 | |
| fungi | ildikov_: intersting. i wonder why that patch isn't hitting the issue consistently | 15:31 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 15:31 | |
| fungi | oh, you're adding a requirement for suds, so maybe that's changing the import order | 15:32 |
| *** markmcclain has joined #openstack-infra | 15:32 | |
| sdague | so the wheels mirror issue fixes that? | 15:32 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVxdWlyZW1lbnQucGFyc2UoJ3NpeD49MS41LjInKSlcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkzMjU1ODY0Mzc0fQ== | 15:33 |
| ildikov_ | fungi: we tried to figure out with my colleague what could go wrong, but we did not succeed | 15:33 |
| sdague | fungi: so I don't entirely understand why this isn't failing every run | 15:37 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVxdWlyZW1lbnQucGFyc2UoJ3NpeD49MS41LjInKSlcIiBBTkQgbWVzc2FnZTpcIm9wZW5zdGFja2NsaWVudC5zaGVsbCBFeGNlcHRpb24gcmFpc2VkXCIgQU5EIGZpbGVuYW1lOmNvbnNvbGUuaHRtbCIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkzMjU2MDQ5MTAwfQ== | 15:37 |
| sdague | but very clearly it's new | 15:37 |
| ildikov_ | fungi: I've checked some requirements.txt files, but I saw 1.4.1 in every project I checked, so I do not know where the 1.5.2 requirement comes from exactly | 15:37 |
| fungi | ildikov_: most likely suds or something suds requires is pulling in a newer version of it | 15:37 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVxdWlyZW1lbnQucGFyc2UoJ3NpeD49MS41LjInKSlcIiBBTkQgbWVzc2FnZTpcIm9wZW5zdGFja2NsaWVudC5zaGVsbCBFeGNlcHRpb24gcmFpc2VkXCIgQU5EIGZpbGVuYW1lOmNvbnNvbGUuaHRtbCIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkzMjU2MDQ5MTAwLCJtb2RlIjoic2NvcmUiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbm9kZSJ9 | 15:37 |
| sdague | fwiw it's all rax iad nodes | 15:38 |
| *** andreaf has joined #openstack-infra | 15:39 | |
| *** sc68cal has quit IRC | 15:39 | |
| avishay | i have no understanding of the issue, just know that our stuff is failing on it :) | 15:39 |
| *** fbo_away is now known as fbo | 15:40 | |
| *** wenlock has joined #openstack-infra | 15:40 | |
| ildikov_ | fungi: a-ha, I see, so it's more complicated, than I thought | 15:40 |
| *** tomhe has joined #openstack-infra | 15:40 | |
| mordred | fungi, sdague: the solum guys have a feature in the bot that's in their channel which notices urls and pastes shortened versions back into channel in response | 15:41 |
| mordred | we should get us some of that | 15:41 |
| sdague | sure | 15:41 |
| mordred | I mean, next time any of us have a pile of free tie | 15:41 |
| mordred | time | 15:41 |
| sdague | but the issue at hand | 15:41 |
| *** sc68cal has joined #openstack-infra | 15:41 | |
| fungi | mordred: i have a pile of something. pretty sure it's not free time | 15:42 |
| mordred | fungi: :) | 15:42 |
| fungi | sdague: that is a very strange correlation. we build images the same in each region, so not sure why only iad would exhibit it | 15:42 |
| *** jcooley_ has joined #openstack-infra | 15:42 | |
| sdague | fungi: no idea | 15:42 |
| fungi | oh! unless maybe there's something different about the ubuntu base image in iad! | 15:42 |
| mordred | fungi: maybe there was a temporary glich in the matrix at image create tie? | 15:43 |
| mordred | time | 15:43 |
| mordred | gah | 15:43 |
| fungi | i wonder if there's a system package of python-six installed on the nodes in that region | 15:43 |
| *** zhiyan- is now known as zhiyan_ | 15:45 | |
| *** beagles is now known as beagles_brb | 15:45 | |
| jeblair | fungi, mordred, sdague: good morning. anything urgent i should look into while you're checking out the six thing? | 15:46 |
| *** oubiwan__ has quit IRC | 15:47 | |
| sdague | that's sort of the urgent thing I think | 15:48 |
| tomhe | Regarding recent improvements on event processing in Zuul: We're having performance problems on event processing on our Zuul setup. Right now we're running on 951d8f3 from Mon Feb 10 20:40:46. This weekend I tried 264b06 from Fri Feb 21 08:28:58 to test the new event processing, but I couldn't get Zuul to launch any jobs on Jenkins. They showed up on the status page as queued, but nothing more happened. Any hints on why Zuul | 15:48 |
| mordred | jeblair: my coffee isn't very good - can you fix that? | 15:48 |
| sdague | interesting our pip freeze says: suds==0.4.1 | 15:49 |
| sdague | but that's not an installable version for me locally | 15:49 |
| *** thomasbiege has joined #openstack-infra | 15:49 | |
| *** mdenny has joined #openstack-infra | 15:49 | |
| jeblair | sdague: i'm reading your reply to hashar and not understanding why his answer isn't the answer to your question... | 15:49 |
| hashar | hello :-] | 15:49 |
| sdague | ok, well I wasn't parsing it this morning. :) | 15:50 |
| jeblair | sdague: it looks like he suggests a single timeout value in jjb that gets interpolated in 2 places (plugin and d-g parameter), and then in d-g you can subtract the appropriate time | 15:50 |
| fungi | tomhe: are you running the new zuul-merger daemon? | 15:50 |
| *** dripton has joined #openstack-infra | 15:51 | |
| fungi | tomhe: see the NEWS file and updated documentation in the openstack-infra/zuul repo | 15:51 |
| fungi | jeblair: nothing urgent i think. just housekeeping | 15:51 |
| jeblair | sdague: oh, he actually doesn't mention the part about it being a parameter to d-g; i think i must have filled that in. but i think it can be done. | 15:51 |
| tomhe | fungi: No, I hoped that it was started the same way as the Gearman server. Separate tool? | 15:51 |
| sdague | yeh, that's why I was confused. because basically that's what I was suggesting, and it seemed kind of circular from there. | 15:51 |
| fungi | tomhe: since it's expected to possibly run on a separate server, it has a separate initscript | 15:52 |
| jeblair | though the trick is that the timeout should be based on the job, not the project | 15:52 |
| sdague | I know what I want, which is that variable to get to the environment | 15:52 |
| fungi | tomhe: and pay attention to the minor configuration changes you'll need to make for the new [merger] section | 15:52 |
| sdague | I just don't know how to make that happen | 15:52 |
| tomhe | fungi: Thanks, I'll go back to the docs. | 15:52 |
| jeblair | and i don't think we can set up a variable interpolation from within the job-template | 15:52 |
| jeblair | so maybe it isn't the answer :( | 15:52 |
| tomhe | fungi: I have the configuration in place, just missed that the merger was started separately | 15:53 |
| fungi | jeblair: sdague: also, i pondered that more. moving options for how the job script is launched into hidden (from a cut-and-paste standpoint) variables makes it harder for people to recreate these outside our infrastructure | 15:54 |
| fungi | d-g currently falls back to a 60-minute timeout if none is provided, so if we stop adding that in the shell snippet for the job then you have to remember to look up what it is and insert it separately | 15:55 |
| fungi | if you're running something which will take more than 60 minutes anyway | 15:55 |
| *** andre__ has joined #openstack-infra | 15:55 | |
| jeblair | fungi: yeah. i think ideally the solution keeps the value within the job-template definition | 15:56 |
| sdague | fungi: honestly, I think removing copy paste is kind of important here, because we keep having errors in this template when people miss updating one of the variables | 15:56 |
| sdague | the current double entry is just error prone | 15:56 |
| *** pcrews has joined #openstack-infra | 15:57 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 15:57 | |
| sdague | jeblair: so do we actually need variable interpolation in the template? is there not a way to create jenkins config that just injects an environment variable directly? | 15:58 |
| jeblair | sdague: i think that's what hashar was saying isn't possible -- because the timeout plugin only accepts a direct value | 15:58 |
| hashar | unfrotunately :(- | 15:58 |
| jeblair | sdague: (so you can get an env variable to the builder, but not the jenkins timeout plugin) | 15:59 |
| hashar | but maybe you could have the JJB {timeout} passed down to the devstack.sh script | 15:59 |
| hashar | as a parameter/env variable | 15:59 |
| sdague | jeblair: sure, but can't we have JJB take the timeout variable and as it to the env | 15:59 |
| *** sdake_ has joined #openstack-infra | 15:59 | |
| sdague | and just keep doing what it's doing with it to the timeout plugin | 15:59 |
| fungi | do we need to use the evinject plugin for that? | 16:00 |
| sdague | 2014-02-24 14:00:57.451 | [EnvInject] - Loading node environment variables. | 16:00 |
| sdague | I mean, that's in every run | 16:00 |
| sdague | this six thing is so confusing | 16:01 |
| sdague | because I don't see anywhere, anyone specifies six>=1.5.2 in this run - http://logs.openstack.org/77/74977/3/check/check-tempest-dsvm-neutron/787f901/console.html | 16:01 |
| jeblair | sdague: i completely agree that this is a good idea -- i would have liked to implement it that way originally if i could have. but you said "we keep having errors in this template when people miss updating one of the variables". when has that happened? | 16:01 |
| jeblair | sdague: all of the jobs in devstack-gate.yaml are correct | 16:01 |
| openstackgerrit | sahid proposed a change to openstack/requirements: Updates six to the last version https://review.openstack.org/75907 | 16:02 |
| sdague | jeblair: I remember dealing with 2 of these issues in jan | 16:02 |
| fungi | sdague: i'm going to hold a node from iad and ord and compare them | 16:03 |
| sdague | basically every time we bulk change one of the job classes, we seem to miss one of the inner variables | 16:03 |
| sdague | also, from other investigations, the inner variables are all actually wrong | 16:03 |
| sdague | because of where they are applied | 16:03 |
| *** jeblair has quit IRC | 16:04 | |
| *** jeblair has joined #openstack-infra | 16:04 | |
| sdague | which was actually the point of - https://review.openstack.org/#/c/75726/ | 16:04 |
| *** dolphm is now known as dolphm_503 | 16:05 | |
| *** oubiwan__ has joined #openstack-infra | 16:05 | |
| *** testing has joined #openstack-infra | 16:06 | |
| sdague | basically there is an assumption that 5 minutes is enough time to handle the outer setup, but it actually turns out that it's not. The workspace setup is very variable, and the cleanup is almost 4 minutes on it's own. | 16:07 |
| jeblair | sdague: i don't remember that happening. perhaps it happened while i was away. | 16:07 |
| sdague | jeblair: I think it was | 16:07 |
| *** testing has quit IRC | 16:07 | |
| *** rlandy_ has joined #openstack-infra | 16:08 | |
| jeblair | sdague: of course -- the 5 mins was just an estimate, naturally it would need to change and i'm very happy to update this to the approach in your proposed change | 16:08 |
| sdague | yep | 16:08 |
| *** eharney has joined #openstack-infra | 16:08 | |
| sdague | honestly, the envinject thing is mostly an error proofing idea | 16:08 |
| sdague | it's the least of my concerns, it just feels dirty to have to duplicate it | 16:09 |
| sdague | but if we are going to dup it, we should at least actually match up the variables to the same values manually and do the offset subtraction for cleanup in d-g, because when we realize cleanup takes 6 minutes, we need to not have to update every job definition. | 16:10 |
| *** jraim_ is now known as jraim | 16:10 | |
| *** mgagne1 has joined #openstack-infra | 16:12 | |
| *** rlandy has quit IRC | 16:14 | |
| sdague | so anyway - https://review.openstack.org/#/q/status:open+project:openstack-infra/devstack-gate+branch:master+topic:clean_logs,n,z a bunch of changes I did yesterday which hopefully help clean up a lot of things, and address issues like this. Comments welcomed. I'm also happy to reorder the series if there are pieces you think are slam dunks, and others that need to be discussed | 16:14 |
| *** malini1 has joined #openstack-infra | 16:14 | |
| sdague | but all of this is sort of trivial right now to the fact that nothing seems to pass on rax iad nodes, and I have no idea why :) | 16:16 |
| mattoliverau | Morning everyone | 16:16 |
| anteaya | mattoliverau: morning | 16:17 |
| anteaya | you are up early/late | 16:17 |
| davidlenwell_ | Morning gentlemen, Do I need to do anything with this review other than be patient? https://review.openstack.org/#/c/75226/ | 16:18 |
| *** davidlenwell_ is now known as davidlenwell | 16:18 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: timestamp sublogs https://review.openstack.org/75683 | 16:18 |
| anteaya | davidlenwell: if you don't mind accepting an answer from a none gentleman, no, just be patient | 16:19 |
| *** KurtMartin has quit IRC | 16:19 | |
| mattoliverau | anteaya: I'm in the US this week and next, so in your time zone.. but very jetlagged :( | 16:19 |
| anteaya | mattoliverau: ah yes that is right, texas | 16:19 |
| anteaya | happy jetlag to you | 16:19 |
| *** lcostantino has joined #openstack-infra | 16:19 | |
| anteaya | how was your trip? | 16:20 |
| davidlenwell | anteaya: I'm happy to accept answers from gentlemen and ladies or some other class of human I am unaware of | 16:20 |
| anteaya | great | 16:20 |
| *** beagles_brb is now known as beagles | 16:21 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 16:21 |
| mattoliverau | Yup texas :) The trip was good, just long. Today is my second day, so I think I'll crash in a few hours. | 16:22 |
| anteaya | so if I am reading backscroll correctly https://bugs.launchpad.net/openstack-ci/+bug/1276242 is the bug dejour and we still don't know why it is still cropping up in rax-iad | 16:22 |
| anteaya | and the bug bot seems to be awol | 16:22 |
| *** dolphm_503 is now known as dolphm | 16:22 | |
| *** julienvey has left #openstack-infra | 16:22 | |
| jeblair | i believe the freenode ddos's are continuing | 16:23 |
| anteaya | mattoliverau: good plan, hard to think when your brain is still in a different timezone | 16:23 |
| anteaya | jeblair: oh great | 16:23 |
| *** kmartin has joined #openstack-infra | 16:23 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for six issue https://review.openstack.org/75913 | 16:25 |
| *** mayu has joined #openstack-infra | 16:25 | |
| *** eharney_ has joined #openstack-infra | 16:26 | |
| mattoliverau | brb, going to find some more coffee! | 16:26 |
| *** eharney has quit IRC | 16:27 | |
| anteaya | :D | 16:27 |
| *** chuck__ has joined #openstack-infra | 16:28 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/cookiecutter: Add src and bug links to README https://review.openstack.org/75915 | 16:29 |
| *** yassine has quit IRC | 16:29 | |
| *** talluri has quit IRC | 16:30 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove neutron-heat-slow job from query for bug 1253896 https://review.openstack.org/75891 | 16:30 |
| mayu | fungi: hi there | 16:31 |
| fungi | dhellmann: any clue what's triggering https://launchpad.net/bugs/1284006 (a second-order dependency being pulled in by openstackclient?) | 16:32 |
| fungi | mayu: hi, did you make any progress? | 16:32 |
| *** eharney_ is now known as eharney | 16:33 | |
| *** thomasbiege has quit IRC | 16:33 | |
| mayu | fungi: I copy ssh keypairs to ~zuul/.ssh directory. zuul debug.log error http://paste.openstack.org/show/69060/ | 16:33 |
| mayu | fungi: It is AuthenticationException: Authentication failed. not Dsa key file problem | 16:34 |
| fungi | mayu: well, that's an improvemement i think. i'll check the gerrit logs to see if i can tell why it failed to authenticate | 16:34 |
| mayu | fungi: greate | 16:35 |
| mayu | fungi: great | 16:35 |
| bnemec_ | So there's no sense rechecking patches that failed due to the six bug until https://review.openstack.org/#/c/74579/ merges, right? | 16:36 |
| *** bnemec_ is now known as bnemec | 16:37 | |
| fungi | mayu: your logs are clearly not in utc. what is your utc offset? | 16:37 |
| fungi | mayu: are you utc +8? | 16:37 |
| mayu | fungi: switch to user zuul, ssh -p 29418 "huawei-ci"@review.openstack.org gerrit stream-events, I can get gerrit event | 16:37 |
| *** lcheng has joined #openstack-infra | 16:37 | |
| mayu | fungi: yes, utc+8 | 16:37 |
| *** jcooley_ has quit IRC | 16:38 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Content-Type can now be set for email-ext publisher https://review.openstack.org/75919 | 16:38 |
| *** chuck__ has quit IRC | 16:38 | |
| anteaya | bnemec: that would be a safe way to go | 16:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1268274 https://review.openstack.org/75649 | 16:40 |
| *** luqas has quit IRC | 16:40 | |
| anteaya | it is in check and almost done there | 16:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for six issue https://review.openstack.org/75913 | 16:40 |
| bnemec | anteaya: Okay, thanks. | 16:41 |
| *** markmcclain has quit IRC | 16:41 | |
| *** pmathews has joined #openstack-infra | 16:41 | |
| fungi | i couldn't find any obvious system installs of six when comparing an iad and a dfw devstack-precise slave | 16:41 |
| anteaya | hmmmm | 16:41 |
| fungi | both seemed to have six==1.4.1 installed according to pip freeze | 16:41 |
| *** e0ne_ has quit IRC | 16:41 | |
| *** dkliban is now known as dkliban_afk | 16:42 | |
| sdague | fungi: yep, I don't think that's the issue. I think something thinks it needs six >= 1.5.2 | 16:42 |
| *** e0ne has joined #openstack-infra | 16:42 | |
| sdague | that is being pulled in | 16:42 |
| anteaya | fungi: how long after 74579 merges before folks can recheck, is there a dissemination phase for this one? | 16:42 |
| *** Ryan_Lane1 has quit IRC | 16:42 | |
| anteaya | we would have to wait for mirrors to rebuild, correct? | 16:42 |
| mayu | fungi: I am waiting for you. Is there some clue be found in gerrit log ? | 16:43 |
| fungi | mayu: i am still looking. i haven't found any obvious indication that gerrit saw an ssh connection from your zuul server at the time in your logs... what is the ip address of your server? | 16:43 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1282876 https://review.openstack.org/75542 | 16:44 |
| *** bearhand1 is now known as comstud | 16:45 | |
| *** garyk has quit IRC | 16:45 | |
| mayu | internal 10.250.201.20 external 202.96.134.* | 16:45 |
| *** pmathews has quit IRC | 16:45 | |
| mayu | fungi: internal 10.250.201.20 external 202.96.134.* | 16:45 |
| *** relaxdiego has joined #openstack-infra | 16:46 | |
| *** luqas has joined #openstack-infra | 16:46 | |
| *** e0ne has quit IRC | 16:46 | |
| fungi | mayu: found you... [2014-02-24 16:25:36,439 +0000] a377d880 'Huawei openstack ci' - AUTH FAILURE FROM 202.96.135.154 user-not-found | 16:48 |
| *** esker has joined #openstack-infra | 16:48 | |
| *** pmathews has joined #openstack-infra | 16:49 | |
| fungi | mayu: you need to change your zuul configuration to use "huawei-ci" instead of "Huawei openstack ci" as your ssh username | 16:55 |
| mayu | fungi: ok | 16:55 |
| mayu | fungi: I will try | 16:55 |
| *** pblaho has quit IRC | 16:55 | |
| *** malini1 has quit IRC | 16:55 | |
| *** markmc has quit IRC | 16:56 | |
| *** nati_ueno has joined #openstack-infra | 16:57 | |
| *** malini has joined #openstack-infra | 16:57 | |
| *** dolphm is now known as dolphm_503 | 16:57 | |
| *** nati_ueno has quit IRC | 16:57 | |
| *** tjones has joined #openstack-infra | 16:58 | |
| *** dolphm_503 is now known as dolphm | 16:58 | |
| *** yassine_ has quit IRC | 16:58 | |
| *** jlibosva has quit IRC | 16:59 | |
| mayu | fungi: good news, there are not error in zuul debug.log | 16:59 |
| *** homeless has quit IRC | 17:00 | |
| *** matrohon has quit IRC | 17:00 | |
| *** zul has quit IRC | 17:00 | |
| *** miqui has quit IRC | 17:00 | |
| *** dkranz has quit IRC | 17:00 | |
| *** dkranz has joined #openstack-infra | 17:01 | |
| *** dangers has joined #openstack-infra | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| *** relaxdiego has quit IRC | 17:04 | |
| *** kmartin has quit IRC | 17:04 | |
| *** lcostantino has quit IRC | 17:04 | |
| *** rcleere has quit IRC | 17:04 | |
| *** mestery has quit IRC | 17:04 | |
| *** jcoufal has quit IRC | 17:04 | |
| *** locke105 has quit IRC | 17:04 | |
| *** rfolco has quit IRC | 17:04 | |
| *** ogelbukh has quit IRC | 17:04 | |
| *** lyxus has quit IRC | 17:04 | |
| *** jang has quit IRC | 17:04 | |
| *** davidlenwell has quit IRC | 17:04 | |
| *** dangers_away has quit IRC | 17:04 | |
| *** asadoughi has quit IRC | 17:04 | |
| *** mattoliverau has quit IRC | 17:04 | |
| *** wenlock has quit IRC | 17:04 | |
| *** EntropyWorks has quit IRC | 17:04 | |
| *** pabelanger has quit IRC | 17:04 | |
| *** stevebaker has quit IRC | 17:04 | |
| *** greghaynes has quit IRC | 17:04 | |
| *** mancdaz has quit IRC | 17:04 | |
| *** eharney has quit IRC | 17:04 | |
| *** rlandy_ has quit IRC | 17:04 | |
| *** oubiwan__ has quit IRC | 17:04 | |
| *** sdake_ has quit IRC | 17:04 | |
| *** andre__ has quit IRC | 17:04 | |
| *** jgrimm has quit IRC | 17:04 | |
| *** gokrokve has quit IRC | 17:04 | |
| *** dkliban_afk has quit IRC | 17:04 | |
| *** blamar has quit IRC | 17:04 | |
| *** SergeyLukjanov has quit IRC | 17:04 | |
| *** ruhe has quit IRC | 17:04 | |
| *** NikitaKonovalov_ has quit IRC | 17:04 | |
| *** DinaBelova has quit IRC | 17:04 | |
| *** mfer has quit IRC | 17:04 | |
| *** lnxnut has quit IRC | 17:04 | |
| *** changbl has quit IRC | 17:04 | |
| *** dhellmann has quit IRC | 17:04 | |
| *** Vivek_ has quit IRC | 17:04 | |
| *** shortstop has quit IRC | 17:04 | |
| *** Guest50062 has quit IRC | 17:04 | |
| *** dims has quit IRC | 17:04 | |
| *** Mithrandir has quit IRC | 17:04 | |
| *** jd__ has quit IRC | 17:04 | |
| *** zhiyan_ has quit IRC | 17:04 | |
| *** ArxCruz has quit IRC | 17:04 | |
| *** mattymo has quit IRC | 17:04 | |
| *** samalba has quit IRC | 17:04 | |
| *** SpamapS_ has quit IRC | 17:04 | |
| *** vipul has quit IRC | 17:04 | |
| *** protux has quit IRC | 17:04 | |
| *** davidhadas__ has quit IRC | 17:04 | |
| *** davidhadas_ has quit IRC | 17:04 | |
| *** davidhadas has quit IRC | 17:04 | |
| *** bookwar has quit IRC | 17:04 | |
| *** luqas has quit IRC | 17:04 | |
| *** mdenny has quit IRC | 17:04 | |
| *** david_lyle_ has quit IRC | 17:04 | |
| *** prad_ has quit IRC | 17:04 | |
| *** julim has quit IRC | 17:04 | |
| *** thomasem has quit IRC | 17:04 | |
| *** mkoderer has quit IRC | 17:04 | |
| *** CaptTofu has quit IRC | 17:04 | |
| *** pdmars has quit IRC | 17:04 | |
| *** dprince has quit IRC | 17:04 | |
| *** exz has quit IRC | 17:04 | |
| *** che-arne has quit IRC | 17:04 | |
| *** enikanorov_ has quit IRC | 17:04 | |
| *** weshay has quit IRC | 17:04 | |
| *** dstanek has quit IRC | 17:04 | |
| *** Hunner has quit IRC | 17:04 | |
| *** hemnafk has quit IRC | 17:04 | |
| *** bradm has quit IRC | 17:04 | |
| *** akscram has quit IRC | 17:04 | |
| *** rahmu has quit IRC | 17:04 | |
| *** dizquierdo has quit IRC | 17:04 | |
| *** jaypipes has quit IRC | 17:04 | |
| *** nibalizer has quit IRC | 17:04 | |
| *** plomakin_ has quit IRC | 17:04 | |
| *** rossella-s has quit IRC | 17:04 | |
| *** dosaboy has quit IRC | 17:04 | |
| *** flaper87 has quit IRC | 17:04 | |
| *** mwagner_bbl has quit IRC | 17:04 | |
| *** vladan has quit IRC | 17:04 | |
| *** antonym has quit IRC | 17:04 | |
| *** spiffxp_ has quit IRC | 17:04 | |
| *** kashyap has quit IRC | 17:04 | |
| *** jcooley has quit IRC | 17:04 | |
| *** esmute has quit IRC | 17:04 | |
| *** belliott has quit IRC | 17:04 | |
| *** adarazs_ has quit IRC | 17:04 | |
| *** jog0_ has quit IRC | 17:04 | |
| *** jooools1 has quit IRC | 17:04 | |
| *** therve has quit IRC | 17:04 | |
| *** luisg has quit IRC | 17:04 | |
| *** guitarzan has quit IRC | 17:04 | |
| *** mburned_` has quit IRC | 17:04 | |
| *** michchap has quit IRC | 17:04 | |
| *** aude has quit IRC | 17:04 | |
| *** Steap has quit IRC | 17:04 | |
| *** jeremyb has quit IRC | 17:04 | |
| *** mugsie has quit IRC | 17:04 | |
| *** medieval1 has quit IRC | 17:04 | |
| *** yamahata_ has quit IRC | 17:04 | |
| *** HenryG has quit IRC | 17:04 | |
| *** katyafervent_awa has quit IRC | 17:04 | |
| *** dstufft has quit IRC | 17:04 | |
| *** Xurong has quit IRC | 17:04 | |
| *** krtaylor has quit IRC | 17:04 | |
| *** gema has quit IRC | 17:04 | |
| *** lifeless has quit IRC | 17:04 | |
| *** geekinutah1 has quit IRC | 17:04 | |
| *** mika has quit IRC | 17:04 | |
| *** dkehn_ has quit IRC | 17:04 | |
| *** jesusaurus has quit IRC | 17:04 | |
| *** primeministerp has quit IRC | 17:04 | |
| *** gpocentek has quit IRC | 17:04 | |
| *** GheRivero has quit IRC | 17:04 | |
| *** BobBall has quit IRC | 17:04 | |
| *** mordred has quit IRC | 17:04 | |
| *** ihrachys has quit IRC | 17:04 | |
| *** dtroyer has quit IRC | 17:04 | |
| *** Daviey has quit IRC | 17:04 | |
| *** sc68cal has quit IRC | 17:04 | |
| *** boris-42_ has quit IRC | 17:04 | |
| *** sileht has quit IRC | 17:04 | |
| *** mrda_away has quit IRC | 17:04 | |
| *** StevenK_ has quit IRC | 17:04 | |
| *** derekh has quit IRC | 17:04 | |
| *** EmilienM has quit IRC | 17:04 | |
| *** tteggel has quit IRC | 17:04 | |
| *** dolphm has quit IRC | 17:04 | |
| *** Adri2000 has quit IRC | 17:04 | |
| *** jraim has quit IRC | 17:04 | |
| *** westmaas has quit IRC | 17:04 | |
| *** maurosr has quit IRC | 17:04 | |
| *** Shrews has quit IRC | 17:04 | |
| *** clarkb has quit IRC | 17:04 | |
| *** koolhead17 has quit IRC | 17:04 | |
| *** chmouel has quit IRC | 17:04 | |
| *** alaski has quit IRC | 17:04 | |
| *** soren has quit IRC | 17:04 | |
| *** dragondm has quit IRC | 17:04 | |
| *** fungi has quit IRC | 17:04 | |
| *** jdurgin has quit IRC | 17:04 | |
| *** zigo has quit IRC | 17:04 | |
| *** pvo has quit IRC | 17:04 | |
| *** Guest34377 has quit IRC | 17:04 | |
| *** wendar has quit IRC | 17:04 | |
| *** jbryce has quit IRC | 17:04 | |
| *** russellb has quit IRC | 17:04 | |
| *** jlk has quit IRC | 17:04 | |
| *** adam_g has quit IRC | 17:04 | |
| *** ttx has quit IRC | 17:04 | |
| *** leifmadsen has quit IRC | 17:04 | |
| *** saschpe has quit IRC | 17:04 | |
| *** tristanC has quit IRC | 17:04 | |
| *** msuriar_ has quit IRC | 17:04 | |
| *** mikal has quit IRC | 17:04 | |
| *** beagles has quit IRC | 17:04 | |
| *** rockstar has quit IRC | 17:04 | |
| *** jeckersb has quit IRC | 17:04 | |
| *** hughsaunders has quit IRC | 17:04 | |
| *** cp16net has quit IRC | 17:04 | |
| *** kevinbenton has quit IRC | 17:04 | |
| *** saper has quit IRC | 17:04 | |
| *** isviridov has quit IRC | 17:04 | |
| *** fbo has quit IRC | 17:04 | |
| *** ChanServ has quit IRC | 17:04 | |
| *** morganfainberg_Z has quit IRC | 17:04 | |
| *** mayu has quit IRC | 17:04 | |
| *** tomhe has quit IRC | 17:04 | |
| *** roeyc has quit IRC | 17:04 | |
| *** ildikov_ has quit IRC | 17:04 | |
| *** jgallard has quit IRC | 17:04 | |
| *** jerryz has quit IRC | 17:04 | |
| *** malini has quit IRC | 17:04 | |
| *** psedlak has quit IRC | 17:04 | |
| *** anteaya has quit IRC | 17:04 | |
| *** ekarlso has quit IRC | 17:04 | |
| *** tru_tru has quit IRC | 17:04 | |
| *** tchaypo has quit IRC | 17:04 | |
| *** gothicmindfood has quit IRC | 17:04 | |
| *** portante has quit IRC | 17:04 | |
| *** hdd has quit IRC | 17:04 | |
| *** jamielennox|away has quit IRC | 17:04 | |
| *** pleia2 has quit IRC | 17:04 | |
| *** hub_cap has quit IRC | 17:04 | |
| *** gmoro has quit IRC | 17:04 | |
| *** niska has quit IRC | 17:04 | |
| *** comstud has quit IRC | 17:04 | |
| *** med_ has quit IRC | 17:04 | |
| *** jamespage has quit IRC | 17:04 | |
| *** notmyname has quit IRC | 17:04 | |
| *** dkehn has quit IRC | 17:04 | |
| *** bogdando has quit IRC | 17:04 | |
| *** sdake has quit IRC | 17:04 | |
| *** shardy has quit IRC | 17:04 | |
| *** NobodyCam has quit IRC | 17:04 | |
| *** devananda has quit IRC | 17:04 | |
| *** zaro has quit IRC | 17:04 | |
| *** Ng has quit IRC | 17:04 | |
| *** SlickNik has quit IRC | 17:04 | |
| *** mriedem has quit IRC | 17:04 | |
| *** jpeeler has quit IRC | 17:04 | |
| *** smarcet has quit IRC | 17:04 | |
| *** alexpilotti has quit IRC | 17:04 | |
| *** jp_at_hp1 has quit IRC | 17:04 | |
| *** vkozhukalov_ has quit IRC | 17:04 | |
| *** bodepd_ has quit IRC | 17:04 | |
| *** cyeoh has quit IRC | 17:04 | |
| *** mkerrin has quit IRC | 17:04 | |
| *** johnthetubaguy has quit IRC | 17:04 | |
| *** persia has quit IRC | 17:04 | |
| *** dachary has quit IRC | 17:04 | |
| *** juice has quit IRC | 17:04 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** Hefeweizen has quit IRC | 17:04 | |
| *** marun has quit IRC | 17:04 | |
| *** simonmcc has quit IRC | 17:04 | |
| *** Alex_Gaynor has quit IRC | 17:04 | |
| *** AaronGr has quit IRC | 17:04 | |
| *** milki has quit IRC | 17:04 | |
| *** DuncanT- has quit IRC | 17:04 | |
| *** david-lyle has quit IRC | 17:04 | |
| *** oubiwann has quit IRC | 17:04 | |
| *** spredzy_ has quit IRC | 17:04 | |
| *** mtreinish has quit IRC | 17:04 | |
| *** gaelL has quit IRC | 17:04 | |
| *** bknudson has quit IRC | 17:04 | |
| *** sdague has quit IRC | 17:04 | |
| *** yolanda has joined #openstack-infra | 17:04 | |
| *** dprince has joined #openstack-infra | 17:06 | |
| *** yolanda is now known as 23LAAA2TD | 17:06 | |
| *** EntropyWorks has joined #openstack-infra | 17:06 | |
| *** wenlock has joined #openstack-infra | 17:06 | |
| *** sarob has joined #openstack-infra | 17:06 | |
| *** rfolco has joined #openstack-infra | 17:06 | |
| *** miqui has joined #openstack-infra | 17:06 | |
| *** homeless has joined #openstack-infra | 17:06 | |
| *** malini has joined #openstack-infra | 17:06 | |
| *** luqas has joined #openstack-infra | 17:06 | |
| *** eharney has joined #openstack-infra | 17:06 | |
| *** mayu has joined #openstack-infra | 17:06 | |
| *** rlandy_ has joined #openstack-infra | 17:06 | |
| *** oubiwan__ has joined #openstack-infra | 17:06 | |
| *** sdake_ has joined #openstack-infra | 17:06 | |
| *** andre__ has joined #openstack-infra | 17:06 | |
| *** mdenny has joined #openstack-infra | 17:06 | |
| *** sc68cal has joined #openstack-infra | 17:06 | |
| *** tomhe has joined #openstack-infra | 17:06 | |
| *** david_lyle_ has joined #openstack-infra | 17:06 | |
| *** jgrimm has joined #openstack-infra | 17:06 | |
| *** Steap has joined #openstack-infra | 17:06 | |
| *** boris-42_ has joined #openstack-infra | 17:06 | |
| *** sileht has joined #openstack-infra | 17:06 | |
| *** gokrokve has joined #openstack-infra | 17:06 | |
| *** dkliban_afk has joined #openstack-infra | 17:06 | |
| *** blamar has joined #openstack-infra | 17:06 | |
| *** prad_ has joined #openstack-infra | 17:06 | |
| *** roeyc has joined #openstack-infra | 17:06 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:06 | |
| *** ruhe has joined #openstack-infra | 17:06 | |
| *** NikitaKonovalov_ has joined #openstack-infra | 17:06 | |
| *** DinaBelova has joined #openstack-infra | 17:06 | |
| *** julim has joined #openstack-infra | 17:06 | |
| *** mfer has joined #openstack-infra | 17:06 | |
| *** lnxnut has joined #openstack-infra | 17:06 | |
| *** thomasem has joined #openstack-infra | 17:06 | |
| *** CaptTofu has joined #openstack-infra | 17:06 | |
| *** pdmars has joined #openstack-infra | 17:06 | |
| *** 18WAFV1VI has joined #openstack-infra | 17:06 | |
| *** exz has joined #openstack-infra | 17:06 | |
| *** che-arne has joined #openstack-infra | 17:06 | |
| *** changbl has joined #openstack-infra | 17:06 | |
| *** dhellmann has joined #openstack-infra | 17:06 | |
| *** mrda_away has joined #openstack-infra | 17:06 | |
| *** mancdaz has joined #openstack-infra | 17:06 | |
| *** greghaynes has joined #openstack-infra | 17:06 | |
| *** stevebaker has joined #openstack-infra | 17:06 | |
| *** pabelanger has joined #openstack-infra | 17:06 | |
| *** Vivek_ has joined #openstack-infra | 17:06 | |
| *** psedlak has joined #openstack-infra | 17:06 | |
| *** shortstop has joined #openstack-infra | 17:06 | |
| *** Guest50062 has joined #openstack-infra | 17:06 | |
| *** dims has joined #openstack-infra | 17:06 | |
| *** Mithrandir has joined #openstack-infra | 17:06 | |
| *** StevenK_ has joined #openstack-infra | 17:06 | |
| *** rahmu has joined #openstack-infra | 17:06 | |
| *** jd__ has joined #openstack-infra | 17:06 | |
| *** zhiyan_ has joined #openstack-infra | 17:06 | |
| *** enikanorov_ has joined #openstack-infra | 17:06 | |
| *** ArxCruz has joined #openstack-infra | 17:06 | |
| *** weshay has joined #openstack-infra | 17:06 | |
| *** mattymo has joined #openstack-infra | 17:06 | |
| *** dizquierdo has joined #openstack-infra | 17:06 | |
| *** samalba has joined #openstack-infra | 17:06 | |
| *** jaypipes has joined #openstack-infra | 17:06 | |
| *** nibalizer has joined #openstack-infra | 17:06 | |
| *** plomakin_ has joined #openstack-infra | 17:06 | |
| *** SpamapS_ has joined #openstack-infra | 17:06 | |
| *** vipul has joined #openstack-infra | 17:06 | |
| *** ildikov_ has joined #openstack-infra | 17:06 | |
| *** protux has joined #openstack-infra | 17:06 | |
| *** derekh has joined #openstack-infra | 17:06 | |
| *** davidhadas__ has joined #openstack-infra | 17:06 | |
| *** davidhadas_ has joined #openstack-infra | 17:06 | |
| *** davidhadas has joined #openstack-infra | 17:06 | |
| *** rossella-s has joined #openstack-infra | 17:06 | |
| *** jgallard has joined #openstack-infra | 17:06 | |
| *** bookwar has joined #openstack-infra | 17:06 | |
| *** dosaboy has joined #openstack-infra | 17:06 | |
| *** flaper87 has joined #openstack-infra | 17:06 | |
| *** dstanek has joined #openstack-infra | 17:06 | |
| *** Hunner has joined #openstack-infra | 17:06 | |
| *** kashyap has joined #openstack-infra | 17:06 | |
| *** mwagner_bbl has joined #openstack-infra | 17:06 | |
| *** tchaypo has joined #openstack-infra | 17:06 | |
| *** EmilienM has joined #openstack-infra | 17:06 | |
| *** tteggel has joined #openstack-infra | 17:06 | |
| *** vladan has joined #openstack-infra | 17:06 | |
| *** dolphm has joined #openstack-infra | 17:06 | |
| *** Adri2000 has joined #openstack-infra | 17:06 | |
| *** hemnafk has joined #openstack-infra | 17:06 | |
| *** antonym has joined #openstack-infra | 17:06 | |
| *** spiffxp_ has joined #openstack-infra | 17:06 | |
| *** belliott has joined #openstack-infra | 17:06 | |
| *** jcooley has joined #openstack-infra | 17:06 | |
| *** esmute has joined #openstack-infra | 17:06 | |
| *** adarazs_ has joined #openstack-infra | 17:06 | |
| *** jog0_ has joined #openstack-infra | 17:06 | |
| *** jooools1 has joined #openstack-infra | 17:06 | |
| *** therve has joined #openstack-infra | 17:06 | |
| *** luisg has joined #openstack-infra | 17:06 | |
| *** jraim has joined #openstack-infra | 17:06 | |
| *** guitarzan has joined #openstack-infra | 17:06 | |
| *** akscram has joined #openstack-infra | 17:06 | |
| *** bradm has joined #openstack-infra | 17:06 | |
| *** mburned_` has joined #openstack-infra | 17:06 | |
| *** michchap has joined #openstack-infra | 17:06 | |
| *** anteaya has joined #openstack-infra | 17:06 | |
| *** ekarlso has joined #openstack-infra | 17:06 | |
| *** tru_tru has joined #openstack-infra | 17:06 | |
| *** aude has joined #openstack-infra | 17:06 | |
| *** gothicmindfood has joined #openstack-infra | 17:06 | |
| *** portante has joined #openstack-infra | 17:06 | |
| *** jeremyb has joined #openstack-infra | 17:06 | |
| *** mugsie has joined #openstack-infra | 17:06 | |
| *** medieval1 has joined #openstack-infra | 17:06 | |
| *** yamahata_ has joined #openstack-infra | 17:06 | |
| *** HenryG has joined #openstack-infra | 17:06 | |
| *** katyafervent_awa has joined #openstack-infra | 17:06 | |
| *** dstufft has joined #openstack-infra | 17:06 | |
| *** krtaylor has joined #openstack-infra | 17:06 | |
| *** Xurong has joined #openstack-infra | 17:06 | |
| *** gema has joined #openstack-infra | 17:06 | |
| *** lifeless has joined #openstack-infra | 17:06 | |
| *** geekinutah1 has joined #openstack-infra | 17:06 | |
| *** mika has joined #openstack-infra | 17:06 | |
| *** dkehn_ has joined #openstack-infra | 17:06 | |
| *** jesusaurus has joined #openstack-infra | 17:06 | |
| *** primeministerp has joined #openstack-infra | 17:06 | |
| *** gpocentek has joined #openstack-infra | 17:06 | |
| *** GheRivero has joined #openstack-infra | 17:06 | |
| *** BobBall has joined #openstack-infra | 17:06 | |
| *** mordred has joined #openstack-infra | 17:06 | |
| *** ihrachys has joined #openstack-infra | 17:06 | |
| *** Daviey has joined #openstack-infra | 17:06 | |
| *** dtroyer has joined #openstack-infra | 17:06 | |
| *** hdd has joined #openstack-infra | 17:06 | |
| *** jamielennox|away has joined #openstack-infra | 17:06 | |
| *** pleia2 has joined #openstack-infra | 17:06 | |
| *** hub_cap has joined #openstack-infra | 17:06 | |
| *** westmaas has joined #openstack-infra | 17:06 | |
| *** maurosr has joined #openstack-infra | 17:06 | |
| *** Shrews has joined #openstack-infra | 17:06 | |
| *** clarkb has joined #openstack-infra | 17:06 | |
| *** koolhead17 has joined #openstack-infra | 17:06 | |
| *** chmouel has joined #openstack-infra | 17:06 | |
| *** alaski has joined #openstack-infra | 17:06 | |
| *** dragondm has joined #openstack-infra | 17:06 | |
| *** soren has joined #openstack-infra | 17:06 | |
| *** fungi has joined #openstack-infra | 17:06 | |
| *** jdurgin has joined #openstack-infra | 17:06 | |
| *** zigo has joined #openstack-infra | 17:06 | |
| *** pvo has joined #openstack-infra | 17:06 | |
| *** Guest34377 has joined #openstack-infra | 17:06 | |
| *** wendar has joined #openstack-infra | 17:06 | |
| *** jbryce has joined #openstack-infra | 17:06 | |
| *** russellb has joined #openstack-infra | 17:06 | |
| *** jlk has joined #openstack-infra | 17:06 | |
| *** adam_g has joined #openstack-infra | 17:06 | |
| *** ttx has joined #openstack-infra | 17:06 | |
| *** gmoro has joined #openstack-infra | 17:06 | |
| *** niska has joined #openstack-infra | 17:06 | |
| *** comstud has joined #openstack-infra | 17:06 | |
| *** med_ has joined #openstack-infra | 17:06 | |
| *** leifmadsen has joined #openstack-infra | 17:06 | |
| *** jerryz has joined #openstack-infra | 17:06 | |
| *** jamespage has joined #openstack-infra | 17:06 | |
| *** notmyname has joined #openstack-infra | 17:06 | |
| *** dkehn has joined #openstack-infra | 17:06 | |
| *** saschpe has joined #openstack-infra | 17:06 | |
| *** tristanC has joined #openstack-infra | 17:06 | |
| *** msuriar_ has joined #openstack-infra | 17:06 | |
| *** mikal has joined #openstack-infra | 17:06 | |
| *** beagles has joined #openstack-infra | 17:06 | |
| *** rockstar has joined #openstack-infra | 17:06 | |
| *** jeckersb has joined #openstack-infra | 17:06 | |
| *** hughsaunders has joined #openstack-infra | 17:06 | |
| *** bogdando has joined #openstack-infra | 17:06 | |
| *** sdake has joined #openstack-infra | 17:06 | |
| *** morganfainberg_Z has joined #openstack-infra | 17:06 | |
| *** cp16net has joined #openstack-infra | 17:06 | |
| *** kevinbenton has joined #openstack-infra | 17:06 | |
| *** saper has joined #openstack-infra | 17:06 | |
| *** isviridov has joined #openstack-infra | 17:06 | |
| *** fbo has joined #openstack-infra | 17:06 | |
| *** ChanServ has joined #openstack-infra | 17:06 | |
| *** shardy has joined #openstack-infra | 17:06 | |
| *** NobodyCam has joined #openstack-infra | 17:06 | |
| *** devananda has joined #openstack-infra | 17:06 | |
| *** zaro has joined #openstack-infra | 17:06 | |
| *** Ng has joined #openstack-infra | 17:06 | |
| *** SlickNik has joined #openstack-infra | 17:06 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 17:06 | |
| *** kiall has quit IRC | 17:06 | |
| *** DuncanT- has joined #openstack-infra | 17:06 | |
| *** mriedem has joined #openstack-infra | 17:06 | |
| *** jpeeler has joined #openstack-infra | 17:06 | |
| *** smarcet has joined #openstack-infra | 17:06 | |
| *** alexpilotti has joined #openstack-infra | 17:06 | |
| *** jp_at_hp1 has joined #openstack-infra | 17:06 | |
| *** vkozhukalov_ has joined #openstack-infra | 17:06 | |
| *** bodepd_ has joined #openstack-infra | 17:06 | |
| *** cyeoh has joined #openstack-infra | 17:06 | |
| *** mkerrin has joined #openstack-infra | 17:06 | |
| *** johnthetubaguy has joined #openstack-infra | 17:06 | |
| *** persia has joined #openstack-infra | 17:06 | |
| *** dachary has joined #openstack-infra | 17:06 | |
| *** juice has joined #openstack-infra | 17:06 | |
| *** openstackgerrit has joined #openstack-infra | 17:06 | |
| *** Hefeweizen has joined #openstack-infra | 17:06 | |
| *** marun has joined #openstack-infra | 17:06 | |
| *** simonmcc has joined #openstack-infra | 17:06 | |
| *** Alex_Gaynor has joined #openstack-infra | 17:06 | |
| *** AaronGr has joined #openstack-infra | 17:06 | |
| *** milki has joined #openstack-infra | 17:06 | |
| *** david-lyle has joined #openstack-infra | 17:06 | |
| *** oubiwann has joined #openstack-infra | 17:06 | |
| *** spredzy_ has joined #openstack-infra | 17:06 | |
| *** gaelL has joined #openstack-infra | 17:06 | |
| *** mtreinish has joined #openstack-infra | 17:06 | |
| *** bknudson has joined #openstack-infra | 17:06 | |
| *** sdague has joined #openstack-infra | 17:06 | |
| *** dkranz has quit IRC | 17:06 | |
| *** dkranz has joined #openstack-infra | 17:06 | |
| *** dangers has quit IRC | 17:06 | |
| *** dangers has joined #openstack-infra | 17:06 | |
| *** oubiwan__ has quit IRC | 17:06 | |
| fungi | mayu: glad it's working! | 17:07 |
| *** 23LAAA2TD has quit IRC | 17:07 | |
| *** yolanda_ has joined #openstack-infra | 17:07 | |
| mayu | fungi: I will test a job on sandbox | 17:08 |
| *** zul has joined #openstack-infra | 17:09 | |
| *** hemnafk is now known as hemna | 17:09 | |
| *** enikanorov_ has quit IRC | 17:10 | |
| *** che-arne has quit IRC | 17:10 | |
| *** roeyc has quit IRC | 17:10 | |
| *** pmathews has quit IRC | 17:10 | |
| pleia2 | good morning | 17:10 |
| *** pmathews has joined #openstack-infra | 17:10 | |
| jeblair | pleia2: hi there | 17:10 |
| *** thomasem has quit IRC | 17:11 | |
| *** jraim has quit IRC | 17:11 | |
| *** 18WAFV1VI has quit IRC | 17:11 | |
| *** thomasem has joined #openstack-infra | 17:11 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 17:11 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv management into a class https://review.openstack.org/75926 | 17:11 |
| *** davidlenwell has joined #openstack-infra | 17:11 | |
| mordred | I started trying to track down why something in pypi mirror wasn't building correctly and ran straight into a terrible pile of code I wrote | 17:12 |
| mordred | there is an attempt to try to make it suck less - no functional difference intended | 17:12 |
| *** gokrokve has quit IRC | 17:13 | |
| *** wchrisj has joined #openstack-infra | 17:13 | |
| pleia2 | returned from scale, we can apply to current version of sysadmin-codereview: git tag -s -m "SCaLE12x, 2014" 2014-scale12x-sysadmin-codereview | 17:13 |
| pleia2 | or Southern California Linux Expo 12x if we prefer | 17:13 |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added new MagnetoDB project to Stackforge https://review.openstack.org/71308 | 17:14 |
| *** jraim_ has joined #openstack-infra | 17:14 | |
| *** kmartin has joined #openstack-infra | 17:14 | |
| jeblair | pleia2: i will do that | 17:14 |
| pleia2 | jeblair: thank you | 17:14 |
| fungi | jeblair: beat me to it | 17:14 |
| fungi | pleia2: so http://docs.openstack.org/infra/publications/sysadmin-codereview/ is what you presented there? | 17:15 |
| pleia2 | fungi: yep | 17:15 |
| *** jraim_ is now known as jraim | 17:15 | |
| fungi | i still love that photo of our notepads. at this point we'd need at least a third sheet for that | 17:15 |
| *** cadenzajon has joined #openstack-infra | 17:16 | |
| jeblair | pleia2: tag pushed | 17:16 |
| pleia2 | I think elasticsearch and logging could use their own pad at this point | 17:16 |
| *** lcostantino has joined #openstack-infra | 17:17 | |
| anteaya | mordred: rebase needed for both patches | 17:17 |
| *** rfolco has quit IRC | 17:18 | |
| *** rfolco has joined #openstack-infra | 17:18 | |
| fungi | pleia2: jobs worked... http://docs.openstack.org/infra/publications/2014-scale12x-sysadmin-codereview/ | 17:18 |
| *** krotscheck has joined #openstack-infra | 17:18 | |
| fungi | and it appears on the index | 17:18 |
| *** dizquierdo has quit IRC | 17:18 | |
| *** andre__ has quit IRC | 17:20 | |
| pleia2 | \o/ | 17:20 |
| *** openstackgerrit has quit IRC | 17:20 | |
| *** Ryan_Lane1 has joined #openstack-infra | 17:20 | |
| *** openstackgerrit has joined #openstack-infra | 17:20 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git https://review.openstack.org/75931 | 17:21 |
| *** lcostantino has quit IRC | 17:21 | |
| *** andre__ has joined #openstack-infra | 17:21 | |
| anteaya | pleia2: very nice | 17:21 |
| pleia2 | clarkb: had a nice chat with the community manager for elasticsearch, she's hoping we could write an article for their blog about how we use it - if you don't mind, I will take a stab at writing, you can do technical editing and we can co-publish | 17:22 |
| *** lcostantino has joined #openstack-infra | 17:22 | |
| pleia2 | anteaya: thanks | 17:22 |
| *** rfolco_ has joined #openstack-infra | 17:23 | |
| dhellmann | fungi: no idea, the tests are running for me here :-/ | 17:23 |
| mayu | clarkb: hi | 17:24 |
| fungi | dhellmann: right, mostly just wondering what second-or-greater-order dependency for openstackclient is demanding six>=1.5.2 (pretty sure it's nothing we release) | 17:24 |
| *** rfolco has quit IRC | 17:24 | |
| mayu | clarkb: following jay's blog, there is a problem for sandbox test. | 17:25 |
| *** davidhadas__ has quit IRC | 17:25 | |
| *** davidhadas_ has quit IRC | 17:25 | |
| *** davidhadas has quit IRC | 17:25 | |
| dhellmann | fungi: I don't know the full origin of that bug, but when I install openstackclient I get six 1.5.2 -- do you know where it was running with an older version? | 17:26 |
| mayu | clarkb: can you help to analysis what's wrong with it. | 17:26 |
| * dhellmann clicks links | 17:26 | |
| *** rfolco_ has quit IRC | 17:26 | |
| anteaya | mayu jaypipes has been active in #openstack-neutron today, he might still be there | 17:26 |
| fungi | dhellmann: we're testing with 1.4.1 because of a bug in pypi-mirror (which a fix for is in the process of landing) | 17:26 |
| anteaya | clarkb hasn't been around yet, mayu | 17:27 |
| jeblair | mordred: when and how would you like to deal with this change? https://review.openstack.org/#/c/51425/ | 17:27 |
| mayu | anteaya: you are so kind | 17:27 |
| anteaya | thanks | 17:27 |
| *** lcostantino has quit IRC | 17:27 | |
| fungi | dhellmann: but clearly something we're pulling in is sometimes insisting on newer versions of six | 17:27 |
| *** relaxdiego has joined #openstack-infra | 17:27 | |
| dhellmann | fungi: ok, I'll have to look into it further | 17:28 |
| fungi | dhellmann: and more disturbingly from my perspective, only seems to happen when we run jobs in rax-iad (which suggests there's something different about the base ubuntu precise image in that region) | 17:28 |
| dhellmann | fungi: do you have any idea if our six lower bound is accurate, aside from this? | 17:28 |
| jaypipes | mayu: I am here. | 17:28 |
| *** rfolco has joined #openstack-infra | 17:29 | |
| fungi | dhellmann: no clue, other than six==1.4.1 has been working for tests for a while since the openstackclient exceptions only just started cropping up | 17:29 |
| mayu | jaypipes: Following your blog. sandbox test doesn't work | 17:29 |
| jaypipes | mayu: in #openstack-neutron... | 17:29 |
| mordred | jeblair: we're running that change at HP - so I'm comfortable with it - but perhaps we want to wait until puppetboard since it's a pretty pervasive change? | 17:30 |
| mayu | jaypipes: ok | 17:30 |
| *** mestery has joined #openstack-infra | 17:30 | |
| *** terrylhowe has joined #openstack-infra | 17:30 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git https://review.openstack.org/75931 | 17:30 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 17:30 |
| fungi | mordred: jeblair: in which case, we have puppetdb.openstack.org up and running, but ci-puppetmaster still has an outstanding change in review to configure it to start reporting there | 17:31 |
| * fungi finds link | 17:31 | |
| jeblair | mordred: well, you have the access to monitor it yourself, and it fixes a problem we're having in infra, so i don't think it needs to wait for puppetboard. | 17:31 |
| *** nicedice has joined #openstack-infra | 17:31 | |
| jeblair | mordred: but perhaps it should wait until after i3? | 17:31 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror https://review.openstack.org/68818 | 17:31 |
| mordred | jeblair: kk | 17:31 |
| fungi | https://review.openstack.org/#/c/74612/ | 17:31 |
| *** eharney has quit IRC | 17:32 | |
| *** hashar has quit IRC | 17:32 | |
| jeblair | fungi: that change is +2 from me; i want to keep the pupetboard project moving | 17:32 |
| fungi | jeblair: agreed. i just realized i hadn't tried it with --noop on ci-puppetmaster yet to confirm it won't have trouble applying, so doing that now | 17:33 |
| jeblair | mfer: you said "a codebase exists for these" in https://review.openstack.org/#/c/71956/2 would you like to populate the repos in gerrit from an existing git repo somewhere, or do you want them to start empty? | 17:33 |
| *** cadenzajon has quit IRC | 17:34 | |
| *** cadenzajon has joined #openstack-infra | 17:34 | |
| mfer | jeblair we can populate from empty. for a clean log we'll be destroying the history | 17:34 |
| mfer | there were some slip ups in commits of things that's should be in the repo history | 17:35 |
| jeblair | mfer: ok | 17:35 |
| jeblair | fungi, mfer, anteaya: what do i need to know about manage-projects? | 17:36 |
| *** rfolco has quit IRC | 17:36 | |
| *** eharney has joined #openstack-infra | 17:36 | |
| jeblair | sorry, mordred, not mfer ^ | 17:36 |
| *** eharney is now known as Guest6896 | 17:36 | |
| anteaya | jeblair: mordred offered some patches yesterday | 17:36 |
| fungi | jeblair: mordred had some changes proposed. i can't remember whether i've finished reviewing them yet | 17:36 |
| mordred | jeblair: they do not fix any bugs we've seen currently in infra - those atches address an issue I saw running internally - the gitlab patch can be ignored | 17:37 |
| jeblair | :( | 17:37 |
| mordred | jeblair: as of now, I believe we're still at running it and trying to find what's going wrong | 17:37 |
| anteaya | jeblair: this one looks the most relevant: https://review.openstack.org/#/c/75691/ | 17:37 |
| jeblair | then i'll leave them at the bottom of the queue | 17:37 |
| *** vkozhukalov_ has quit IRC | 17:37 | |
| anteaya | mordred: but you have 75691 dependant on the gitlab change | 17:37 |
| mordred | anteaya: I can rebase that | 17:37 |
| anteaya | I think hitting github less would be worth merging | 17:37 |
| *** oubiwann_ has joined #openstack-infra | 17:38 | |
| *** rfolco has joined #openstack-infra | 17:38 | |
| jeblair | mordred, fungi: i'm concerned about that change because people do update descriptions, etc, and expect them to change | 17:38 |
| *** oubiwann_ has quit IRC | 17:39 | |
| fungi | in which case hitting github less means finding a way for manage-projects to know what's changed from one run to the next, and retry things which failed previously | 17:39 |
| mordred | jeblair: I could be convinced otherwise ... we're just hitting github a lot for a reasonably infrequent occurance. I don't feel strongly about this patch | 17:39 |
| jeblair | mordred: it just seems to substitute one problem for another | 17:40 |
| *** jcoufal has joined #openstack-infra | 17:40 | |
| *** oubiwann_ has joined #openstack-infra | 17:40 | |
| *** Guest6896 is now known as eharney | 17:41 | |
| *** lcostantino has joined #openstack-infra | 17:41 | |
| *** yolanda_ has quit IRC | 17:41 | |
| *** oubiwann_ has quit IRC | 17:42 | |
| *** relaxdiego has quit IRC | 17:43 | |
| *** enikanorov has joined #openstack-infra | 17:44 | |
| *** talluri has joined #openstack-infra | 17:44 | |
| *** reed has joined #openstack-infra | 17:44 | |
| *** jgriffith has joined #openstack-infra | 17:44 | |
| *** jang has joined #openstack-infra | 17:44 | |
| *** atiwari has joined #openstack-infra | 17:44 | |
| *** yassine has joined #openstack-infra | 17:44 | |
| *** rcleere has joined #openstack-infra | 17:44 | |
| *** locke105 has joined #openstack-infra | 17:44 | |
| *** ogelbukh has joined #openstack-infra | 17:44 | |
| *** lyxus has joined #openstack-infra | 17:44 | |
| *** asadoughi has joined #openstack-infra | 17:44 | |
| *** mattoliverau has joined #openstack-infra | 17:44 | |
| fungi | nibalizer: a couple of oddities with the puppetdb server... which i've enumerated in a comment on https://review.openstack.org/74612 | 17:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Modernize ATC list format https://review.openstack.org/73347 | 17:46 |
| fungi | nibalizer: if you have ideas/suggestions for how we should be tackling those, it would be most appreciated | 17:46 |
| *** oubiwann_ has joined #openstack-infra | 17:46 | |
| jeblair | has anyone notified tripleo that it looks like we're not getting nodes from their cloud? | 17:47 |
| jeblair | fungi: ? | 17:47 |
| jeblair | (i just joined the tripleo channel so have no backlog) | 17:47 |
| fungi | jeblair: i hadn't noticed. i'll give them a heads up | 17:47 |
| jeblair | fungi: i can do it | 17:47 |
| *** garyk has joined #openstack-infra | 17:48 | |
| fungi | yeah, their /topic usually mentions any problems they're aware of, but in this case it's not talking about anything which i expect should affect the tripleo-ci cloud | 17:48 |
| jeblair | agreed | 17:49 |
| *** lcostantino has quit IRC | 17:50 | |
| *** lcostantino has joined #openstack-infra | 17:50 | |
| *** lcostantino has quit IRC | 17:50 | |
| *** lcostantino has joined #openstack-infra | 17:50 | |
| *** jgriffit2 has joined #openstack-infra | 17:51 | |
| *** chandan_kumar has quit IRC | 17:51 | |
| fungi | the last time this came up, lifeless suggested raising the alarm for everyone listed in http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/tripleo-cloud/tripleo-cd-admins | 17:51 |
| *** sarob_ has joined #openstack-infra | 17:51 | |
| jeblair | i recall, so i did that, approximately at the same time you did i think. :) | 17:52 |
| fungi | yup | 17:52 |
| *** jgriffith has quit IRC | 17:52 | |
| jeblair | it would be cool if they also had highlighting set for "tripleo-admin" or something. :) | 17:52 |
| *** chandan_kumar has joined #openstack-infra | 17:53 | |
| *** reed has quit IRC | 17:53 | |
| *** gokrokve has joined #openstack-infra | 17:53 | |
| fungi | yeah | 17:53 |
| *** sarob has quit IRC | 17:53 | |
| *** reed has joined #openstack-infra | 17:54 | |
| *** jgriffit2 is now known as jgriffith | 17:54 | |
| nibalizer | fungi: sure give me a sec | 17:54 |
| clarkb | morning | 17:54 |
| sdague | so is there any further progress on the six issue? | 17:55 |
| clarkb | pleia2: nice sounds good | 17:55 |
| *** jgallard has quit IRC | 17:55 | |
| *** terrylhowe has left #openstack-infra | 17:55 | |
| *** sarob_ has quit IRC | 17:56 | |
| sdague | clarkb: https://review.openstack.org/#/c/75719/ - I have timestamps on the sublogs now, so I think this is right to get them into ES | 17:56 |
| *** relaxdiego has joined #openstack-infra | 17:56 | |
| mordred | have I mentioned I hate setuptools/easy_install? | 17:57 |
| *** dizquierdo has joined #openstack-infra | 17:57 | |
| fungi | sdague: i'm still unclear on what suddenly started demanding six>=1.5.2 within python-openstackclient and why it's only cropping up in rax-iad, but the fix to pypi-mirror which should get it into our mirror is about 15 minutes out from merging now | 17:57 |
| clarkb | sdague: we need to write logstash rules to parse them too | 17:58 |
| clarkb | sdague: also we should stop logging to console log if we want to consume them that way | 17:58 |
| sdague | clarkb: they should be the same as console | 17:58 |
| fungi | sdague: then the mirror runs need to fire, and i'll keep tabs on them to make sure this solved the failure we were hitting for it | 17:58 |
| jeblair | fungi: can you add https://review.openstack.org/#/c/73157/ to your list? it's not urgent but is ossg related so i want your eyes on it. | 17:58 |
| *** oubiwann_ has quit IRC | 17:59 | |
| sdague | clarkb: so once you get to here in my d-g changes - https://review.openstack.org/#/c/75698/ we stop having duplicate content in the index | 17:59 |
| clarkb | sdague: cool | 17:59 |
| sdague | I matched their date format exactly to console to hopefully make it easy | 17:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Make gantt unit tests voting https://review.openstack.org/74149 | 18:00 |
| fungi | jeblair: sure. to answer your question, i think the reason it ended up targeting the openstack/ repo hierarchy is because the contents may be published to the docs site (i need to review that recent ml thread) | 18:01 |
| *** luqas has quit IRC | 18:01 | |
| *** lcostantino has quit IRC | 18:01 | |
| jeblair | fungi: it was in stackforge/ | 18:01 |
| fungi | add | 18:01 |
| fungi | er, ahh | 18:01 |
| jeblair | fungi: the ossg is like an official thing, yeah? | 18:02 |
| jeblair | actually, are they a program? | 18:02 |
| fungi | yes, i agree with you now that i've reread your comment and refrained from inserting words which you did not use ;) | 18:02 |
| jeblair | fungi: it's okay to insert funny words | 18:02 |
| jeblair | fungi: actually, they aren't a program according to openstack/governance | 18:03 |
| fungi | jeblair: they are not an official program. i think the upshot of the ml thread was that this was a documentation-oriented repo and would be under the docs program umbrella | 18:03 |
| nibalizer | fungi: can you give me 'cat /etc/puppetdb/conf.d/jetty.ini | grep ssl-host' and 'facter -p fqdn' on puppetdb? | 18:03 |
| fungi | which would make anyone publishing security notes a docs contributor | 18:03 |
| *** derekh is now known as derekh_afk | 18:03 | |
| *** jgriffith has quit IRC | 18:03 | |
| fungi | nibalizer: you bet! on it | 18:03 |
| jeblair | fungi: cool, we should probably start keeping an eye out to make sure that openstack/governance gets updated in these cases | 18:04 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 18:04 |
| fungi | jeblair: i'm in full agreement | 18:04 |
| *** sabari_ has joined #openstack-infra | 18:04 | |
| *** julim_ has joined #openstack-infra | 18:04 | |
| *** david_lyle_ is now known as david_lyle | 18:04 | |
| fungi | nibalizer: http://paste.openstack.org/show/69090/ (i included /etc/hosts too because you almost certainly were going to ask for that next) | 18:05 |
| *** lcostantino has joined #openstack-infra | 18:06 | |
| *** julim has quit IRC | 18:07 | |
| *** Ryan_Lane1 has quit IRC | 18:07 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Do not download wheels when running "pip install" https://review.openstack.org/74579 | 18:07 |
| fungi | sdague: ^ that was the probable pypi-mirror fix | 18:07 |
| *** dprince has quit IRC | 18:08 | |
| nibalizer | fungi: wow, so i guess when jetty does a listen on a name, it just finds the first entry in /etc/hosts that matches (which is that ten net addr) and binds to that | 18:08 |
| sdague | fungi: cool, will that trigger now? | 18:08 |
| nibalizer | is that how you read it? | 18:08 |
| fungi | sdague: https://jenkins.openstack.org/job/post-mirror-python26/474/ https://jenkins.openstack.org/job/post-mirror-python27/475/ https://jenkins.openstack.org/job/post-mirror-python33/383/ | 18:08 |
| mordred | jeblair, fungi ^^ YorikSar's patches there are much nicer looking than mine | 18:08 |
| sdague | great | 18:08 |
| fungi | nibalizer: that is precisely my suspicion based on the evidence | 18:09 |
| sdague | because of clean check enforcement that issue basically has kept the gate super small all day - but it's our top issue - http://status.openstack.org/elastic-recheck/index.html | 18:09 |
| fungi | nibalizer: perhaps there's a way to tell jetty to bind to all interfaces? | 18:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow for etherpad title to be parameterized https://review.openstack.org/73751 | 18:10 |
| *** harlowja has joined #openstack-infra | 18:10 | |
| nibalizer | fungi: would you rather I talk you through a one line change, or go through gerrit for it? | 18:10 |
| nibalizer | i think if we put ssl-host = 0.0.0.0 it'll listen on all interfaces, and if it doesn't listen on ipv6 well ... grump | 18:11 |
| *** dripton has quit IRC | 18:11 | |
| clarkb | are we not proxying with something like apache? | 18:11 |
| *** kmartin has quit IRC | 18:11 | |
| fungi | nibalizer: i can test that, but we'll still need to punch it through review once we're sure of the solution | 18:11 |
| nibalizer | absolutely | 18:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Incompatible chrome extension has been fixed https://review.openstack.org/74796 | 18:12 |
| *** nati_ueno has joined #openstack-infra | 18:12 | |
| *** jgriffith has joined #openstack-infra | 18:12 | |
| *** ryanpetrello has joined #openstack-infra | 18:13 | |
| *** e0ne has joined #openstack-infra | 18:13 | |
| nibalizer | clarkb: if you're talking to me, puppetdb just runs its own server using jetty, and we aren't proxying it | 18:14 |
| *** dripton has joined #openstack-infra | 18:14 | |
| clarkb | nibalizer: fungi: maybe we should proxy it? | 18:14 |
| fungi | clarkb: nibalizer: i think i agree. apache will listen on ipv4 and ipv6 addresses. telling jetty to bind to 0.0.0.0 will probably only bind to all ipv4 addresses | 18:15 |
| nibalizer | that wouldn't be trivial, since puppetdb uses client cert authentication, we'd have to terminate the ssl and verify the cert at the apache vhost, then proxy the traffic with another cert to puppetdb | 18:15 |
| clarkb | ugh | 18:15 |
| fungi | ahh, nope, 0.0.0.0 actually listens to all... | 18:16 |
| fungi | tcp6 0 0 :::8081 :::* LISTEN | 18:16 |
| fungi | so this is viable | 18:16 |
| nibalizer | okay, hoorah for specail ip addrs | 18:16 |
| nibalizer | hooray* | 18:16 |
| *** lcostantino has quit IRC | 18:17 | |
| nibalizer | okay i'll fire up a gerrit review to make that change (by the way, what do we call the gerrit pull request things?) | 18:17 |
| nibalizer | clarkb: fungi unless you want to further explore proxying | 18:17 |
| jeblair | nibalizer: 'propose a change' ? | 18:17 |
| clarkb | nibalizer: 'changes' | 18:17 |
| nibalizer | okay, thanks | 18:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow devstack-precise-check for non-voting jobs https://review.openstack.org/71103 | 18:18 |
| *** sabari_ is now known as sabari | 18:19 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:20 | |
| fungi | nibalizer: so that's one of two problems i noticed | 18:20 |
| jeblair | lifeless: https://review.openstack.org/#/c/73130/1 | 18:20 |
| openstackgerrit | A change was merged to openstack-infra/config: python-ceilometerclient: make the py33 gate voting https://review.openstack.org/74875 | 18:20 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting https://review.openstack.org/75949 | 18:20 |
| fungi | nibalizer: i can reach 8081/tcp on puppetdb's ipv4 address now, but ci-puppetmaster actually seems to want to reach puppetdb's ipv6 address instead, even though ci-puppetmaster has no global ipv6 addresses or routes itself | 18:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove depreciated htmlify-screen-log https://review.openstack.org/74241 | 18:21 |
| openstackgerrit | A change was merged to openstack-infra/config: oslo-incubator: gate on py33 https://review.openstack.org/74097 | 18:21 |
| fungi | nibalizer: i can work around that by deleting the aaaa rr for puppetdb.openstack.org but it probably implies a bug in the module | 18:22 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Set puppetdb server to listen on all interfaces https://review.openstack.org/75950 | 18:22 |
| fungi | nibalizer: unless that's just an (unexpected) artifact of testing with --noop, which i would also be willing to believe | 18:22 |
| nibalizer | so, you're testing my change to add puppetdb to the master with noop? and in the diff you're seing an ipv6 address put in puppetdb.yaml or some other file? | 18:24 |
| fungi | nibalizer: i'm seeing a "no route to host" for the connection attempt | 18:25 |
| *** ildikov_ has quit IRC | 18:25 | |
| jeblair | pleia2: what do you think of the comment i left on https://review.openstack.org/#/c/72287/ ? | 18:25 |
| fungi | nibalizer: which strongly suggests that it's trying to reach the ipv6 address (the log only seems to mention the destination by dns name, not ip address) | 18:26 |
| fungi | nibalizer: though it could just be a fluke of testing via --noop | 18:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Add documentation jobs for taskflow https://review.openstack.org/74837 | 18:27 |
| *** kmartin has joined #openstack-infra | 18:27 | |
| pleia2 | jeblair: you're probably right :) | 18:27 |
| pleia2 | I have a fedora server sitting here, so I can actually check | 18:27 |
| *** cadenzajon has quit IRC | 18:28 | |
| *** salv-orlando has joined #openstack-infra | 18:33 | |
| *** johnthetubaguy has quit IRC | 18:34 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git https://review.openstack.org/75931 | 18:34 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 18:34 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 18:34 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting https://review.openstack.org/75949 | 18:34 |
| *** dripton has quit IRC | 18:35 | |
| *** esker has quit IRC | 18:35 | |
| *** rossella-s has quit IRC | 18:35 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix pip on py3k/pypy nodes https://review.openstack.org/75213 | 18:37 |
| *** lcostantino has joined #openstack-infra | 18:37 | |
| jeblair | mordred: updated your change there ^ | 18:38 |
| pleia2 | jeblair: re: https://review.openstack.org/#/c/72872/ should we just register it for them with the openstackinfra account? if they can't even figure out registration I'm not sure we want them with +F anyway :) | 18:38 |
| jeblair | pleia2: oh, no they understand -- the problem is that they _can't_ register it because they don't have ops and there are too many people in channel | 18:39 |
| nibalizer | fungi: are you getting 'Failed to connect to puppetdb; sleeping 2 seconds before retry' | 18:39 |
| *** dkliban_afk is now known as dkliban | 18:39 | |
| pleia2 | jeblair: oh, should I follow up with freenode staff, or is someone handling it? | 18:39 |
| *** yamahata has quit IRC | 18:40 | |
| *** yolanda_ has joined #openstack-infra | 18:40 | |
| fungi | jeblair: i wonder if that channel is available on oftc ;) | 18:41 |
| *** hemna has quit IRC | 18:41 | |
| nibalizer | fungi: so the puppetdb module implements a type/provider to provide a pre-flight check against the puppetdb server inside the puppet code. I've never seen this before. I would say there is a good chance this is noop being useless | 18:41 |
| jeblair | fungi: i already registered it there. :) | 18:41 |
| fungi | nibalizer: makes sense. we can just give it a shot once we merge the jetty config fix | 18:41 |
| jeblair | pleia2: 1 sec | 18:42 |
| nibalizer | excellent | 18:42 |
| *** jgriffit2 has joined #openstack-infra | 18:42 | |
| *** hemna has joined #openstack-infra | 18:42 | |
| *** SpamapS_ is now known as SpamapS | 18:43 | |
| fungi | nibalizer: mainly it was the "no route to host" which worried me, because ci-puppetmaster *does* have a route to the ipv4 address, but not the ipv6 address, and it really looked from the logs like it was actually trying to connect, thus my suspicion... | 18:43 |
| *** sabari has quit IRC | 18:43 | |
| *** mdenny has quit IRC | 18:43 | |
| *** rfolco has quit IRC | 18:43 | |
| *** sabari has joined #openstack-infra | 18:43 | |
| *** SpamapS has quit IRC | 18:43 | |
| *** SpamapS has joined #openstack-infra | 18:43 | |
| *** mdenny has joined #openstack-infra | 18:43 | |
| fungi | nibalizer: though in retrospect, it could also have been trying to connect to the rfc-1918 eth1 interface of the puppetdb server | 18:43 |
| fungi | i didn't fire up tcpdump. but i will if the issue persists | 18:43 |
| openstackgerrit | A change was merged to openstack-infra/config: Add tempest coverage job to post https://review.openstack.org/75250 | 18:44 |
| *** rfolco has joined #openstack-infra | 18:44 | |
| *** dripton_ has joined #openstack-infra | 18:44 | |
| *** jgriffith has quit IRC | 18:44 | |
| jeblair | pleia2: so rather than just getting ops on that one channel, bswartz has graciously attempted to fix the global problem by attempting to help us fix the group registration situation for openstack | 18:45 |
| pleia2 | jeblair: awesome \o/ | 18:45 |
| *** mrmartin has joined #openstack-infra | 18:45 | |
| jeblair | pleia2: sadly, we've been trying and failing to do that for years | 18:45 |
| *** cadenzajon has joined #openstack-infra | 18:45 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 18:45 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting https://review.openstack.org/75949 | 18:45 |
| nibalizer | that would be weird, im used to linux not using v6 if it doesn't have v6 | 18:45 |
| pleia2 | jeblair: yeah, I spoke with another freenode admin about it last week when he was in town, he gave me the epic rundown of their rewrite of the GC system | 18:45 |
| jeblair | pleia2: so the current state is that the staffer that he was working with believes that i am who i say i am, and says that the groups folks are ready to handle it | 18:46 |
| pleia2 | jeblair: I told him to use a spreadsheet :) | 18:46 |
| pleia2 | hehe | 18:46 |
| jeblair | pleia2: but that was last week. | 18:46 |
| * pleia2 nods | 18:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: fix indentation in install-buck macro https://review.openstack.org/75310 | 18:46 |
| jeblair | pleia2: no kidding (spreadsheet) | 18:46 |
| jeblair | pleia2: i would bug them today, except ddos | 18:46 |
| pleia2 | yeah | 18:46 |
| jeblair | pleia2: so i'm thinking let it subside a bit and then ping mquin again (the staffer that mswartz found who has confirmed my identity) | 18:47 |
| pleia2 | jeblair: sounds good, in the meantime do we want to let these patches go through so folks can get bots? | 18:47 |
| jeblair | pleia2: or anyone else who you might know who would actually respond to you :) | 18:47 |
| fungi | nibalizer: i concur, but we have seen that some applications try to open direct sockets anyway, rather than leaving it up to the system libs to decide how to get somewhere, so not entirely outside the realm of possibilities | 18:48 |
| *** dprince has joined #openstack-infra | 18:48 | |
| pleia2 | Corey is responsive, and he works from SF often so we go out for whiskey :) | 18:48 |
| jeblair | pleia2: no; i don't want to have anything to do with any channels we don't have access to | 18:48 |
| pleia2 | jeblair: ok, good | 18:48 |
| jeblair | pleia2: people don't seem to be following the directions to talk to us before creating a channel. maybe we need to rework that. | 18:49 |
| fungi | pleia2: perhaps expense several whiskies for him and then turn him loose with his netops hat on | 18:49 |
| pleia2 | fungi: haha | 18:49 |
| *** esker has joined #openstack-infra | 18:49 | |
| *** leifmadsen has quit IRC | 18:50 | |
| *** dizquierdo has quit IRC | 18:51 | |
| *** rlandy_ has quit IRC | 18:51 | |
| clarkb | fungi: I like this idea :P | 18:54 |
| *** chenxu_ has joined #openstack-infra | 18:56 | |
| chenxu_ | folks, my patch is failing jenkins test on totally unrelated errors: https://review.openstack.org/#/c/75427/ | 18:56 |
| chenxu_ | I only touched vif related code, but the check-grenade-dsvm is failing on volume related stuff | 18:57 |
| chenxu_ | any ideas? | 18:57 |
| *** yolanda_ has quit IRC | 18:59 | |
| *** coolsvap has joined #openstack-infra | 18:59 | |
| anteaya | chenxu_: https://bugs.launchpad.net/openstack-ci/+bug/1276242 | 19:02 |
| anteaya | ERROR: openstackclient.shell Exception raised: (six 1.4.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('six>=1.5.2')) | 19:03 |
| fungi | anteaya: are you sure it's not that the fix for bug 1223469 needs to be backported to stable/havana? | 19:03 |
| anteaya | http://logs.openstack.org/27/75427/8/check/check-tempest-dsvm-cells-full/716bc54/console.html | 19:03 |
| *** jlibosva has joined #openstack-infra | 19:03 | |
| fungi | anteaya: that's non-voting | 19:03 |
| anteaya | then I am wrong | 19:03 |
| anteaya | and no, I am not sure | 19:03 |
| fungi | chenxu_: https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures | 19:04 |
| jeblair | SpamapS: https://review.openstack.org/#/c/71743/1 | 19:04 |
| fungi | anteaya: actually, you did have the right bug, it's just cropping up in a more obscure place than we usually see it! http://logs.openstack.org/27/75427/8/check/check-grenade-dsvm/1a9a198/logs/old/screen-c-vol.txt.gz | 19:05 |
| fungi | Stderr: 'Traceback (most recent call last):\n File "/usr/local/bin/cinder-rootwrap", line 5, in <module>\n from pkg_resources import require\n File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2720, in <module>\n iter_entry_points = working_set.iter_entry_points\n File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 592, in resolve\n return to_activate # return list of | 19:05 |
| fungi | distros to activate\npkg_resources.VersionConflict: (six 1.4.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse(\'six>=1.5.2\'))\n' | 19:05 |
| *** amrith has joined #openstack-infra | 19:06 | |
| anteaya | correct by fluke | 19:06 |
| *** rlandy_ has joined #openstack-infra | 19:07 | |
| fungi | anteaya: however it gives us a new clue as to where the six>=1.5.2 is coming from, i think | 19:07 |
| anteaya | yay | 19:07 |
| anteaya | more data | 19:07 |
| sdague | oh... ffs | 19:08 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:08 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/73709 | 19:08 |
| sdague | is this because oslo.rootwrap isn't in the gr sync? | 19:08 |
| fungi | that's what i was checking next | 19:08 |
| sdague | yeh, it's not in projects.txt | 19:08 |
| *** rcarrillocruz has joined #openstack-infra | 19:09 | |
| *** esker has quit IRC | 19:09 | |
| fungi | however it's got >=1.4.1 | 19:09 |
| fungi | http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/requirements.txt | 19:09 |
| rcarrillocruz | guys, it seems there's no way to change the gerrit username, the one that is linked in my launchpad profile . Who should I contact, do you know? | 19:09 |
| fungi | rcarrillocruz: why do you need to change it? | 19:09 |
| fungi | rcarrillocruz: you can tell git what ssh username to use for a particular remote host | 19:10 |
| rcarrillocruz | for the sake of consistency, i changed my launchpad id, but that didn't change the gerrit username, it still refers to the old username | 19:10 |
| jeblair | clarkb: do you want to look at https://review.openstack.org/#/c/68042/ | 19:11 |
| fungi | rcarrillocruz: gerrit removed the ability for users to change their ssh username once it's set. that was way back in gerrit 2.1 i think | 19:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Added requirements gates for Murano repositories https://review.openstack.org/75451 | 19:11 |
| zaro | morning. | 19:11 |
| *** ok_delta has joined #openstack-infra | 19:11 | |
| anteaya | morning zaro | 19:11 |
| anteaya | how was montreal? | 19:11 |
| rcarrillocruz | yay | 19:11 |
| rcarrillocruz | k, nm then, thanks fungi | 19:12 |
| zaro | anteaya: not there | 19:12 |
| openstackgerrit | A change was merged to openstack-infra/config: run-unittests.sh: fix a typo https://review.openstack.org/74941 | 19:12 |
| anteaya | zaro: weren't you at confoo? | 19:12 |
| reed | nagging for review : https://review.openstack.org/#/c/69636/ | 19:12 |
| * anteaya is confused | 19:12 | |
| reed | fungi, jeblair: pretty please :) | 19:13 |
| rcarrillocruz | have you seen codysomerville online today? | 19:13 |
| zaro | anteaya: it's this week. | 19:13 |
| chenxu_ | https://bugs.launchpad.net/nova/+bug/1240728 | 19:13 |
| *** sarob has joined #openstack-infra | 19:13 | |
| chenxu_ | this seems more directly related | 19:13 |
| clarkb | jeblair: sure, I am almost done writing a simple logstash watchdog, will review once I have pushed this change | 19:13 |
| *** chandan_kumar has quit IRC | 19:14 | |
| chenxu_ | fungi: ^^ | 19:14 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:14 | |
| SpamapS | jeblair: not sure why I'm attached to https://review.openstack.org/#/c/71743/1 | 19:14 |
| anteaya | zaro: ah, safe travels to you then | 19:14 |
| jeblair | reed: i'm reviewing it now, actually | 19:15 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:15 | |
| reed | jeblair, wonderful | 19:15 |
| * reed keeps fingers crossed | 19:15 | |
| jeblair | reed: sadly, i'm having trouble asking gerrit to diff patchset 7-13 | 19:15 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest https://review.openstack.org/75960 | 19:15 |
| *** MarkAtwood has joined #openstack-infra | 19:15 | |
| fungi | chenxu_: http://logs.openstack.org/27/75427/8/check/check-grenade-dsvm/1a9a198/logs/old/screen-c-vol.txt.gz looks like what we're trying to fix in https://launchpad.net/bugs/1276242 | 19:16 |
| jeblair | so it's going a bit slower than i would like | 19:16 |
| harlowja | has anyone seen the following, maybe just something simple wrong, http://logs.openstack.org/20/54220/77/check/gate-taskflow-docs/a376650/console.html | 19:17 |
| chenxu_ | fungi: I was referring to this one http://logs.openstack.org/27/75427/8/check/check-grenade-dsvm/1a9a198/console.html | 19:17 |
| harlowja | error: 'source_dir' must be a directory name (got `/home/jenkins/workspace/gate-taskflow-docs/doc/source`) | 19:17 |
| harlowja | this source_dir is actually a directory | 19:17 |
| chenxu_ | fungi: which has this line 2014-02-24 17:07:07.204 | FAIL: setUpClass (tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON) | 19:17 |
| chenxu_ | can I do two rechecks on two different bugs? | 19:17 |
| fungi | chenxu_: you can, though you'd do it in separate comments (can't put them both in the same comment) | 19:18 |
| harlowja | dhellmann do u have any idea about the above, if u are free | 19:18 |
| chenxu_ | fungi: cool, thx | 19:18 |
| dhellmann | harlowja: there's a known distutils bug related to that | 19:18 |
| harlowja | ah | 19:18 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Add token header to requests https://review.openstack.org/75961 | 19:18 |
| * dhellmann looks for link | 19:18 | |
| harlowja | dhellmann ya, odd that it works locally fine | 19:18 |
| dhellmann | harlowja: http://bugs.python.org/issue19570 | 19:19 |
| dhellmann | harlowja: it has to do with filesystem encoding or something, I think | 19:19 |
| harlowja | hmmm | 19:19 |
| harlowja | k | 19:19 |
| harlowja | weird | 19:19 |
| dhellmann | sometimes the filename is a unicode string instead of a bytestring | 19:19 |
| zaro | clarkb: now i know why upstream gerrit doesn't run tests in their CI.. https://review.openstack.org/#/c/75769 | 19:20 |
| dhellmann | harlowja: the fix is to ensure that the right version of sphinx is used, and not allow the beta in | 19:20 |
| harlowja | dhellmann, k, let me repoke this, perhaps the requirements adjustment that went in fixed it | 19:20 |
| *** rfolco has quit IRC | 19:20 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori as a new project https://review.openstack.org/73667 | 19:21 |
| *** resker has joined #openstack-infra | 19:22 | |
| clarkb | zaro: nice | 19:23 |
| * anteaya goes for a walk | 19:24 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add a simple watchdog for logstash-indexer service https://review.openstack.org/75966 | 19:25 |
| clarkb | jeblair: ^ | 19:25 |
| clarkb | now I will review horizon tx change | 19:25 |
| fungi | sdague: anyway, we've got a separate bug (probably a bug in pycparser's packaging/setup) preventing pypi-mirror from making it through the grizzly branch run and moving on to later branches. though i'm unclear on why it's finding cliff in the requirements list on the grizzly branch... http://logs.openstack.org/0d/0de0e10e753d00f9440a7eb9b5699a3e5a167ede/post/post-mirror-python33/1a7e9ad/console.html | 19:25 |
| *** markmcclain has joined #openstack-infra | 19:25 | |
| *** vkozhukalov_ has joined #openstack-infra | 19:25 | |
| jeblair | sdague: in https://review.openstack.org/#/c/75683/6 i don't understand why you matched the console log format instead of the devstack log format? | 19:26 |
| fungi | unless it's not actually checking out stable/grizzly and just running on master after all | 19:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Set a ratelimit for tripleo-test-cloud. https://review.openstack.org/68510 | 19:27 |
| jeblair | sdague: surely since you also want to index the devstack log, we shoud have all the sublogs use that format instead? | 19:27 |
| clarkb | wait we want to index the devstack log? | 19:27 |
| jeblair | clarkb: why not? we do now by virtue of having it in the console log | 19:27 |
| clarkb | did that end up on the console log before? | 19:27 |
| clarkb | oh right its d-g logs that don't end up in console log | 19:27 |
| jeblair | clarkb: so if we remove it from the console log, we should index it | 19:28 |
| clarkb | ya | 19:28 |
| fungi | oic, xattr requires cffi and not the other way around | 19:29 |
| sdague | jeblair: mostly readability and the fact that I knew the format was already parsed | 19:29 |
| jeblair | sdague: what's the plan for devstack logs then? | 19:30 |
| sdague | I stared at the logs this weekend, and felt the | helped | 19:30 |
| jeblair | sdague: do you want to change the format in devstack to match this then? | 19:30 |
| sdague | was going to see what dtroyer thought of it | 19:30 |
| sdague | I did match the time resolution | 19:31 |
| sdague | https://review.openstack.org/#/c/75689/ | 19:31 |
| jeblair | sdague: that's fair. ultimately, i'd like d-g and devstack to have the same format. | 19:31 |
| openstackgerrit | A change was merged to openstack-infra/config: remove flaky gerrit tests https://review.openstack.org/75769 | 19:31 |
| *** hashar has joined #openstack-infra | 19:31 | |
| sdague | sure | 19:32 |
| dtroyer | I think you guys are the ones that changing the devstack log format affects the most... | 19:32 |
| jeblair | dtroyer: fortunately, i think we can change the timestamp format right now without affecting us, since i believe we're actually getting the timestamps from the console log | 19:32 |
| dtroyer | I hadn't looked at that change yet…will after this meeting | 19:33 |
| sdague | yep | 19:33 |
| *** rfolco has joined #openstack-infra | 19:33 | |
| jeblair | so a switch in devstack to match that format, then to have logstash start parsing/indexing the separate logfile instead of console should be smooth | 19:33 |
| sdague | dtroyer: so the remaining question is just about adding the | , though realistically I'd rather not change that format until we actually get close to that patch landing, just so we don't double pipe people for a long time | 19:34 |
| sdague | so I guess that brings up the question of the rest of the series | 19:34 |
| harlowja | clarkb was there a bug created for that mysql permission issue that happened friday (if u remember) | 19:34 |
| clarkb | harlowja: I didn't create one | 19:35 |
| harlowja | k, np | 19:35 |
| harlowja | creating one | 19:35 |
| clarkb | harlowja: though its a simple change if we just want to give the openstack_citest user ALL privs | 19:35 |
| clarkb | mordred: ^ | 19:35 |
| *** davidhadas has joined #openstack-infra | 19:35 | |
| harlowja | brb, one sec, fogot a meeting | 19:35 |
| harlowja | lol | 19:35 |
| jeblair | sdague: https://review.openstack.org/#/c/75719/ looks like it currently assumes the devstack log is the same format | 19:35 |
| *** davidhadas__ has joined #openstack-infra | 19:36 | |
| *** davidhadas_ has joined #openstack-infra | 19:36 | |
| *** dolphm is now known as dolphm_503 | 19:37 | |
| sdague | how is console.html getting tagged? | 19:38 |
| *** yolanda_ has joined #openstack-infra | 19:39 | |
| clarkb | jeblair: reviewed horizon change, let me know if you think my comment doesn't need to be addressed | 19:39 |
| Steap | How often is the OpenStack PyPI mirror udpated ? | 19:40 |
| jeblair | clarkb: seems reasonable | 19:40 |
| jeblair | Steap: at least daily or as needed by requirements changes or openstack-related releases | 19:41 |
| *** dstanek_afk has joined #openstack-infra | 19:41 | |
| jeblair | sdague: what do you mean? | 19:41 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 19:42 | |
| sdague | so it looks like on the logstash side we need to tag things as console.html to get the log parsing to work anyway | 19:42 |
| Steap | jeblair: ok, thanks :) | 19:42 |
| sdague | now I'm curious where that will happen | 19:42 |
| *** jcooley_ has joined #openstack-infra | 19:42 | |
| *** davidhadas has quit IRC | 19:42 | |
| *** jlibosva has quit IRC | 19:42 | |
| sdague | I also updated this with the additional format - https://review.openstack.org/#/c/75689/ | 19:43 |
| fungi | sdague: eureka! i overlooked that the failing changes are all on devstack-precise-check-rax-iad nodes (not devstack-precise-rax-iad) which we know have somewhat different and more stripped-down images. so chances are the difference triggering this is that something which isn't installed system-wide is getting updated by devstack instead | 19:43 |
| *** dstanek has quit IRC | 19:43 | |
| sdague | fungi: oh, interesting | 19:43 |
| sdague | any idea what that might be? | 19:44 |
| *** oubiwann_ has joined #openstack-infra | 19:44 | |
| lifeless | jeblair: controller node just threw an NMI | 19:44 |
| fungi | sdague: it's a fairly long list of debs on normal images which aren't on the pvhvm images. a couple dozen are python-.* debs too. hard to say what the trigger is | 19:44 |
| lifeless | jeblair: rebooting and we'll see if its a kernel / driver issue (and thus cleared) or deeper hardware fault | 19:44 |
| zaro | sdague: so your looking to query for the build timeout? | 19:45 |
| sdague | zaro: I'd love to :) | 19:45 |
| sdague | instead of us having to add it explicitly to every job | 19:45 |
| *** jgrimm has quit IRC | 19:46 | |
| zaro | sdague: isn't already added to every job? how do you have a timeout without adding to a job? | 19:47 |
| zaro | sdague: ohh you mean using the envinject it into every job? | 19:48 |
| sdague | zaro: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#L7 | 19:48 |
| sdague | and https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#L16 | 19:48 |
| sdague | we effectively duplicate the same static piece of info for every job | 19:49 |
| sdague | so that we can do the inner timeout | 19:49 |
| *** hashar has quit IRC | 19:49 | |
| fungi | what if jjb had a parameter which turned on a feature to duplicate the entire job configuration in an envvar passed to the worker via envinject? | 19:51 |
| clarkb | fungi: re those different images. If we can make the pvhvm image happy we should kill the -check images and use pvhvm everywhere | 19:51 |
| sdague | that work work | 19:51 |
| sdague | would work | 19:51 |
| fungi | clarkb: yeah, but now we have evidence that it's weird in some ways | 19:51 |
| sdague | honestly, I don't care how it gets there, or even what it's called once it's there. I just find it silly that we are duping it :) | 19:51 |
| fungi | clarkb: would be nicer if rackspace would make them more consistent | 19:52 |
| clarkb | that++ | 19:52 |
| jeblair | jnoller: ^ pvhvm images != normal images == not entirely expected behavior | 19:53 |
| clarkb | sdague: to be fair we are duping timeouts | 19:53 |
| clarkb | sdague: not just the values but the implementations | 19:53 |
| *** garyk has quit IRC | 19:53 | |
| fungi | so that we don't get strange differences in userspace like we ran into here | 19:53 |
| *** dolphm_503 is now known as dolphm | 19:53 | |
| sdague | clarkb: right, but if I included some python code that had 2 constants defined for the same purpose, you'd -1 that | 19:53 |
| sdague | anyway, this isn't a high priority thing | 19:53 |
| fungi | jnoller: right... the ubuntu precise pvhvm and non-pvhvm images... do a dpkg -l in both and see all teh extra stuff which is installed by default on the latter | 19:53 |
| sdague | it was just an annoyance once I started staring at this yesterday | 19:54 |
| sdague | so I figured I'd ask if there was a way to clean it up | 19:54 |
| clarkb | sdague: I'd -1 because there were two separate implementation of the same thing | 19:54 |
| clarkb | but we can't get away from that because EJENKINS | 19:54 |
| sdague | I guess | 19:54 |
| sdague | duping constants still seems silly | 19:54 |
| sdague | but in reality - getting this in will help much more than fixing the duplication issue - https://review.openstack.org/#/c/75726/ | 19:55 |
| sdague | because, it actually turns out our inner time outs are all wrong anyway | 19:56 |
| *** thomasem has quit IRC | 19:56 | |
| zaro | that was the work around i was thinking of but i agree it's not a good thing. | 19:56 |
| sdague | and they aren't actually working like anyone thinks they are | 19:56 |
| clarkb | :/ the delta between them is too small? | 19:56 |
| sdague | yep | 19:57 |
| sdague | but more importantly, the timeout is just wrapped around the gate_hook | 19:57 |
| sdague | and there is 3-8minutes setup time before that, and 3-4 minutes clean up | 19:58 |
| zaro | sdague: i don't know a good solution off the top of my head. but if you think it's something to work on. enter a bug and i'll look into it. | 19:58 |
| sdague | so we need to stop doing the math in the job template, just pass the same value down to the jobs, calculate how much time is left before we start the timer, and cut the 5 minutes off for cleanup in d-g | 19:58 |
| sdague | which is basically 75726 | 19:59 |
| *** thomasem has joined #openstack-infra | 19:59 | |
| *** dolphm is now known as dolphm_503 | 19:59 | |
| *** rossella-s has joined #openstack-infra | 19:59 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Don't blow up on a MemoryError https://review.openstack.org/75973 | 19:59 |
| *** hashar has joined #openstack-infra | 20:00 | |
| *** nati_uen_ has joined #openstack-infra | 20:00 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add a simple watchdog for logstash-indexer service https://review.openstack.org/75966 | 20:01 |
| clarkb | that should pass check tests | 20:01 |
| SergeyLukjanov | fungi, mordred, clarkb, jeblair, folks, what do you think about adding more templates in layout.yaml? like check-requrements, tarballs, pypi-release | 20:01 |
| SergeyLukjanov | we already have docs template with two jobs, and IMO it could make layout.yaml much simpler | 20:02 |
| openstackgerrit | A change was merged to openstack-infra/config: Uses python-jobs template in zuul for python-tuskarclient. https://review.openstack.org/74757 | 20:02 |
| fungi | SergeyLukjanov: i think that was intended, but nobody's had time to create them yet | 20:02 |
| *** tjones has quit IRC | 20:02 | |
| *** dstanek_afk is now known as dstanek | 20:02 | |
| SergeyLukjanov | fungi, ok, if mordred, clarkb, jeblair will not have any objection, I'll start doing it | 20:02 |
| *** tjones has joined #openstack-infra | 20:03 | |
| *** ArxCruz has quit IRC | 20:03 | |
| *** pdmars_ has joined #openstack-infra | 20:03 | |
| *** nati_ueno has quit IRC | 20:03 | |
| *** cody-somerville has joined #openstack-infra | 20:03 | |
| *** cody-somerville has quit IRC | 20:03 | |
| *** cody-somerville has joined #openstack-infra | 20:03 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 20:05 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting https://review.openstack.org/75949 | 20:05 |
| *** pdmars has quit IRC | 20:06 | |
| sdague | fungi: can you kick openstackrecheck bot | 20:06 |
| sdague | I expect in all the netsplits it didn't recover well | 20:07 |
| sdague | also, clarkb we're still about 2 hrs behind on indexing? - http://logstash.openstack.org/#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjQzMjAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzI3MjM2NDM0N30= | 20:07 |
| *** SumitNaiksatam has joined #openstack-infra | 20:07 | |
| *** tjones has quit IRC | 20:07 | |
| *** chenxu_ has quit IRC | 20:08 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Don't blow up on a MemoryError https://review.openstack.org/75973 | 20:08 |
| *** dolphm_503 is now known as dolphm | 20:09 | |
| *** pdmars has joined #openstack-infra | 20:09 | |
| *** pdmars_ has quit IRC | 20:09 | |
| jeblair | SergeyLukjanov: ++ | 20:10 |
| * jeblair -> lunch | 20:10 | |
| *** dangers is now known as dangers_away | 20:10 | |
| sdague | SergeyLukjanov: yeh, I was thinking about those layout changes when I was trying to drop the unit tests. Because it would be good to make an integrated gate template that includes the basic devstack and grenade jobs we expect everyone to run | 20:12 |
| SergeyLukjanov | sdague, added to my backlog | 20:12 |
| SergeyLukjanov | sdague, good idea | 20:12 |
| mordred | SergeyLukjanov: sounds awesome | 20:12 |
| sdague | SergeyLukjanov: awesome | 20:13 |
| *** tomhe has quit IRC | 20:13 | |
| *** dstanek_afk has joined #openstack-infra | 20:15 | |
| *** dstanek has quit IRC | 20:15 | |
| *** dstanek_afk is now known as dstanek | 20:16 | |
| jog0_ | sdague: ping | 20:18 |
| sdague | pong | 20:18 |
| jog0_ | sdague: so the grenade changes | 20:19 |
| sdague | yeh | 20:19 |
| sdague | it seems super weird that the service lists are hard coded to me | 20:19 |
| jog0_ | sdague: agreed | 20:20 |
| sdague | is there a way not to do that that isn't crazy hard? | 20:20 |
| jog0_ | I am going to fix that one | 20:20 |
| sdague | ok, cool | 20:20 |
| *** jog0_ is now known as jog0 | 20:20 | |
| jog0 | sdague: I think there is | 20:20 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 20:20 |
| sdague | ok, so lets do that. I realistically don't want to hold this one up too much, because it would be useful to nova to gate on it | 20:21 |
| jog0 | so Ithink i did it that way because of https://review.openstack.org/#/c/63965/13/upgrade-nova | 20:22 |
| jog0 | so you had some further concerns about https://review.openstack.org/#/c/64074/ | 20:22 |
| sdague | so what's the job definition for what's going to run this? | 20:23 |
| jog0 | so funny you should mention that... that code landed already | 20:23 |
| sdague | link? | 20:23 |
| jog0 | https://review.openstack.org/#/c/64200/ | 20:24 |
| jog0 | https://review.openstack.org/#/c/73460/ | 20:24 |
| *** mrda_away is now known as mrda | 20:24 | |
| sdague | no tempest? | 20:24 |
| *** Sukhdev has joined #openstack-infra | 20:25 | |
| jog0 | nope | 20:25 |
| jog0 | as no no tempest patch needed for this | 20:25 |
| jog0 | as in* | 20:25 |
| *** lcostantino has quit IRC | 20:26 | |
| Sukhdev | folks - I am seeing the following error when i clone nova - any idea why? ---- 2014-02-24 12:01:03 + git clone git://git.openstack.org/openstack/nova.git /opt/stack/nova | 20:26 |
| Sukhdev | 2014-02-24 12:01:16 Cloning into '/opt/stack/nova'... | 20:26 |
| Sukhdev | 2014-02-24 12:02:16 fatal: The remote end hung up unexpectedly | 20:26 |
| Sukhdev | 2014-02-24 12:02:16 fatal: early EOF | 20:26 |
| Sukhdev | 2014-02-24 12:02:16 fatal: recursion detected in die handler | 20:26 |
| SpamapS | sdague: Installed /opt/stack/python-glanceclient | 20:26 |
| SpamapS | Processing dependencies for python-glanceclient==0.12.0.56.gb8a850c | 20:26 |
| SpamapS | error: Installed distribution six 1.4.1 conflicts with requirement six>=1.5.2 | 20:26 |
| *** lcostantino has joined #openstack-infra | 20:26 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix log Footer README's for 'check-tempest-dsvm' https://review.openstack.org/74238 | 20:26 |
| sdague | SpamapS: yeh, so we are still trying to figure out what is calling for that version | 20:26 |
| sdague | and failing | 20:26 |
| SpamapS | oh so this isn't actually that the mirror has wheel issues? | 20:27 |
| sdague | so the issue might have moved after that | 20:27 |
| fungi | SpamapS: yeah, it's a twofold problem | 20:27 |
| sdague | you guys have a super minimal imagE? | 20:27 |
| sdague | image | 20:27 |
| *** gokrokve has quit IRC | 20:27 | |
| fungi | SpamapS: well, threefold now because we can't build an updated mirror for other reasons, so i've gone off in a corner to reproduce it and see if i can fix it now | 20:27 |
| *** gokrokve has joined #openstack-infra | 20:28 | |
| *** cadenzajon has quit IRC | 20:28 | |
| *** mwagner_bbl is now known as mwagner_wfh | 20:28 | |
| SpamapS | sdague: if you checkout diskimage-builder and tri[Cpleo-image-elements you can reproduce with just 'disk-image-create -a i386 -u ubuntu openstack-clients' .. I think.. testing that theory now. | 20:28 |
| sdague | so one of our dependencies, which is already installed by a basic precise server install, isn't in the phvm images on rax | 20:29 |
| fungi | SpamapS: if we knew what was demanding six>=1.5.2, it would help ease some related concerns at least | 20:29 |
| sdague | so when pip installs the latest and greatest version of it, it requires new six | 20:29 |
| *** resker has quit IRC | 20:29 | |
| sdague | or at least that's my guess | 20:30 |
| sdague | however, the logs have not been very helpful in figuring that out | 20:30 |
| sdague | if yuo have a broken system can you grep the python tree to find what's requiring new six? | 20:30 |
| SpamapS | sdague: Yeah one second | 20:30 |
| fungi | i can at least confirm that pypi-mirror itself is caching six-1.5.2.tar.gz | 20:32 |
| fungi | when i run it manually | 20:32 |
| *** gokrokve has quit IRC | 20:32 | |
| *** ildikov_ has joined #openstack-infra | 20:33 | |
| fungi | still waiting to see if it tickles the same error we're hitting when it's run within the jenkins job | 20:33 |
| anteaya | Sukhdev: I just checked my nova repo from git.o.o and was able to update correctly | 20:34 |
| anteaya | Sukhdev: are you using a proxy? | 20:34 |
| SpamapS | sdague: Ok I have a broken chroot | 20:35 |
| SpamapS | sdague: with a broken virtualenv ... | 20:35 |
| *** harlowja is now known as harlowja_away | 20:35 | |
| sdague | SpamapS: well, you guys have a broken that looks like our broken | 20:35 |
| jog0 | sdague: looping back to https://review.openstack.org/#/c/64074/12 I am going to make the ENABLED_SERVICES change , but you had some other concerns on that patch | 20:35 |
| SpamapS | ./python-glanceclient/lib/python2.7/site-packages/pyOpenSSL-0.14-py2.7.egg-info/requires.txt:six>=1.5.2 | 20:35 |
| SpamapS | sdague: ^^ | 20:36 |
| sdague | bingo | 20:36 |
| SpamapS | bingo! | 20:36 |
| SpamapS | hahahah | 20:36 |
| SpamapS | I said the same thing out loud | 20:36 |
| fungi | SpamapS: sdague: yep, that's one of the system-installed debs on the non-pvhvm precise images | 20:36 |
| *** DinaBelova is now known as DinaBelova_ | 20:37 | |
| *** ok___delta has joined #openstack-infra | 20:37 | |
| *** enikanorov_ has joined #openstack-infra | 20:37 | |
| sdague | hmmm... no that doesn't help | 20:37 |
| SpamapS | 0.14 uploaded yesterday | 20:37 |
| *** malini is now known as malini_afk | 20:38 | |
| *** enikanorov has quit IRC | 20:38 | |
| *** kynes has joined #openstack-infra | 20:39 | |
| sdague | hmmmm.... but why only the rax nodes hitting this? | 20:39 |
| sdague | jog0: so, actually, I think ENABLED_SERVICES is all I need on it | 20:40 |
| sdague | I was thinking about patch 2 incorrectly | 20:40 |
| jog0 | kk | 20:40 |
| sdague | I +Aed patch #2 | 20:40 |
| fungi | grr... when i run it by hand i get pypimirror/mirror/openstack/six/six-1.5.2.tar.gz | 20:41 |
| fungi | i'm tempted to move aside the local caches on the mirror27 slave and try retriggering that update | 20:43 |
| *** coolsvap has quit IRC | 20:43 | |
| fungi | i was completely unable to reproduce the traceback in http://logs.openstack.org/0d/0de0e10e753d00f9440a7eb9b5699a3e5a167ede/post/post-mirror-python33/1a7e9ad/console.html | 20:43 |
| SpamapS | so.. is this going to be another one where we pin everybody to pyOpenSSL <=0.13 ? | 20:44 |
| fungi | er, in http://logs.openstack.org/0d/0de0e10e753d00f9440a7eb9b5699a3e5a167ede/post/post-mirror-python27/74581bc/console.html whatever | 20:44 |
| *** ok_delta has quit IRC | 20:44 | |
| fungi | SpamapS: no, i think once pypi-mirror is getting six 1.5.1 properly installed into our mirror, it should sort itself out | 20:44 |
| anteaya | fungi is Client disconnected before sending all data to backend a manifestation of the mirror error? | 20:44 |
| fungi | er, 1.5.2 | 20:44 |
| anteaya | https://jenkins05.openstack.org/job/gate-tempest-dsvm-large-ops/3148/console | 20:44 |
| fungi | anteaya: i have no idea what you said there ;) | 20:45 |
| anteaya | I am seeing an error in https://jenkins05.openstack.org/job/gate-tempest-dsvm-large-ops/3148/console | 20:45 |
| *** jgriffit2 is now known as jgriffith | 20:45 | |
| anteaya | part of the output is Client disconnected before sending all data to backend | 20:45 |
| *** lcheng has quit IRC | 20:45 | |
| anteaya | I am wondering if this is a manifestation of the mirror error | 20:45 |
| anteaya | it is affecting a nova patch in the gate | 20:46 |
| anteaya | sorry was trying to be consise and I missed | 20:46 |
| fungi | anteaya: looks like glance --os-auth-token is failing. no idea why. would need to dig into the glance logs for that job next, probably | 20:47 |
| openstackgerrit | Sahdev Zala proposed a change to openstack-infra/config: New StackForge project heat-translator https://review.openstack.org/75988 | 20:47 |
| anteaya | okay I will look in glance bugs to see if it is known | 20:47 |
| *** relaxdiego has quit IRC | 20:48 | |
| *** dprince has quit IRC | 20:49 | |
| clarkb | back from lunch | 20:49 |
| anteaya | so far no known glance --os-auth-token bugs filed that I can find | 20:51 |
| fungi | anteaya: you probably need to look into the glance screen logs to see what error they're actually hitting, and then possibly recurse into screen logs for other services as indicated | 20:52 |
| *** nati_uen_ has quit IRC | 20:52 | |
| *** pdmars_ has joined #openstack-infra | 20:52 | |
| *** tjones has joined #openstack-infra | 20:52 | |
| *** ok___delta has quit IRC | 20:53 | |
| clarkb | sdague: yes we are way behind on indexing, I guess you missed the comments late last week | 20:53 |
| sdague | fungi: I guess given that neither image has python-openssl, I'm still confused why we hit it on only the phvm images | 20:53 |
| anteaya | I'm not sure what changes I need to do to the url to get those logs | 20:53 |
| clarkb | sdague: tl;dr is we index too much stuff and our cluster is too small | 20:53 |
| sdague | clarkb: I guess I did | 20:53 |
| clarkb | sdague: are are now indexing about as much data as we were before we stopped indexing DEBUG level logs | 20:53 |
| clarkb | sdague: which is unhappy times | 20:53 |
| sdague | interesting | 20:53 |
| sdague | any idea what's filling that up? | 20:53 |
| *** emagana_ has joined #openstack-infra | 20:54 | |
| *** tjones has quit IRC | 20:54 | |
| clarkb | sdague: I think console logs mostly | 20:54 |
| *** tjones has joined #openstack-infra | 20:54 | |
| clarkb | but also possibly the new libvirt logs, though they seemed relatively small | 20:54 |
| clarkb | sdague: I think its more volume | 20:54 |
| clarkb | we are running a lot of tests especially with the new check rules | 20:54 |
| anteaya | fungi: how do I get to teh screen logs for a job that is still in the gate? | 20:54 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 20:54 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor virtualenv resetting https://review.openstack.org/75949 | 20:54 |
| *** pdmars has quit IRC | 20:55 | |
| clarkb | sdague: curent vague plan is to work on using rax performance nodes that are bigger than our current nodes to incrase the size of the cluster | 20:55 |
| *** nati_ueno has joined #openstack-infra | 20:55 | |
| *** pdmars_ has quit IRC | 20:56 | |
| fungi | anteaya: look at the end where it says that it's copying logs to the logserver, but change everything up through /srv/static/logs/ to http://logs.openstack.org/ | 20:56 |
| mattoliverau | clarkb and/or fungi would one of you take a look at http://is.gd/6EAYkf it's for a guy here at work :) | 20:56 |
| *** pdmars has joined #openstack-infra | 20:56 | |
| clarkb | sdague: anywas we are seeing a lot of memory pressure in the cluster and lots of cache evictions and so on, this appears to be affecting the rate at which we can index | 20:56 |
| anteaya | fungi: thanks, will try | 20:57 |
| clarkb | sdague: so nodes with more memory should relieve that and quicker nodes (yay pvhvm) will be an overall improvement | 20:57 |
| sdague | clarkb: so there might be other culprits too | 20:57 |
| sdague | like the ceilometer logs are enormous, and garbage | 20:57 |
| clarkb | we don't index those | 20:57 |
| clarkb | and if we do that is a bug /me double checks | 20:58 |
| *** harlowja_away is now known as harlowja | 20:58 | |
| clarkb | ya we don't | 20:58 |
| anteaya | I've always wondered how to do that | 20:58 |
| *** ociuhandu has joined #openstack-infra | 20:59 | |
| *** chenxu_ has joined #openstack-infra | 20:59 | |
| *** banix has joined #openstack-infra | 20:59 | |
| *** banix has quit IRC | 21:00 | |
| *** denis_makogon_ has joined #openstack-infra | 21:00 | |
| *** oubiwann_ has quit IRC | 21:00 | |
| mattoliverau | thanks clarkb | 21:01 |
| *** denis_makogon_ is now known as denis_makogon | 21:01 | |
| clarkb | mattoliverau: I havne't looked yet :P | 21:01 |
| chenxu_ | Just had another issue with turbo-hipster | 21:01 |
| chenxu_ | https://review.openstack.org/#/c/75427/ | 21:01 |
| clarkb | now I feel compelled, I see how this works :) | 21:01 |
| chenxu_ | db migration taking too long | 21:01 |
| *** esker has joined #openstack-infra | 21:01 | |
| sdague | clarkb: interesting | 21:01 |
| chenxu_ | doesn't look like an existing bug | 21:01 |
| fungi | mattoliverau: ^ | 21:02 |
| *** lcheng has joined #openstack-infra | 21:02 | |
| fungi | mikal: ^ | 21:02 |
| fungi | i don't see jhesketh around at the moment | 21:02 |
| *** relaxdiego has joined #openstack-infra | 21:03 | |
| anteaya | fungi: check me here http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/screen-g-api.txt.gz?level=ERROR | 21:03 |
| anteaya | I'm seeing glance is failing due to a swiftclient socket error | 21:03 |
| fungi | anteaya: at which point you want to look at the swift screen logs, probably | 21:04 |
| anteaya | the only swift screen with stacktrace is http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/screen-s-proxy.txt.gz | 21:04 |
| anteaya | pkg_resources.DistributionNotFound: ceilometer | 21:04 |
| anteaya | am I on the right track? | 21:04 |
| Sukhdev | anteaya: I am using devstack (stack.sh) and as a part of it, it does git clone of nova. Most of the times it works OK - once in a while it fails with this error | 21:04 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] MVP Storyboard Client https://review.openstack.org/70897 | 21:05 |
| anteaya | Sukhdev: are you behind a proxy or firewall when you run stack.sh? | 21:05 |
| Sukhdev | anteaya: I believe so | 21:05 |
| *** jgrimm has joined #openstack-infra | 21:06 | |
| anteaya | you are going to either have to use a vm that is not behind a proxy or firewall | 21:06 |
| *** vkozhukalov_ has quit IRC | 21:06 | |
| Sukhdev | anteaya: However, it has been running just fine - no change in settings. I have started to see this issue as of yesterday | 21:06 |
| anteaya | or address the proxy/firewall that is blocking git.o.o | 21:06 |
| clarkb | sdague: also, I think the fails we had before where we lost logs were masking that we had a lot more logs | 21:06 |
| anteaya | notmyname: ping | 21:06 |
| clarkb | sdague: so we will address this by using more HP | 21:06 |
| anteaya | Sukhdev: has your proxy/firewall changed since yesterday? | 21:07 |
| clarkb | fungi: speaking of, are you going to have time to talk about spinning up performance nodes for that today? | 21:07 |
| Sukhdev | anteaya:good point - do not think so | 21:07 |
| *** dolphm is now known as dolphm_503 | 21:07 | |
| notmyname | anteaya: pong | 21:07 |
| clarkb | I am about to switch to reviewing as many changes as possible so should be able to context switch ES stuff relatively easily | 21:07 |
| anteaya | can you try on a node that is not behind a proxy or firewall | 21:07 |
| anteaya | notmyname: hi we have a nova patch failing in the gate | 21:07 |
| anteaya | the breadcrumb trail leads me to swift | 21:08 |
| Sukhdev | anteaya: will give it a try | 21:08 |
| anteaya | notmyname: http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/screen-s-proxy.txt.gz | 21:08 |
| fungi | clarkb: perhaps. let's see how far i get with pypi-mirror | 21:08 |
| anteaya | Sukhdev: if you still have problems when you are free of the firewall then let me know | 21:08 |
| mordred | fungi: whatcha doing with pypi-mirror? | 21:08 |
| anteaya | if no, then you need to investigate your settings | 21:09 |
| notmyname | anteaya: last line on that looks like ceilometer isn't installed | 21:09 |
| Sukhdev | anteaya: will do - thanks | 21:09 |
| fungi | mordred: continuing to try to reproduce and debug why it's not updating the mirror | 21:09 |
| anteaya | notmyname: I concur | 21:09 |
| anteaya | Sukhdev: thank you | 21:09 |
| *** andreaf has quit IRC | 21:09 | |
| fungi | mordred: the mirror job logs with tracebacks i linked you to earlier | 21:09 |
| anteaya | notmyname: what happens now? | 21:09 |
| Sukhdev | anteaya: do we not have neutron meeting today? no body is on that channel | 21:09 |
| anteaya | Sukhdev: I see that they are | 21:09 |
| anteaya | #openstack-meeting | 21:10 |
| anteaya | try connecting again | 21:10 |
| anteaya | you might have been caught in the net split | 21:10 |
| notmyname | anteaya: install ceilometer, I'd guess. but I'm also guessing you've tried that? | 21:10 |
| sdague | clarkb: so, maybe we can start tuning down the xtracing in devstack | 21:10 |
| Sukhdev | anteaya: oops my bad I was looking in the openstack-neutron channel - thanks for pointing it out | 21:10 |
| anteaya | Sukhdev: np | 21:10 |
| sdague | it's probably worth seeing what pruning changes we can do there | 21:11 |
| anteaya | notmyname: I had not no, this is on a nova patch failing in the gate | 21:11 |
| *** cadenzajon has joined #openstack-infra | 21:11 | |
| clarkb | sdague: ++ also removing the error'd logs from the console log | 21:11 |
| sdague | clarkb: https://review.openstack.org/#/c/75997/ | 21:11 |
| *** hogepodge has joined #openstack-infra | 21:11 | |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: enable doc generation for pycadf https://review.openstack.org/75998 | 21:11 |
| sdague | well, I kind of want to keep the error logs in the console log | 21:11 |
| clarkb | sdague: its redundant | 21:11 |
| anteaya | notmyname: https://review.openstack.org/#/c/61985/ | 21:11 |
| sdague | because people actually seemed to be paying attention to it | 21:11 |
| clarkb | put them in a file is fine | 21:11 |
| clarkb | I really dislike them in the console log | 21:11 |
| clarkb | because most of them are noise and it makes it really hard to see where in the console something broke | 21:12 |
| notmyname | anteaya: I'd guess that at some point you have some devstack setup logs for that instance. and I'd also guess that there may have been a problem during the setup where ceilometer wasn't installed for some reason | 21:12 |
| sdague | clarkb: well, actually once we started dumping them there I noticed core developers working on them to get rid of them | 21:12 |
| *** chadlung has joined #openstack-infra | 21:12 | |
| anteaya | notmyname: devstack setup logs: https://jenkins05.openstack.org/job/gate-tempest-dsvm-large-ops/3148/console | 21:12 |
| *** mrmartin has quit IRC | 21:12 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Update README https://review.openstack.org/74714 | 21:13 |
| anteaya | notmyname: correct 2014-02-24 20:38:13.609 | 21:13 |
| *** lcheng has quit IRC | 21:13 | |
| clarkb | sdague: hrm, that is valuable | 21:14 |
| *** melwitt has joined #openstack-infra | 21:14 | |
| *** lcheng has joined #openstack-infra | 21:14 | |
| sdague | yeh, I think the ugliness is a motivator, so I'm hesitant to lose that | 21:14 |
| jeblair | back | 21:15 |
| sdague | I do think we could probably trim a lot of redundant out of there by tuning down xtrace on well known functions | 21:15 |
| *** gokrokve has joined #openstack-infra | 21:17 | |
| clarkb | sdague: yeah that devstack change lgtm, I will give it a propre review and +1 | 21:17 |
| *** dolphm_503 is now known as dolphm | 21:18 | |
| sdague | I'm now actually curious how small we could get console with a few iterations of this | 21:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Publish link to docs-drafts for successful builds https://review.openstack.org/73185 | 21:19 |
| mikal | So, turbo hipster... | 21:20 |
| mikal | Failures like those are rare, a recheck should resolve them if its a slow test node | 21:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Make sure gawk is installed https://review.openstack.org/75785 | 21:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Uses python-jobs template in zuul for tuskar. https://review.openstack.org/74756 | 21:20 |
| notmyname | anteaya: I see a _lot_ of different errors in those logs. also it's very difficult to find in there any logs about what was installed or not. I've got something else going on, so I'm not sure I have any answers for you. assuming swift not starting is the cause of the other errors, then looking for why ceilometer isn't installed would be the next step | 21:21 |
| anteaya | notmyname: /node | 21:21 |
| anteaya | nod even | 21:21 |
| anteaya | okay thanks | 21:21 |
| anteaya | will continue | 21:21 |
| mfer | jeblair did you ever get a chance to look at https://review.openstack.org/#/c/71956/ | 21:21 |
| anteaya | would http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/screen-s-proxy.txt.gz be a manifestation of the mirror issues? | 21:22 |
| jeblair | mfer: thanks, i forgot to update after you answered my q. | 21:23 |
| fungi | anteaya: unlikely. also the errors we've been noticing from that problem so far have been entirely on rackspace pvhvm check nodes, and that job ran in hpcloud | 21:23 |
| anteaya | hmmm | 21:23 |
| anteaya | okay | 21:23 |
| anteaya | how is ceilometer usually downloaded onto a node? | 21:24 |
| anteaya | it comes in the git cache with the image correct/ | 21:24 |
| anteaya | ? | 21:24 |
| clarkb | anteaya: right | 21:24 |
| fungi | anteaya: correct, and it's showing as being a working clone reset to master in the setup workspace log for that job | 21:24 |
| tchaypo | my patch only just got merged, but http://docs.openstack.org/developer/tripleo-incubator/devtest.html is already looking stylish again. that was faster than I expected. | 21:24 |
| anteaya | so if pkg_resourses.DistributionNotFound can't find ceilometer? | 21:25 |
| *** dolphm is now known as dolphm_503 | 21:25 | |
| * anteaya checks out the setup workspace log | 21:25 | |
| fungi | anteaya: then devstack or tempest probably had some trouble making sure it was installed | 21:25 |
| clarkb | anteaya: do you know what the status of new project creations is? eg if we approve changes that create projects does that require a manual manage-projects trigger? | 21:25 |
| chenxu_ | mikal: thx.. I did submit a bug report, want me to cancel that, or leave it there? | 21:26 |
| anteaya | it requires monitoring and a manual push if need be | 21:26 |
| *** dolphm_503 is now known as dolphm | 21:26 | |
| anteaya | it will trigger, it might just not complete correctly, clarkb | 21:26 |
| fungi | clarkb: it will run automagically, and if there's an upstream to import then it seems to work (modulo needing to re-trigger replication got the project in gerrit after giving create-cgitrepos enough time to see the projects.yaml update and rerun) | 21:26 |
| fungi | s/replication got the project/replication for the project/ | 21:26 |
| clarkb | fungi: anteaya thanks | 21:27 |
| fungi | clarkb: if there is no upstream to import, it currently seems to fail spectacularly | 21:27 |
| fungi | clarkb: but pounding on it with several reruns and removing the git repos it creates in between on the gerrit server seems to eventually make it happen, at least in the one case i ended up cleaning up behind | 21:28 |
| fungi | clarkb: and we do get tracebacks/errors from manage-projects in the syslog now too, courtesy of puppet | 21:29 |
| anteaya | this is interesting: HEAD is now at 2f9300f Merge "Fixed spelling error in Ceilometer" | 21:29 |
| anteaya | I wonder if that plays a role | 21:29 |
| jeblair | YAY! I'm caught up on infra/config reviews to _today_! | 21:29 |
| *** ociuhandu has quit IRC | 21:29 | |
| clarkb | jeblair: wow | 21:29 |
| clarkb | that is impressive | 21:29 |
| * clarkb goes back to trying to catch up | 21:29 | |
| anteaya | *applause* | 21:29 |
| fungi | jeblair: you are waaaay ahead of me on reviews in that case | 21:30 |
| jeblair | clarkb: you can do it! | 21:30 |
| HenryG | Hi, sorry, is there a bug for the requires six 1.5.2 error? | 21:30 |
| jeblair | I only did it thanks to SergeyLukjanov's help. :) | 21:30 |
| fungi | oh, wait *just* infra/config, not all the other infra projects | 21:30 |
| *** juice- has joined #openstack-infra | 21:30 | |
| jeblair | fungi: yes. *just* that. :) | 21:30 |
| anteaya | HenryG: https://bugs.launchpad.net/openstack-ci/+bug/1276242 | 21:30 |
| *** dripton_ is now known as dripton | 21:30 | |
| fungi | jeblair: heh, still no small task | 21:30 |
| HenryG | anteaya: Thanks! | 21:30 |
| anteaya | np | 21:31 |
| anteaya | thanks for asking | 21:31 |
| HenryG | anteaya: can I recheck on that bug, or is there no point? | 21:31 |
| anteaya | you can recheck on that bug | 21:31 |
| HenryG | anteaya: thanks again | 21:32 |
| anteaya | np | 21:32 |
| fungi | HenryG: in another few minutes, the mirror will have six 1.5.2 | 21:32 |
| jeblair | fungi: what broke? | 21:32 |
| clarkb | ++ /me wants to know too and has picked up only snippets on irc | 21:33 |
| fungi | because apparently the traceback i as looking at was a red herring, and the real reason it didn't update is that on merging a change to pypi-mirror we then proceed to run the mirror job using the copy of pypi-mirror previously installed on the slave by puppet :/ | 21:33 |
| *** lcheng has quit IRC | 21:33 | |
| jeblair | whoopsiedoo. | 21:33 |
| fungi | just retriggering the job seems to have added it | 21:33 |
| *** mbacchi has quit IRC | 21:34 | |
| fungi | it's not on the mirror yet, but it's now appearing in the local tree on teh slave which is about to get rsync'd out to it | 21:34 |
| *** jhesketh has joined #openstack-infra | 21:34 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves https://review.openstack.org/70283 | 21:34 |
| fungi | and rsync says... | 21:34 |
| fungi | 2014-02-24 21:34:17.424 | <f+++++++++ openstack/six/six-1.5.2.tar.gz | 21:34 |
| clarkb | jeblair: ^ fixes a merge conflict in a change you approved | 21:34 |
| fungi | yep, http://pypi.openstack.org/openstack/six/ has it now | 21:35 |
| *** dkliban has quit IRC | 21:35 | |
| jhesketh | Howdy | 21:35 |
| fungi | sdague: ^ | 21:35 |
| sdague | yay | 21:35 |
| *** lcheng has joined #openstack-infra | 21:35 | |
| jeblair | jhesketh: hi there | 21:35 |
| clarkb | fungi: oh fun, we can fix that with a virtualenv install for the non periodic triggered jobs maybe? | 21:35 |
| anteaya | this looks important fatal: Couldn't find remote ref refs/zuul/master/Z4f0542f087064faf8b46ce3b08c936f7 | 21:35 |
| openstackgerrit | A change was merged to openstack-infra/config: python-tuskarclient: enable the py33 gate https://review.openstack.org/73130 | 21:36 |
| jeblair | anteaya: it's not, that's why we hide it outside the main log. | 21:36 |
| anteaya | jeblair: okay then, i will keep looking | 21:36 |
| fungi | clarkb: i expect so | 21:36 |
| jeblair | anteaya: what job? | 21:36 |
| clarkb | jeblair: though now that I think about it maybe that sed line should've been 's/ext3/ext4/g' | 21:37 |
| anteaya | http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/devstack-gate-setup-workspace-new.txt.gz | 21:37 |
| clarkb | its funny how the brain goes "NO WAIT" when you approve a thing | 21:37 |
| jeblair | anteaya: yeah | 21:37 |
| anteaya | jeblair: a nova patch was failing in the gate | 21:37 |
| clarkb | jeblair: fungi: any opinions on https://review.openstack.org/#/c/70283/3/modules/openstack_project/files/nodepool/scripts/prepare_node.sh needing the g flag? | 21:37 |
| anteaya | jeblair: I thought I would dig in in case it was larger than 1 job: https://review.openstack.org/#/c/61985/ | 21:37 |
| jeblair | anteaya: that just means zuul did not prepare a ref for that project. the message comes from git, and there's not much we can do about it (other than hide it, but also possibly any real errors, completely) | 21:37 |
| *** oubiwann_ has joined #openstack-infra | 21:38 | |
| jeblair | clarkb: probably should | 21:38 |
| fungi | clarkb: can't hurt... let me check a system real quick and see if it has any lines with more than one match | 21:38 |
| anteaya | jeblair: okay, well at least I have context now | 21:38 |
| *** juice- has quit IRC | 21:38 | |
| *** juice- has joined #openstack-infra | 21:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves https://review.openstack.org/70283 | 21:39 |
| clarkb | I went ahead and short circuited the approval with a new patchset now with g flag | 21:39 |
| anteaya | okay so in the gate-setup-workspace-new log for that job it had a ceilometer with HEAD is now at 2f9300f | 21:40 |
| *** jnoller has quit IRC | 21:40 | |
| anteaya | so I am not understanding how it couldn't be found by swift | 21:40 |
| SergeyLukjanov | heh, looks like I have a very bad karma - https://review.openstack.org/#/c/74646/ | 21:40 |
| fungi | clarkb: i checked current systems in rax and hpcloud and it looks like it wouldn't make any difference, but it's still good for future-proofing agreed | 21:41 |
| *** amcrn has joined #openstack-infra | 21:41 | |
| fungi | anteaya: swift is looking for an installed ceilometer. what you see there is a git repository for ceilometer cached locally om the server. so the missing link is what does the installation of the service (devstack, i'd assume) | 21:43 |
| anteaya | ah okay | 21:43 |
| *** jergerber has joined #openstack-infra | 21:43 | |
| *** jcoufal has quit IRC | 21:44 | |
| *** tjones has quit IRC | 21:44 | |
| fungi | clarkb: so performance+pvhvm? | 21:47 |
| *** thomasem has quit IRC | 21:47 | |
| *** cody-somerville has quit IRC | 21:47 | |
| fungi | clarkb: was it the elasticsearch or logstash worker nodes which needed replacing? | 21:48 |
| clarkb | fungi: elasticsearch | 21:48 |
| clarkb | fungi: the logstash worker nodes seem to be mostly ok and trouble lies further down the pipeline | 21:48 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Retry ssh connections on auth failure. https://review.openstack.org/74825 | 21:48 |
| fungi | and did we want to scale the cluster out onto new elasticsearch workers, then scale off the old ones, or replace a few at a time (basically scale up then down, reusing old node names)? | 21:48 |
| fungi | er, scale up then down, OR reusing old node names? | 21:49 |
| clarkb | fungi: I think we want to add a node, remove old node, and iterate | 21:49 |
| clarkb | fungi: and we can use names like elasticsearch01.openstack.org for the new nodes | 21:49 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add fedora support https://review.openstack.org/74529 | 21:49 |
| *** juice- has quit IRC | 21:49 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest https://review.openstack.org/75960 | 21:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Mount ext3 filesystems as ext4 on single use slaves https://review.openstack.org/70283 | 21:49 |
| fungi | clarkb: okay, so the 6 which we currently have? | 21:50 |
| clarkb | fungi: but before we spin up new nodes we probably want to do a couple things. Edit the elasticsearch node list(s) in site.pp so that firewalls are updated appropriately. And we need to make the ES heap size allocation more dynamic. Right now it is hard coded to 16GB iirc | 21:50 |
| clarkb | fungi: the six we currently have a elasticsearch.o.o and elasticsearch2-6.o.o | 21:51 |
| clarkb | zaro: can you address https://review.openstack.org/#/c/61321/ before you travel? | 21:51 |
| NobodyCam | ok I was just asked a question that I did not know the answer to. So I am asking here hopping to learn somehting new today. the question is: are there any issues tagging a review with more then one BluePrint? | 21:51 |
| fungi | clarkb: right the same ones i was looking at in the nova list then. perfect | 21:51 |
| *** juice- has joined #openstack-infra | 21:52 | |
| fungi | NobodyCam: ooh, absolutely no idea, but i expect the hook which updates the topic and adds notes to the bp on lp might get confused by that | 21:52 |
| fungi | i think it assumes no more than one bp associated with a change | 21:52 |
| clarkb | do bp links do funcy stuff like that? | 21:52 |
| clarkb | it may just be bugs that do | 21:52 |
| NobodyCam | I didn't know | 21:52 |
| * anteaya wonders if this is important: Could not find any downloads that satisfy the requirement alembic>=0.4.1 (from ceilometer==2014.1.dev189.g2f9300f) | 21:52 | |
| fungi | clarkb: the update_blueprint.py hook changes the review topic at least | 21:53 |
| fungi | i'd have to read back through the script in jeepyb to find out what else | 21:53 |
| fungi | since i don't normally use blueprints, i haven't paid close attention | 21:53 |
| clarkb | same here | 21:54 |
| NobodyCam | ok so Safe answer is don't do it | 21:54 |
| fungi | anteaya: perhaps. is that in the devstack log? | 21:54 |
| anteaya | yes | 21:54 |
| anteaya | http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/devstacklog.txt.gz | 21:54 |
| clarkb | fungi: so ya for ES nodes I think we want to address those two puppet things first, then spin up a machine that joins the cluster | 21:54 |
| fungi | NobodyCam: or try it out on a change to openstack-dev/sandbox with a throwaway blueprint and see what happens | 21:54 |
| NobodyCam | anteaya: sounds like a out of date repo | 21:54 |
| anteaya | NobodyCam: it should be our pypi mirror I think | 21:54 |
| clarkb | fungi: http://git.openstack.org/cgit/openstack-infra/config/tree/manifests/site.pp#n333 and http://git.openstack.org/cgit/openstack-infra/config/tree/manifests/site.pp#n364 is first item that needs addressing (just add elasticsearch01.o.o to the lists), http://git.openstack.org/cgit/openstack-infra/config/tree/modules/elasticsearch/manifests/init.pp#n19 is the second thing. We want 16g heap on the | 21:56 |
| clarkb | 30G nodes, but probably want 30G heap on 60G nodes and so on | 21:56 |
| *** protux has quit IRC | 21:57 | |
| *** juice has quit IRC | 21:57 | |
| *** juice- is now known as juice | 21:57 | |
| clarkb | that needs to be made slightly more flexible in puppet, maybe a fact? or just a second elasticsearch_node manifest that sets a different value for the new nodes | 21:57 |
| fungi | anteaya: "Cannot fetch index base URL http://pypi.openstack.org/openstack/" i think is where the fun is | 21:57 |
| fungi | anteaya: possibly that slave ran into network trouble reaching the pypi mirror | 21:57 |
| fungi | clarkb: working on those now | 21:58 |
| clarkb | fungi: rule of thumb from upstream ES is have heap size set to about 50% of total available RAM | 21:58 |
| clarkb | that gives the OS plenty of room for caches | 21:59 |
| anteaya | fungi: so do we have a bug open for that? | 21:59 |
| fungi | yeah, taking a look at the options in flavor-list now | 21:59 |
| fungi | anteaya: maybe? i don't recall. it does happen from time to time... internets and all that | 21:59 |
| clarkb | fungi: as far as choices for nodes go I have been leaning towards a 60G performance node with pvhvm image and elasticsearch data on cinder volume(s) | 21:59 |
| *** e0ne has quit IRC | 22:00 | |
| fungi | anteaya: putting pypi mirrors in each provider (or maybe in each region/availability zone) could help mitigate it some | 22:00 |
| clarkb | I think doubling the node size from 30G should be a big improvement and if we outgrow the 60G nodes we can move the cinder block mounts | 22:00 |
| fungi | clarkb: oh, fun, so we're adding cinder volumes on these too. excellent | 22:01 |
| anteaya | fungi: not seeing an open bug for it, open one or no? | 22:01 |
| clarkb | fungi: well we don't absolutely need to if we go with 120G nodes :) | 22:01 |
| fungi | anteaya: i guess open one and if it turns out to be a dupe later when we triage, we can just mark it as such at that point | 22:01 |
| clarkb | the 120G nodes have enough ephemeral disk, but I think 120G nodes are overkill and going with cinder blocks is more appropriate | 22:02 |
| clarkb | jeblair: ^ thoughts? | 22:02 |
| fungi | clarkb: right. 60g w/ cinder sounds fine | 22:02 |
| *** smarcet has quit IRC | 22:02 | |
| anteaya | fungi: here is one that looks close if not exact: https://bugs.launchpad.net/openstack-ci/+bug/1266616 | 22:02 |
| anteaya | or are they different? | 22:03 |
| fungi | anteaya: nope. unrelated. that's a problem with hpcloud nodes deciding they have ipv6 routes when they don't | 22:03 |
| anteaya | okay | 22:03 |
| anteaya | new bug it is | 22:03 |
| *** rlandy_ has quit IRC | 22:04 | |
| jeblair | clarkb: yeah, let's not overkill on ram. | 22:04 |
| *** mgagne1 is now known as mgagne | 22:05 | |
| anteaya | fungi: looks like this might be the same bug: https://bugs.launchpad.net/openstack-ci/+bug/1272417 | 22:05 |
| fungi | anteaya: yep | 22:05 |
| anteaya | k | 22:06 |
| anteaya | I will add a comment to the report | 22:06 |
| *** dkliban has joined #openstack-infra | 22:07 | |
| anteaya | notmyname: thanks for your help, it was a pypi issue | 22:07 |
| *** emagana_ has quit IRC | 22:08 | |
| notmyname | anteaya: ah, good to know. I'm glad you found it :-) | 22:08 |
| anteaya | thanks for the chat | 22:09 |
| anteaya | :D | 22:09 |
| *** yassine has quit IRC | 22:09 | |
| fungi | clarkb: any reason not to just pass heap_size in from manifests/site.pp temporarily? | 22:10 |
| clarkb | fungi: and have two different node matches? I think that would work well | 22:10 |
| fungi | and then move it to modules/openstack_project/ later once they're consistent | 22:10 |
| clarkb | ++ | 22:11 |
| fungi | i'll do that. no need to overengineer the module for the sake of temporary disparity in heap size between old and new servers | 22:11 |
| sdague | so my xtrace change only trimmed the console.html by 15-20% | 22:11 |
| fungi | sdague: that's 15-20% of a pretty big number though | 22:12 |
| sdague | sure | 22:12 |
| sdague | it got them down to 1.9MB | 22:12 |
| sdague | instead of 2.3MB | 22:12 |
| *** thomasem has joined #openstack-infra | 22:12 | |
| *** SumitNaiksatam has quit IRC | 22:12 | |
| sdague | honestly my biggest gripe is the fact that now there are a lot of xtrace lines because I keep turning it off | 22:13 |
| lifeless | jeblair: fungi: tripleo cloud should be functional again now, some cleanup remaining | 22:14 |
| lifeless | jeblair: fungi: http://paste.openstack.org/show/69129/ | 22:14 |
| lifeless | is our biggest current clue | 22:14 |
| fungi | lifeless: don't rule out some n00b in the data center wondering what the nmi button does | 22:15 |
| lifeless | fungi: already checked that :) | 22:16 |
| lifeless | infra folk had not touched the rack | 22:16 |
| fungi | ha | 22:16 |
| lifeless | or the switches | 22:16 |
| *** mfer has quit IRC | 22:16 | |
| * fungi has DEFINITELY seen that happen before | 22:16 | |
| fungi | nmi button exposed on the server's bezel is such a bad, bad idea | 22:16 |
| *** moted has joined #openstack-infra | 22:17 | |
| jeblair | fungi, sdague: there are some changes to use six in zuul; should we add six as a direct dependency? it seems to be getting it from somewhere implecit because it ends up in the venv... | 22:18 |
| jeblair | fungi, sdague: (i ask because you know everything there is to know about six right now) | 22:18 |
| fungi | jeblair: i think if zuul directly imports things from six, it should declare that in its reqs | 22:19 |
| sdague | yes, agreed | 22:19 |
| fungi | just like with any python package | 22:19 |
| jeblair | that was my thinking as well. these changes got a lot of +'s though so i thought i'd double check. :) | 22:19 |
| fungi | people will + anything (i'm living proof!) | 22:19 |
| *** miqui has quit IRC | 22:20 | |
| *** nati_ueno has quit IRC | 22:20 | |
| *** matrohon has joined #openstack-infra | 22:21 | |
| lifeless | fungi: can you check nodepool thinks the cloud is back ? | 22:21 |
| *** nati_ueno has joined #openstack-infra | 22:21 | |
| *** sandywalsh has quit IRC | 22:22 | |
| openstackgerrit | A change was merged to openstack/requirements: Don't update pbr's setup.py https://review.openstack.org/73431 | 22:22 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 22:22 |
| openstackgerrit | Jenkins proposed a change to openstack-dev/pbr: Updated from global requirements https://review.openstack.org/73119 | 22:22 |
| openstackgerrit | A change was merged to openstack/requirements: Standardize read ops to 'with open' construction https://review.openstack.org/73808 | 22:23 |
| sdague | jeblair: with your devstack +A powers you could +A the log format change - https://review.openstack.org/#/c/75689/ | 22:23 |
| jeblair | sdague: that seems in-scope for me, sure. | 22:25 |
| *** Sukhdev has quit IRC | 22:25 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add bugdaystats to openstack-infra https://review.openstack.org/69489 | 22:25 |
| *** lcostantino has quit IRC | 22:26 | |
| *** esker has quit IRC | 22:27 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Create new 30g heap elasticsearch workers https://review.openstack.org/76026 | 22:29 |
| fungi | clarkb: that ^ i think. gonna grab dinner and then work on spinning up new virtual machines if the change looks okay | 22:29 |
| *** lnxnut has quit IRC | 22:29 | |
| * clarkb looks | 22:30 | |
| jeblair | sdague: aprvd | 22:30 |
| fungi | clarkb: the elasticsearch module seems to already be configurable but defaults to 16g, so not changing it there in case anyone else is also using it | 22:31 |
| *** chenxu_ has quit IRC | 22:31 | |
| *** julim_ has quit IRC | 22:32 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 22:32 |
| *** jgrimm has quit IRC | 22:32 | |
| clarkb | fungi: reviewed | 22:34 |
| fungi | lifeless: i see some building for a little while and some very recently ready nodepool nodes in tripleo-ci | 22:34 |
| *** sandywalsh has joined #openstack-infra | 22:35 | |
| fungi | clarkb: any reason not to make the elasticsearch_nodes and discover_nodes lists a second change then, and spin up all 6 new workers at once? | 22:35 |
| *** lcheng has quit IRC | 22:36 | |
| fungi | i'm happy to split it up | 22:36 |
| *** lcheng has joined #openstack-infra | 22:36 | |
| *** ociuhandu has joined #openstack-infra | 22:37 | |
| clarkb | fungi: my only concern is that growing the cluster by 2x all at once will cause it to freak out and rebalance all the things then as nodes are removed it will rebalance all over again | 22:37 |
| clarkb | whereas doing one node at a time should keep the rebalance thrash to a minimum | 22:37 |
| fungi | clarkb: so 6 changes, one for each addition to elasticsearch_nodes and discover_nodes, with corresponding removal changes for the old ones after each? | 22:38 |
| clarkb | fungi: we should be able to do a single removal change to remove nodes (we can manually 'remove' them by turning them off) | 22:39 |
| clarkb | fungi: or, we put up bogus A records for now | 22:39 |
| clarkb | then as we get IPs kick iptables-peristent | 22:39 |
| clarkb | as I think about that the idea isn't too terrible | 22:39 |
| *** lcostantino has joined #openstack-infra | 22:41 | |
| *** ianw has joined #openstack-infra | 22:42 | |
| *** yamahata has joined #openstack-infra | 22:43 | |
| fungi | which, the temporary dummy rrs? | 22:44 |
| *** denis_makogon has quit IRC | 22:45 | |
| *** yolanda_ has quit IRC | 22:45 | |
| *** dkranz has quit IRC | 22:45 | |
| fungi | i'll just add those in that case, and remove them before updating dns for each new launch | 22:45 |
| clarkb | fungi: for elasticsearch01 02 03 and so on | 22:47 |
| *** StevenK_ is now known as StevenK | 22:50 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add Zuul Job Queue graph https://review.openstack.org/75706 | 22:53 |
| *** jamielennox|away is now known as jamielennox | 22:53 | |
| *** pdmars has quit IRC | 22:54 | |
| fungi | clarkb: right | 22:55 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add Zuul Job Queue graph https://review.openstack.org/75706 | 22:56 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: MVP Storyboard Client https://review.openstack.org/70897 | 22:57 |
| *** CaptTofu has quit IRC | 22:57 | |
| jeblair | sorry about the churn on that; i think i'm happy with ps3 now. ^ | 22:58 |
| *** e0ne has joined #openstack-infra | 22:58 | |
| jeblair | and, btw, you can see the tripleo outage on it. | 22:59 |
| jeblair | (waiting climbs while workers falls == provider outage) | 22:59 |
| *** jergerber has quit IRC | 23:00 | |
| *** e0ne_ has joined #openstack-infra | 23:00 | |
| *** bhuvan has joined #openstack-infra | 23:02 | |
| fungi | clarkb: added | 23:02 |
| fungi | clarkb: i set them all as 127.127.127.127 for now. nothing should be listening on lo for that. hopefully making them all the same address won't confuse anything? | 23:03 |
| bhuvan | jeblair: https://review.openstack.org/#/c/72445/ . one quick question ... | 23:03 |
| clarkb | fungi: stuff will listen on that because linux | 23:03 |
| *** e0ne has quit IRC | 23:04 | |
| clarkb | fungi: :) might be better to use bogus 172 or 169.whatever addresses | 23:04 |
| fungi | clarkb: oh, right. bsd brain. i'll make up something else | 23:04 |
| bhuvan | the current approach is to download the package from sss website directly and install. if there are any security fixes, we might want to change to download new version | 23:04 |
| bhuvan | i presume the security fixes will be shipped as part of new package automatically ... | 23:05 |
| jeblair | bhuvan: right, but that requires us to monitor that project so that we notice those security fixes and then update that version | 23:05 |
| bhuvan | yes | 23:05 |
| *** e0ne_ has quit IRC | 23:05 | |
| jeblair | bhuvan: i'm pretty sure no one's going to do that for an irc log analyzer | 23:05 |
| bhuvan | do you propose to bundle that software within our code? | 23:06 |
| jeblair | bhuvan: i would never propose that! | 23:06 |
| *** oubiwann_ has quit IRC | 23:06 | |
| jeblair | bhuvan: so we really need something that's packaged by ubuntu, or else, a continuous deployment mechanism that installs it from their git repo, or something similar | 23:06 |
| bhuvan | hmm, how do we programatically detect for new security updates? | 23:07 |
| jeblair | bhuvan: we install a number of things that aren't distro-packaged in that manner (from git) | 23:07 |
| *** oubiwann_ has joined #openstack-infra | 23:07 | |
| jeblair | bhuvan: it means they are more likely to break, but less likely to sit around on old versions and miss security updates | 23:07 |
| jeblair | bhuvan: but honestly, something packaged by a distro would be ideal for this. | 23:08 |
| *** weshay has quit IRC | 23:08 | |
| bhuvan | unfortunately, sss isn't part of any distro yet | 23:09 |
| jog0 | I take it http://logs.openstack.org/18/75918/1/check/gate-nova-python27/9c26936/console.html is a known issue | 23:09 |
| bhuvan | i'm fix the patch to deploy from git | 23:09 |
| bhuvan | https://github.com/tommyrot/superseriousstats | 23:09 |
| jeblair | jog0: not by me | 23:09 |
| bhuvan | jeblair: i think it's better than downloading package directly. what you think? | 23:10 |
| jog0 | jeblair: uh oh | 23:10 |
| jeblair | bhuvan: i think so. i'd ask some of the other infra core folks for their opinion first. clarkb and fungi in particular. | 23:11 |
| fungi | clarkb: okay, i've switched them to 172.31.255.1-6 | 23:11 |
| *** CaptTofu has joined #openstack-infra | 23:11 | |
| *** rossella-s has quit IRC | 23:12 | |
| clarkb | fungi: wfm | 23:12 |
| clarkb | will remove my -1 | 23:12 |
| fungi | jeblair: bhuvan: yes, i'd rather see the irc stats generation fail than the server get compromised. also deploying from git should probably allow us to do things not as root (whereas installing packages means running maintscripts with root privs) | 23:13 |
| jog0 | fungi clarkb: ^ | 23:13 |
| *** rossella-s has joined #openstack-infra | 23:13 | |
| jeblair | jog0: looking at nodepool logs, i can tell you that nodepool did not remove that node before the job aborted. | 23:14 |
| clarkb | fungi: I +2'd but did not approve. Mostly because I would like to see the first node get spun up using that in a different puppet env | 23:15 |
| fungi | jog0: that's a new one on me | 23:15 |
| jeblair | WARNING: Making bare-precise-hpcloud-az2-1572924 offline temporarily due to the lack of disk space | 23:15 |
| *** oubiwann_ has quit IRC | 23:15 | |
| jeblair | jog0: ^ | 23:15 |
| clarkb | fungi: the other potential issue is that elasticsearch will happily get installed and join the cluster before we can attach the cinder volume I think | 23:15 |
| fungi | clarkb: sure, can do | 23:15 |
| clarkb | fungi: Thoughts on how to solve that problem? | 23:15 |
| fungi | hmmm... let me think it over while i finish eating | 23:16 |
| clarkb | this is an issue beacuse it will start indexing and stuff then a new mount will shadow that data | 23:16 |
| jog0 | ohh | 23:16 |
| fungi | clarkb: can we set the service not to start at boot? | 23:16 |
| fungi | in puppet manifests | 23:17 |
| *** eharney has quit IRC | 23:17 | |
| jog0 | jeblair: message:" Looks like the node went offline during the build" AND message:"slave.log \(No such file or directory\)" AND filename:"console.html" | 23:17 |
| clarkb | fungi: probably? I think the upstream pacakges start the service | 23:17 |
| jog0 | if that looks like a good query to you, I will go ahead and file a e-s patch for it | 23:17 |
| clarkb | fungi: you know more about packaging than I, if we can make that work it sounds good to me | 23:17 |
| fungi | oh, hrm, right | 23:17 |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: enable doc generation for pycadf https://review.openstack.org/75998 | 23:18 |
| jeblair | jog0: yeah. it might cover slightly more nuanced problems than this, but it's always some kind of jenkins problem so is a good one. | 23:18 |
| jog0 | jeblair: kk | 23:18 |
| jog0 | http://logs.openstack.org/35/75535/1/check/check-tempest-dsvm-postgres-full/409e8e8/logs/df.txt.gz | 23:18 |
| *** lcostantino has quit IRC | 23:18 | |
| jog0 | we don't use that much data by the end of the run | 23:18 |
| fungi | clarkb: what port/protocol do they connect over? can we lock that down more tightly in iptables? | 23:19 |
| jeblair | jog0: yeah, it could be a false error from jenkins (like perhaps its attempt to perform that check encountered a network error) | 23:19 |
| * jog0 files a bug | 23:20 | |
| jeblair | zaro, jog0: i think this might be adverse interaction with gearman-plugin | 23:20 |
| clarkb | fungi: its the es protocol ports 9200-9399 | 23:21 |
| clarkb | fungi: one half uses http REST the other half a home grown thing | 23:21 |
| clarkb | fungi: what if we do the intial install without your puppet change so that the default manifest is used | 23:22 |
| *** dims has quit IRC | 23:22 | |
| clarkb | fungi: configure cinder, then add the nodes using your manifest | 23:22 |
| clarkb | fungi: that should allow us to prime all of the nodes before we add them to the cluster | 23:22 |
| fungi | clarkb: great idea | 23:22 |
| fungi | clarkb: actually, no | 23:22 |
| fungi | the current pattern is \d* | 23:23 |
| fungi | so it will match the new names too | 23:23 |
| clarkb | :( | 23:23 |
| bhuvan | jeblair: fungi: thank you. i'll fix the patch to install latest sss changes using puppet/vcsrepo. the script ircstats.py will be run from cron to generate the stats ... | 23:23 |
| *** derekh_afk is now known as derekh | 23:23 | |
| jeblair | bhuvan: cool, thanks | 23:23 |
| bhuvan | the script might fail if sss is not already installed | 23:23 |
| fungi | clarkb: so a quick tweak to make the current pattern \d? should make that possible | 23:23 |
| clarkb | fungi: that wfm, though doesn't handle the next time we need to do this | 23:24 |
| jeblair | bhuvan: you can have the cron require the vcsrepo in puppet, that should be good enough | 23:24 |
| bhuvan | i presume puppet agent is run as daemon and it'll take care of pulling latest changes from git automatically | 23:24 |
| jeblair | bhuvan: yes | 23:24 |
| bhuvan | jeblair: yep | 23:24 |
| jeblair | jog0, zaro: the incidences of this in the jenkins log strongly correlate with the gearman plugin de-registering jobs | 23:25 |
| jeblair | jog0, zaro: it's possible it's more complicated than that, but it's worth a look. | 23:26 |
| jeblair | jog0, zaro: http://paste.openstack.org/show/69188/ | 23:26 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Auth support https://review.openstack.org/73219 | 23:26 |
| bhuvan | fungi: as discussed couple of weeks back, review/merge https://review.openstack.org/#/c/71323/ when you find time ... | 23:26 |
| jeblair | jog0: can you throw that in the bug report if you still have it up? | 23:26 |
| *** tjones has joined #openstack-infra | 23:27 | |
| jog0 | https://bugs.launchpad.net/openstack-ci/+bug/1284371 | 23:27 |
| *** lcheng has quit IRC | 23:27 | |
| *** branen has joined #openstack-infra | 23:28 | |
| jeblair | jog0: thanxs | 23:28 |
| *** hashar has quit IRC | 23:29 | |
| jog0 | done | 23:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: index sublog files https://review.openstack.org/75719 | 23:29 |
| sdague | clarkb: that should be the tag fix | 23:29 |
| *** rossella-s has quit IRC | 23:32 | |
| *** alexpilotti has quit IRC | 23:32 | |
| fungi | clarkb: i think i'll see if i can work out a way to make service enabling an option to the module, so we can throw the switch via puppet | 23:33 |
| clarkb | fungi: sounds good | 23:33 |
| fungi | bhuvan: thanks and apologies for the delay. i'm trying to get the pull request closer running again first, but trying to figure out why the orgs list api call is coming back with an empty set for our account (i did log into github with the account and it sees it's a member of a group on each org) | 23:34 |
| clarkb | jeblair: for gearman merge workers. if they should be talking to a gerrit replica instead of gerrit itself (assume replication is atomic and instantaneous), zuul can't do that today because the url info comes from gerrit itself irght? | 23:34 |
| *** zehicle_at_dell has joined #openstack-infra | 23:34 | |
| *** hemna is now known as hemnafk | 23:34 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use statsd in logstash client https://review.openstack.org/76042 | 23:35 |
| *** amcrn has quit IRC | 23:35 | |
| jeblair | clarkb: ^ what we were missing this wknd | 23:35 |
| sdague | clarkb: ok, so actually 3.4 M -> 2.7 M with the xtrace changes | 23:36 |
| sdague | 2.6 M | 23:36 |
| sdague | http://logs.openstack.org/97/75997/3/check/check-tempest-dsvm-postgres-full/1cc4026/ | 23:36 |
| bhuvan | fungi: thanks | 23:36 |
| jeblair | clarkb: i don't understand the question | 23:36 |
| sdague | clarkb: if you see other good things to slice out of the logs, now is a good time to speak up | 23:37 |
| clarkb | jeblair: gerrit emits a patchset created event to zuul, zuul kicks off the merge worker to merge that. We don't want the merge worker to do its initial clone or fetches from gerrit directly but a gerrit replica instead | 23:37 |
| clarkb | jeblair: that doesn't seem supported in zuul today | 23:37 |
| clarkb | sdague: I trust your judgement :) | 23:38 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1284371 https://review.openstack.org/76043 | 23:38 |
| jog0 | jeblair: ^ | 23:38 |
| jeblair | clarkb: why don't we want it to fetch from gerrit directly? | 23:38 |
| clarkb | jeblair: because lol firewalls | 23:38 |
| *** dims has joined #openstack-infra | 23:39 | |
| jeblair | clarkb: what firewalls? our gerrit ..... | 23:39 |
| clarkb | jeblair: I am mostly trying to frame the issue and understand where things might break spectacularly | 23:39 |
| clarkb | jeblair: no not our gerrit | 23:39 |
| clarkb | hypothetical hp gerrit | 23:39 |
| jeblair | clarkb: you are using "we" in a way you have not previously used "we" :( | 23:39 |
| clarkb | oh sorry | 23:39 |
| jeblair | me too | 23:39 |
| jeblair | clarkb: to answer your question, there may or may not be a race condition there, i would recommend finding a definitive answer to that first. | 23:41 |
| clarkb | jeblair: there definitely is a race condition between gerrit emitting the ssh event and the replica having the data related to that event | 23:41 |
| lifeless | fungi: still seeing queued everywhere :( | 23:41 |
| fungi | lifeless: i'll have another look at nodepool | 23:42 |
| *** dolphm is now known as dolphm_503 | 23:43 | |
| *** thomasem has quit IRC | 23:43 | |
| jeblair | clarkb: how is it possible for you to connect to gerrit over ssh and not fetch refs over that same ssh connection? | 23:44 |
| *** sarob_ has joined #openstack-infra | 23:44 | |
| clarkb | jeblair: zuul is on one side of the firewall and the zuul merger is on the other | 23:44 |
| clarkb | so zuul gets ssh events fine, but the merger doesn't have ssh access | 23:45 |
| clarkb | * ssh access in | 23:45 |
| SpamapS | FYI tripleo images that build with the pypi.openstack.org mirror are working again. | 23:45 |
| lifeless | fungi: ah - ERROR: Quota exceeded for instances: Requested 1, but already used 100 of 100 instances (HTTP 413) (Request-ID: req-3ad1e797-d33d-47be-87d1-2c63dc24f542) | 23:46 |
| lifeless | fungi: I think I see the problem | 23:46 |
| fungi | lifeless: i'm clearing all old nodes | 23:46 |
| lifeless | fungi: I'm going to do a rolling restart of nova-compute to clear out the stuck deleting instances | 23:46 |
| fungi | all tripleo nodes in any state for more than an hour | 23:46 |
| fungi | k | 23:46 |
| lifeless | SpamapS: can you help me? I'll take novacompute0-4, you do 5-9 ? | 23:46 |
| *** sarob has quit IRC | 23:47 | |
| jeblair | clarkb: zuul does not support this, but you could patch it to have the merger busy-wait for the ref until a timeout. | 23:47 |
| fungi | lifeless: we had ~90 nodes which were more than an hour in any given state in your cloud | 23:47 |
| *** oubiwann_ has joined #openstack-infra | 23:47 | |
| clarkb | jeblair: thanks | 23:48 |
| SpamapS | lifeless: sure | 23:49 |
| *** sarob_ has quit IRC | 23:49 | |
| *** prad_ has quit IRC | 23:49 | |
| *** chadlung has quit IRC | 23:49 | |
| jeblair | 7 of 9 changes in the gate are devstack or devstack-gate | 23:50 |
| jeblair | sorry 6 of 8 | 23:50 |
| lifeless | fungi: I issued a bulk (API) delete of ERROR state instances | 23:50 |
| fungi | k | 23:51 |
| lifeless | SpamapS: shout when thats done please | 23:51 |
| lifeless | SpamapS: I filed https://bugs.launchpad.net/tripleo/+bug/1284356 about this | 23:51 |
| lifeless | SpamapS: not sure its amenable to 'just go fix' though | 23:51 |
| SpamapS | 5,6 done | 23:51 |
| lifeless | ok, I'll grab 8, 9 | 23:51 |
| lifeless | done | 23:52 |
| SpamapS | 7 done | 23:52 |
| jeblair | pushed 0.5.3 tag to gear | 23:52 |
| SpamapS | hm 7 showing down still | 23:53 |
| jog0 | so oslo is wedged! https://review.openstack.org/#/c/74097/ | 23:53 |
| *** alexpilotti has joined #openstack-infra | 23:53 | |
| SpamapS | n/m it came up | 23:53 |
| jog0 | http://logs.openstack.org/63/74963/2/check/gate-oslo-incubator-python33/053fb97/console.html#_2014-02-24_22_06_00_333 | 23:53 |
| jog0 | jd__: ping | 23:53 |
| lifeless | fungi: there are some unallocated floating-ip's | 23:54 |
| lifeless | fungi: (of nodepools) | 23:54 |
| lifeless | fungi: but all ERROR instances are now cleared | 23:54 |
| dhellmann | jog0: looking | 23:54 |
| lifeless | fungi: and I see a bunch of active slaves | 23:54 |
| jog0 | 2014-02-24 22:06:00.332 | Downloading/unpacking eventlet>=0.13.0 (from oslo.messaging>=1.3.0a4->pycadf>=0.1.9->-r /home/jenkins/workspace/gate-oslo-incubator-python33/requirements-py3.txt (line 14)) | 23:54 |
| *** salv-orlando has quit IRC | 23:54 | |
| lifeless | and jobs have attached. Yay. thanks | 23:54 |
| jog0 | eventlet!=python 3 friendly | 23:55 |
| *** salv-orlando has joined #openstack-infra | 23:55 | |
| * jog0 files a bug | 23:55 | |
| jeblair | jog0: i think we just merged the change that enabled that job today | 23:55 |
| fungi | lifeless: the delete-state nodecount is dropping too | 23:55 |
| jog0 | jeblair: yup | 23:55 |
| jog0 | this morning | 23:55 |
| dhellmann | yeah, we can fix it in the incubator repo | 23:56 |
| dhellmann | for some reason the incubator is installing oslo.messaging, which brings in eventlet | 23:56 |
| jog0 | dhellmann: ohh pycadf | 23:56 |
| dhellmann | wtf, why does pycadf require oslo.messaging? | 23:56 |
| jog0 | dhellmann: to keep you awake at night | 23:57 |
| lifeless | jeblair: fungi: were you impacted by this outage in any non-trivial way ? | 23:57 |
| dhellmann | :-| | 23:57 |
| jeblair | lifeless: no | 23:57 |
| lifeless | yaaaaay | 23:57 |
| * lifeless runs arund like kermit | 23:57 | |
| jeblair | :) | 23:58 |
| fungi | lifeless: flail those kermit arms wildly | 23:58 |
| jog0 | dhellmann: so no py33 gate on pyacdf | 23:59 |
| jeblair | clarkb: can you review https://review.openstack.org/#/c/76042/ ? | 23:59 |
| dhellmann | jog0: yeah, I'm testing whether that's enough | 23:59 |
| jog0 | ergo it shouldn't be in py33 set in oslo-incubator | 23:59 |
| *** fbo is now known as fbo_away | 23:59 | |
| dhellmann | yeah | 23:59 |
| *** e0ne has joined #openstack-infra | 23:59 | |
| fungi | clarkb: so, there are no trivial mechanisms i can find to preconfigure elasticsearch not to automatically start when the package gets installed. instead i think i'll just move the class instantiation for that pattern to a separate change | 23:59 |
| clarkb | jog0: sure | 23:59 |
| jeblair | clarkb: also, i'd like to get that into prod asap. do you think there's a chance logstash might catch up tonight? if not, i'm thinking about declaring queue bankruptcy to get that in | 23:59 |
| jog0 | dhellmann: ahh https://github.com/openstack/pycadf/commit/fa802a753d00b4e61eebbc7360caecffba3d7852 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!