| *** emagana has quit IRC | 00:00 | |
| *** zz_avozza is now known as avozza | 00:01 | |
| *** shashankhegde has joined #openstack-infra | 00:01 | |
| *** armax has quit IRC | 00:01 | |
| *** harlowja_at_home has joined #openstack-infra | 00:02 | |
| *** sandywalsh_ has quit IRC | 00:03 | |
| *** Masahiro has joined #openstack-infra | 00:07 | |
| *** e0ne has joined #openstack-infra | 00:08 | |
| *** alexpilotti has quit IRC | 00:08 | |
| *** Longgeek has joined #openstack-infra | 00:09 | |
| *** e0ne has quit IRC | 00:10 | |
| *** Masahiro has quit IRC | 00:11 | |
| grue_pm | mm seems I'm getting invited to a lot of 3/4am meetings | 00:12 |
|---|---|---|
| *** shashankhegde has quit IRC | 00:13 | |
| grue_pm | mordred while time may be irrelevent in some cases like this it sucks. I either need to change my sleep patterns or miss stuff. | 00:13 |
| *** Longgeek has quit IRC | 00:14 | |
| *** pc_m has quit IRC | 00:14 | |
| * grue_pm wishes there was more recording of meetings to listent to for information at least. | 00:14 | |
| *** e0ne has joined #openstack-infra | 00:14 | |
| grue_pm | or a summary but it doesnt really help sometimess | 00:15 |
| mordred | grue_pm: well .. all of the pertinent IRC channels are logged - and we _do_ have meeting summaries and stuff | 00:16 |
| grue_pm | mordred: by we do you mean gozer. This is in relation to some hp coffee cloud talk I just got | 00:18 |
| *** e0ne has quit IRC | 00:19 | |
| grue_pm | mordred: cody is rotating team meeitngs a bit - 5:30-6:30 am for me is easier. | 00:19 |
| *** e0ne has joined #openstack-infra | 00:23 | |
| nibalizer | grue_pm: i've not gone to those | 00:24 |
| *** e0ne has quit IRC | 00:26 | |
| *** ddieterly has quit IRC | 00:29 | |
| *** ddieterly has joined #openstack-infra | 00:29 | |
| *** e0ne has joined #openstack-infra | 00:30 | |
| grue_pm | nibilizer: ah okay - I'm just trying to work out which of the volumous meetings/seminars I should be attending in order to learn | 00:32 |
| *** oomichi has joined #openstack-infra | 00:35 | |
| fungi | as far as i'm concerned, all the relevant meetings are in openstack meeting channels in irc ;) | 00:36 |
| fungi | then again, it's been a couple years since i was in any hp-specific meetings. maybe they've added rodeo clowns or something similarly entertaining since then? | 00:37 |
| grue_pm | fungi: I'm trying to work out generally what is good to attend be it corporate or general openstack stuff. | 00:38 |
| jroll | grue_pm: if it's openstack or has food, attend :) | 00:39 |
| * jroll is not hp and you should probably not listen to me :P | 00:39 | |
| * fungi tries to remember how to set his fax machine to receive pizza | 00:39 | |
| jroll | ha | 00:40 |
| grue_pm | jroll: doesnt have to be company related as I want to learn everything and irc pizza is a bit less filling than I'd like | 00:40 |
| *** stevemar has quit IRC | 00:40 | |
| *** alex7376 has joined #openstack-infra | 00:40 | |
| jroll | grue_pm: right, I just have no idea which hp meetings are valuable :P | 00:41 |
| *** alex7376 is now known as asettle | 00:41 | |
| grue_pm | jroll: I wasn't specifically meaning corporate stuff - openstack is way beyond that | 00:42 |
| * grue_pm is sorry she mentioned a company name in infra now ;-p | 00:42 | |
| jroll | ah, I see, sorry | 00:43 |
| grue_pm | jroll: I've already been reminded to ask company specific questions in a different area. I blame the lurgy for my muddy head currently. | 00:44 |
| *** Masahiro has joined #openstack-infra | 00:44 | |
| *** ryanpetrello has joined #openstack-infra | 00:44 | |
| fungi | grue_pm: on a more pleasant note, seeing you in here again, i guess that means after evaluating our crazy project, you didn't run screaming, and actually decided to join in? | 00:45 |
| *** penguinRaider has quit IRC | 00:45 | |
| grue_pm | fungi: yup - I'm now officially working with Cody's bunch. They are awesome and the crazy learning curve actually attracted me. | 00:46 |
| fungi | in that case, welcome to openstack! stick around and we'll have ever so much fun | 00:46 |
| *** e0ne has quit IRC | 00:49 | |
| *** emagana has joined #openstack-infra | 00:50 | |
| *** ryanpetrello has quit IRC | 00:51 | |
| *** boris-42 has quit IRC | 00:52 | |
| *** imcsk8 has quit IRC | 00:53 | |
| *** imcsk8 has joined #openstack-infra | 00:53 | |
| *** e0ne has joined #openstack-infra | 00:53 | |
| *** emagana has quit IRC | 00:54 | |
| *** otter768 has joined #openstack-infra | 00:55 | |
| *** otter768 has quit IRC | 00:59 | |
| *** ryanpetrello has joined #openstack-infra | 01:00 | |
| *** marun has quit IRC | 01:02 | |
| *** e0ne has quit IRC | 01:06 | |
| *** shashankhegde has joined #openstack-infra | 01:08 | |
| *** e0ne has joined #openstack-infra | 01:09 | |
| *** e0ne has quit IRC | 01:11 | |
| *** viglesias has quit IRC | 01:12 | |
| *** ZZelle has quit IRC | 01:13 | |
| *** viglesias has joined #openstack-infra | 01:17 | |
| *** viglesias has joined #openstack-infra | 01:22 | |
| *** avozza is now known as zz_avozza | 01:22 | |
| *** shashankhegde has quit IRC | 01:23 | |
| *** viglesias has joined #openstack-infra | 01:27 | |
| *** wuhg has joined #openstack-infra | 01:31 | |
| *** salv-orlando has joined #openstack-infra | 01:34 | |
| *** harlowja_at_home has quit IRC | 01:37 | |
| *** ryanpetrello has quit IRC | 01:37 | |
| *** emagana has joined #openstack-infra | 01:38 | |
| *** tnovacik has quit IRC | 01:42 | |
| *** yaguang has joined #openstack-infra | 01:50 | |
| *** ddieterly has quit IRC | 01:54 | |
| *** emagana has quit IRC | 01:57 | |
| *** Longgeek has joined #openstack-infra | 01:58 | |
| *** Masahiro has quit IRC | 02:01 | |
| *** Longgeek has quit IRC | 02:02 | |
| *** Masahiro has joined #openstack-infra | 02:04 | |
| *** stevemar has joined #openstack-infra | 02:05 | |
| *** fifieldt has joined #openstack-infra | 02:05 | |
| *** otter768 has joined #openstack-infra | 02:07 | |
| *** ryanpetrello has joined #openstack-infra | 02:10 | |
| *** viglesias has quit IRC | 02:20 | |
| *** viglesias has joined #openstack-infra | 02:20 | |
| *** xyang1 has quit IRC | 02:22 | |
| *** camunoz is now known as camunoz_away | 02:24 | |
| *** ddieterly has joined #openstack-infra | 02:24 | |
| *** ddieterly has quit IRC | 02:26 | |
| *** ddieterly has joined #openstack-infra | 02:26 | |
| *** xyang1 has joined #openstack-infra | 02:27 | |
| *** emagana has joined #openstack-infra | 02:28 | |
| *** ivar-lazzaro has quit IRC | 02:30 | |
| *** ddieterly has quit IRC | 02:31 | |
| *** baoli has joined #openstack-infra | 02:31 | |
| *** emagana has quit IRC | 02:33 | |
| *** ddieterly has joined #openstack-infra | 02:37 | |
| *** ryanpetrello has quit IRC | 02:44 | |
| *** dkranz has quit IRC | 02:45 | |
| *** baoli has quit IRC | 02:49 | |
| *** baoli has joined #openstack-infra | 02:49 | |
| *** ryanpetrello has joined #openstack-infra | 02:51 | |
| *** sputnik13 has joined #openstack-infra | 02:55 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers https://review.openstack.org/118993 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Fix pep8 issues https://review.openstack.org/115187 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals https://review.openstack.org/97390 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically https://review.openstack.org/119534 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept https://review.openstack.org/121528 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Call driver methods more dynamically https://review.openstack.org/119533 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 02:58 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 02:58 |
| *** achanda has joined #openstack-infra | 03:00 | |
| *** achanda has quit IRC | 03:02 | |
| *** sputnik13 has quit IRC | 03:02 | |
| *** yamamoto_ has joined #openstack-infra | 03:04 | |
| *** Masahiro has quit IRC | 03:07 | |
| *** baoli has quit IRC | 03:13 | |
| *** ddieterly has quit IRC | 03:15 | |
| *** ihrachyshka has quit IRC | 03:21 | |
| *** bhunter71 has joined #openstack-infra | 03:22 | |
| *** ryanpetrello has quit IRC | 03:23 | |
| *** fandi has joined #openstack-infra | 03:23 | |
| *** emagana has joined #openstack-infra | 03:23 | |
| *** camunoz_away is now known as camunoz | 03:29 | |
| *** Masahiro has joined #openstack-infra | 03:30 | |
| *** sabari is now known as zz_sabari | 03:31 | |
| *** david-lyle_afk has quit IRC | 03:31 | |
| *** zz_sabari is now known as sabari | 03:31 | |
| *** baoli has joined #openstack-infra | 03:32 | |
| *** mestery has joined #openstack-infra | 03:34 | |
| *** Masahiro has quit IRC | 03:35 | |
| *** bhunter71 has quit IRC | 03:36 | |
| *** sabari is now known as zz_sabari | 03:36 | |
| *** obondarev has joined #openstack-infra | 03:38 | |
| *** Masahiro has joined #openstack-infra | 03:40 | |
| *** miqui_ has quit IRC | 03:40 | |
| *** ddieterly has joined #openstack-infra | 03:40 | |
| *** amotoki__away is now known as amotoki | 03:41 | |
| *** david-lyle_afk has joined #openstack-infra | 03:43 | |
| *** baoli has quit IRC | 03:46 | |
| *** yaguang has quit IRC | 03:46 | |
| *** baoli has joined #openstack-infra | 03:46 | |
| *** Masahiro has quit IRC | 03:48 | |
| *** david-lyle_afk has quit IRC | 03:50 | |
| *** Masahiro has joined #openstack-infra | 03:52 | |
| *** bradjones has quit IRC | 03:53 | |
| *** ivar-lazzaro has joined #openstack-infra | 03:56 | |
| anteaya | heyongli: okay great | 03:58 |
| *** zz_sabari is now known as sabari | 04:02 | |
| *** david-lyle_afk has joined #openstack-infra | 04:02 | |
| *** david-lyle_afk has quit IRC | 04:02 | |
| *** Masahiro has quit IRC | 04:02 | |
| *** marun has joined #openstack-infra | 04:04 | |
| *** sabari is now known as zz_sabari | 04:07 | |
| *** otter768 has quit IRC | 04:09 | |
| *** Masahiro has joined #openstack-infra | 04:09 | |
| *** dimsum__ has quit IRC | 04:11 | |
| *** marun has quit IRC | 04:12 | |
| *** jerryz has joined #openstack-infra | 04:13 | |
| *** koolhead17 has joined #openstack-infra | 04:13 | |
| *** yaguang has joined #openstack-infra | 04:19 | |
| *** dkehnm_ has joined #openstack-infra | 04:20 | |
| *** dkehnm has quit IRC | 04:21 | |
| *** dkehnm_ is now known as dkehnm | 04:21 | |
| *** dkehnm has quit IRC | 04:27 | |
| *** dkehnm has joined #openstack-infra | 04:30 | |
| *** emagana has quit IRC | 04:30 | |
| *** emagana has joined #openstack-infra | 04:31 | |
| *** sarob has joined #openstack-infra | 04:32 | |
| *** craigbr has joined #openstack-infra | 04:34 | |
| *** emagana has quit IRC | 04:35 | |
| *** sarob has quit IRC | 04:38 | |
| *** ivar-lazzaro has quit IRC | 04:39 | |
| *** ivar-lazzaro has joined #openstack-infra | 04:39 | |
| *** chandankumar has joined #openstack-infra | 04:39 | |
| *** craigbr has quit IRC | 04:40 | |
| *** ddieterly has quit IRC | 04:42 | |
| *** craigbr has joined #openstack-infra | 04:43 | |
| *** mestery has quit IRC | 04:43 | |
| *** zz_sabari is now known as sabari | 04:45 | |
| *** Longgeek has joined #openstack-infra | 04:47 | |
| *** sabeen has joined #openstack-infra | 04:48 | |
| *** koolhead17 has quit IRC | 04:50 | |
| *** Longgeek has quit IRC | 04:52 | |
| *** koolhead_ has joined #openstack-infra | 04:56 | |
| *** hdd has joined #openstack-infra | 04:56 | |
| *** shashankhegde has joined #openstack-infra | 04:58 | |
| *** david-lyle_afk has joined #openstack-infra | 05:01 | |
| *** koolhead_ has quit IRC | 05:02 | |
| *** koolhead17 has joined #openstack-infra | 05:03 | |
| *** rushiagr_away is now known as rushiagr | 05:07 | |
| *** koolhead17 has quit IRC | 05:07 | |
| *** obondarev has quit IRC | 05:11 | |
| *** dimsum__ has joined #openstack-infra | 05:11 | |
| *** dkehnm has quit IRC | 05:12 | |
| *** ddieterly has joined #openstack-infra | 05:12 | |
| *** ddieterly has quit IRC | 05:14 | |
| *** ddieterl_ has joined #openstack-infra | 05:14 | |
| *** dimsum__ has quit IRC | 05:17 | |
| *** emagana has joined #openstack-infra | 05:18 | |
| *** ddieterl_ has quit IRC | 05:19 | |
| *** baoli has quit IRC | 05:22 | |
| *** emagana has quit IRC | 05:23 | |
| *** ddieterly has joined #openstack-infra | 05:25 | |
| *** hdd has quit IRC | 05:28 | |
| *** craigbr has quit IRC | 05:29 | |
| *** emagana has joined #openstack-infra | 05:29 | |
| *** ddieterly has quit IRC | 05:30 | |
| *** emagana_ has joined #openstack-infra | 05:33 | |
| *** shashankhegde has quit IRC | 05:33 | |
| *** sarob has joined #openstack-infra | 05:34 | |
| *** emagana has quit IRC | 05:34 | |
| *** shashankhegde has joined #openstack-infra | 05:36 | |
| *** koolhead17 has joined #openstack-infra | 05:37 | |
| *** koolhead17 has joined #openstack-infra | 05:37 | |
| *** serverascode___ has quit IRC | 05:37 | |
| *** Masahiro has quit IRC | 05:38 | |
| *** Masahiro has joined #openstack-infra | 05:38 | |
| *** sarob has quit IRC | 05:39 | |
| *** serverascode___ has joined #openstack-infra | 05:40 | |
| *** xyang1 has quit IRC | 05:41 | |
| *** yfried_ has quit IRC | 05:42 | |
| *** shashankhegde has quit IRC | 05:43 | |
| *** sabeen has quit IRC | 05:46 | |
| *** talluri has joined #openstack-infra | 05:57 | |
| *** k4n0 has joined #openstack-infra | 06:01 | |
| *** sputnik13 has joined #openstack-infra | 06:02 | |
| openstackgerrit | Sabari proposed openstack/requirements: Update urllib>=1.8.3 https://review.openstack.org/139545 | 06:09 |
| *** otter768 has joined #openstack-infra | 06:10 | |
| sabari | Can someone pl approve the patch https://review.openstack.org/#/c/139545/ to unblock VMware CI ? | 06:11 |
| *** talluri has quit IRC | 06:12 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add is_debian and uses_debs functions https://review.openstack.org/130768 | 06:14 |
| *** otter768 has quit IRC | 06:15 | |
| openstackgerrit | Merged openstack-infra/os-loganalyze: Add ignored Editors to .gitignore https://review.openstack.org/133982 | 06:16 |
| *** talluri has joined #openstack-infra | 06:18 | |
| openstackgerrit | Merged openstack-infra/puppet-yum: Add .gitreview Rakefile files and update puppet coding style https://review.openstack.org/102876 | 06:21 |
| *** sarob has joined #openstack-infra | 06:23 | |
| *** zz_dimtruck is now known as dimtruck | 06:25 | |
| *** emagana_ has quit IRC | 06:25 | |
| *** ddieterly has joined #openstack-infra | 06:25 | |
| *** emagana has joined #openstack-infra | 06:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Update ssh module to support SUSE https://review.openstack.org/123955 | 06:26 |
| *** sarob has quit IRC | 06:27 | |
| openstackgerrit | Merged openstack-infra/system-config: Document rebuilding the MediaWiki search index https://review.openstack.org/129443 | 06:28 |
| *** Masahiro has quit IRC | 06:29 | |
| *** ddieterly has quit IRC | 06:30 | |
| *** emagana has quit IRC | 06:30 | |
| *** Masahiro has joined #openstack-infra | 06:31 | |
| *** tylerdurden has joined #openstack-infra | 06:32 | |
| *** shakamunyi has quit IRC | 06:34 | |
| *** Longgeek has joined #openstack-infra | 06:36 | |
| *** sputnik13 has quit IRC | 06:36 | |
| *** vigneshvar has joined #openstack-infra | 06:38 | |
| *** asettle has quit IRC | 06:38 | |
| *** liusheng has joined #openstack-infra | 06:39 | |
| *** Longgeek has quit IRC | 06:40 | |
| *** mrunge has joined #openstack-infra | 06:47 | |
| *** mrunge has quit IRC | 06:47 | |
| *** yfried_ has joined #openstack-infra | 06:48 | |
| *** liusheng has quit IRC | 06:51 | |
| *** liusheng has joined #openstack-infra | 06:52 | |
| *** koolhead17 has quit IRC | 06:53 | |
| *** koolhead17 has joined #openstack-infra | 06:54 | |
| *** koolhead17 has joined #openstack-infra | 06:54 | |
| *** emagana has joined #openstack-infra | 06:59 | |
| *** yfried_ has quit IRC | 07:00 | |
| *** koolhead_ has joined #openstack-infra | 07:03 | |
| *** stevemar has quit IRC | 07:03 | |
| *** emagana has quit IRC | 07:04 | |
| *** koolhead17 has quit IRC | 07:05 | |
| *** yfried_ has joined #openstack-infra | 07:07 | |
| *** camunoz is now known as camunoz_gone | 07:09 | |
| *** dimtruck is now known as zz_dimtruck | 07:10 | |
| *** ddieterly has joined #openstack-infra | 07:25 | |
| *** jlibosva has joined #openstack-infra | 07:29 | |
| *** ddieterly has quit IRC | 07:30 | |
| *** jgjl has joined #openstack-infra | 07:30 | |
| heyongli | anteaya, the spec is uploaded : https://review.openstack.org/#/c/139927/ | 07:33 |
| *** ivar-lazzaro has quit IRC | 07:33 | |
| grue_pm | 1 | 07:36 |
| *** talluri has quit IRC | 07:36 | |
| *** Longgeek has joined #openstack-infra | 07:37 | |
| *** belmoreira has joined #openstack-infra | 07:37 | |
| *** Longgeek has quit IRC | 07:37 | |
| *** Longgeek has joined #openstack-infra | 07:37 | |
| *** alex7376 has joined #openstack-infra | 07:40 | |
| *** mrunge has joined #openstack-infra | 07:41 | |
| *** jgallard has joined #openstack-infra | 07:48 | |
| *** miqui has quit IRC | 07:51 | |
| *** emagana has joined #openstack-infra | 07:53 | |
| *** teran has quit IRC | 07:55 | |
| *** rakhmerov has quit IRC | 07:58 | |
| *** emagana has quit IRC | 07:58 | |
| *** tsufiev has quit IRC | 07:58 | |
| *** katyafervent has quit IRC | 07:59 | |
| *** dteselkin has quit IRC | 07:59 | |
| *** HeOS_ has quit IRC | 07:59 | |
| *** akuznetsova has quit IRC | 08:00 | |
| *** IvanBerezovskiy1 has joined #openstack-infra | 08:01 | |
| *** ala_ has joined #openstack-infra | 08:02 | |
| *** liusheng has quit IRC | 08:02 | |
| *** liusheng has joined #openstack-infra | 08:03 | |
| *** talluri has joined #openstack-infra | 08:05 | |
| *** rakhmerov has joined #openstack-infra | 08:07 | |
| *** akuznetsova has joined #openstack-infra | 08:08 | |
| *** ihrachyshka has joined #openstack-infra | 08:08 | |
| *** BobBall_AWOL is now known as BobBall | 08:09 | |
| *** katyafervent has joined #openstack-infra | 08:09 | |
| *** dteselkin has joined #openstack-infra | 08:10 | |
| *** otter768 has joined #openstack-infra | 08:11 | |
| *** HeOS_ has joined #openstack-infra | 08:11 | |
| *** Masahiro has quit IRC | 08:11 | |
| *** sarob has joined #openstack-infra | 08:11 | |
| *** jamielennox is now known as jamielennox|away | 08:12 | |
| *** talluri has quit IRC | 08:14 | |
| *** tsufiev has joined #openstack-infra | 08:14 | |
| *** talluri has joined #openstack-infra | 08:15 | |
| *** otter768 has quit IRC | 08:16 | |
| *** jistr has joined #openstack-infra | 08:16 | |
| *** armax has joined #openstack-infra | 08:16 | |
| *** Masahiro has joined #openstack-infra | 08:16 | |
| *** arxcruz has joined #openstack-infra | 08:16 | |
| *** sarob has quit IRC | 08:16 | |
| *** ildikov_ has quit IRC | 08:20 | |
| *** amuller has joined #openstack-infra | 08:21 | |
| *** zz_avozza is now known as avozza | 08:25 | |
| *** ddieterly has joined #openstack-infra | 08:25 | |
| *** tnovacik has joined #openstack-infra | 08:26 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (eavesdrop) https://review.openstack.org/133704 | 08:28 |
| *** ddieterly has quit IRC | 08:30 | |
| *** Longgeek has quit IRC | 08:31 | |
| *** fandi has quit IRC | 08:31 | |
| *** ildikov_ has joined #openstack-infra | 08:32 | |
| *** emagana has joined #openstack-infra | 08:34 | |
| *** _nadya_ has joined #openstack-infra | 08:36 | |
| *** _nadya_ has quit IRC | 08:38 | |
| *** yolanda has quit IRC | 08:38 | |
| *** _nadya__ has joined #openstack-infra | 08:38 | |
| *** Nakato_ is now known as Nakato | 08:38 | |
| *** emagana has quit IRC | 08:39 | |
| *** Longgeek has joined #openstack-infra | 08:40 | |
| *** ddieterly has joined #openstack-infra | 08:42 | |
| *** pblaho has joined #openstack-infra | 08:42 | |
| *** markus_z has joined #openstack-infra | 08:43 | |
| *** mpaolino has joined #openstack-infra | 08:43 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (eavesdrop) https://review.openstack.org/133704 | 08:44 |
| *** ddieterly has quit IRC | 08:47 | |
| *** yfried_ is now known as yfried|afk | 08:47 | |
| *** mrmartin has joined #openstack-infra | 08:48 | |
| *** MaxV has joined #openstack-infra | 08:48 | |
| *** gothicmindfood has quit IRC | 08:48 | |
| *** yfried|afk is now known as yfried_ | 08:49 | |
| *** yamamoto_ has quit IRC | 08:50 | |
| openstackgerrit | Merged openstack/requirements: Update urllib>=1.8.3 https://review.openstack.org/139545 | 08:51 |
| *** Longgeek has quit IRC | 08:51 | |
| *** jcoufal has joined #openstack-infra | 08:54 | |
| *** armax has quit IRC | 08:57 | |
| *** teran has joined #openstack-infra | 08:58 | |
| *** Longgeek has joined #openstack-infra | 08:58 | |
| odyssey4me | hi there - if anyone's around, since https://review.openstack.org/#/c/139676/ merged our check is showing as 'NOT_REGISTERED' - does a dvsm job have to be registered somewhere? | 08:59 |
| *** jlibosva has quit IRC | 09:00 | |
| *** jlibosva has joined #openstack-infra | 09:02 | |
| *** Masahiro has quit IRC | 09:05 | |
| *** fandi has joined #openstack-infra | 09:06 | |
| *** Hal_ has joined #openstack-infra | 09:06 | |
| *** Masahiro has joined #openstack-infra | 09:08 | |
| *** salv-orlando has quit IRC | 09:10 | |
| *** emagana has joined #openstack-infra | 09:13 | |
| *** dimsum__ has joined #openstack-infra | 09:15 | |
| *** gothicmindfood has joined #openstack-infra | 09:15 | |
| *** yfried_ is now known as yfried|afk | 09:16 | |
| *** yfried|afk is now known as yfried_ | 09:17 | |
| *** derekh has joined #openstack-infra | 09:18 | |
| *** emagana has quit IRC | 09:18 | |
| *** yamamoto has joined #openstack-infra | 09:19 | |
| *** afazekas has joined #openstack-infra | 09:19 | |
| *** dimsum__ has quit IRC | 09:20 | |
| *** yamamoto has quit IRC | 09:22 | |
| *** oomichi has quit IRC | 09:22 | |
| *** jedimike has joined #openstack-infra | 09:24 | |
| *** yaguang has quit IRC | 09:31 | |
| *** jlibosva has quit IRC | 09:31 | |
| *** jlibosva has joined #openstack-infra | 09:32 | |
| *** dtantsur|afk is now known as dtantsur | 09:34 | |
| *** yaguang has joined #openstack-infra | 09:44 | |
| *** skolekonov has joined #openstack-infra | 09:44 | |
| *** nfedotov has joined #openstack-infra | 09:46 | |
| *** rlandy has joined #openstack-infra | 09:46 | |
| *** e0ne has joined #openstack-infra | 09:48 | |
| *** sabari is now known as zz_sabari | 09:48 | |
| *** lttrl has joined #openstack-infra | 09:50 | |
| *** Masahiro has quit IRC | 09:56 | |
| *** kingia has joined #openstack-infra | 09:57 | |
| *** kingia is now known as iain-king | 09:57 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Pass change and pipeline to the status_url https://review.openstack.org/86900 | 09:58 |
| *** belmoreira has quit IRC | 09:58 | |
| *** Masahiro has joined #openstack-infra | 10:00 | |
| *** sarob has joined #openstack-infra | 10:00 | |
| *** fandi has quit IRC | 10:02 | |
| *** e0ne has quit IRC | 10:04 | |
| *** sarob has quit IRC | 10:05 | |
| *** e0ne has joined #openstack-infra | 10:06 | |
| *** yfried_ has quit IRC | 10:07 | |
| *** emagana has joined #openstack-infra | 10:07 | |
| *** MaxV has quit IRC | 10:08 | |
| *** jp_at_hp has joined #openstack-infra | 10:08 | |
| *** fandi has joined #openstack-infra | 10:09 | |
| *** Masahiro has quit IRC | 10:09 | |
| *** yaguang has quit IRC | 10:10 | |
| *** salv-orlando has joined #openstack-infra | 10:11 | |
| *** _nadya__ has quit IRC | 10:11 | |
| *** emagana has quit IRC | 10:12 | |
| *** otter768 has joined #openstack-infra | 10:12 | |
| *** salv-orlando has quit IRC | 10:16 | |
| *** otter768 has quit IRC | 10:16 | |
| *** cdent has joined #openstack-infra | 10:17 | |
| *** MaxV has joined #openstack-infra | 10:22 | |
| *** yamamoto has joined #openstack-infra | 10:23 | |
| *** bradjones has joined #openstack-infra | 10:25 | |
| *** yamamoto has quit IRC | 10:27 | |
| *** ociuhandu has joined #openstack-infra | 10:29 | |
| *** pelix has joined #openstack-infra | 10:36 | |
| *** ildikov_ has quit IRC | 10:40 | |
| *** sdake has quit IRC | 10:44 | |
| *** andreykurilin_ has joined #openstack-infra | 10:44 | |
| *** rediskin has joined #openstack-infra | 10:44 | |
| *** ldnunes has joined #openstack-infra | 10:47 | |
| *** jistr has quit IRC | 10:51 | |
| *** ildikov_ has joined #openstack-infra | 10:53 | |
| *** ZZelle has joined #openstack-infra | 10:57 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix the postbuildscript documentation https://review.openstack.org/138952 | 10:58 |
| *** ildikov_ has quit IRC | 10:58 | |
| *** ildikov_ has joined #openstack-infra | 10:58 | |
| *** Hal_ has quit IRC | 10:58 | |
| *** emagana has joined #openstack-infra | 11:01 | |
| *** erlon has joined #openstack-infra | 11:02 | |
| *** simonmcc_ is now known as simonmcc | 11:03 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: include publisher.xunit test as doc https://review.openstack.org/135034 | 11:03 |
| *** aysyd has joined #openstack-infra | 11:05 | |
| *** emagana has quit IRC | 11:06 | |
| *** jgjl has quit IRC | 11:07 | |
| *** yfried_ has joined #openstack-infra | 11:08 | |
| *** jistr has joined #openstack-infra | 11:09 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: spec2bp v2 -- support in-review specs https://review.openstack.org/108041 | 11:09 |
| *** Masahiro has joined #openstack-infra | 11:10 | |
| *** mpaolino has quit IRC | 11:11 | |
| *** jgjl has joined #openstack-infra | 11:13 | |
| *** Masahiro has quit IRC | 11:14 | |
| *** aysyd has quit IRC | 11:15 | |
| *** yolanda has joined #openstack-infra | 11:16 | |
| *** andreaf has joined #openstack-infra | 11:19 | |
| *** mpaolino has joined #openstack-infra | 11:20 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/jenkins-job-builder: fix typo, must be 'classifier' to appear on pypi.python.org https://review.openstack.org/139289 | 11:21 |
| *** jgallard has quit IRC | 11:23 | |
| *** ihrachyshka has quit IRC | 11:25 | |
| *** HeOS has quit IRC | 11:28 | |
| *** ihrachyshka has joined #openstack-infra | 11:29 | |
| *** HeOS has joined #openstack-infra | 11:30 | |
| *** andreaf has quit IRC | 11:32 | |
| *** mrmartin has quit IRC | 11:35 | |
| *** fandi has quit IRC | 11:41 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 11:44 | |
| *** andreaf has joined #openstack-infra | 11:47 | |
| *** dimsum__ has joined #openstack-infra | 11:47 | |
| *** avozza is now known as zz_avozza | 11:55 | |
| *** emagana has joined #openstack-infra | 11:56 | |
| *** salv-orlando has joined #openstack-infra | 12:00 | |
| *** emagana has quit IRC | 12:00 | |
| *** acruz_ has joined #openstack-infra | 12:01 | |
| *** dimsum__ is now known as dims | 12:05 | |
| *** salv-orlando has quit IRC | 12:05 | |
| *** luisjariz has joined #openstack-infra | 12:09 | |
| *** otter768 has joined #openstack-infra | 12:12 | |
| *** andreykurilin_ has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| *** mase_x200 has joined #openstack-infra | 12:15 | |
| *** otter768 has quit IRC | 12:17 | |
| *** Hal_ has joined #openstack-infra | 12:20 | |
| *** jpich has joined #openstack-infra | 12:23 | |
| *** mrmartin has joined #openstack-infra | 12:23 | |
| *** HACKING-FACEBOOK has quit IRC | 12:24 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 12:26 | |
| *** zz_avozza is now known as avozza | 12:26 | |
| *** jistr has quit IRC | 12:27 | |
| *** mpaolino has quit IRC | 12:28 | |
| *** jistr has joined #openstack-infra | 12:28 | |
| *** HACKING-FACEBOOK has quit IRC | 12:29 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add examples for 'predefined-parameters' https://review.openstack.org/137890 | 12:29 |
| *** jgjl has quit IRC | 12:30 | |
| *** HACKING-FACEBOOK has joined #openstack-infra | 12:30 | |
| *** jamielennox|away is now known as jamielennox | 12:31 | |
| *** _nadya_ has joined #openstack-infra | 12:32 | |
| *** mrunge has quit IRC | 12:32 | |
| openstackgerrit | Antoine Musso proposed stackforge/python-jenkins: Workflow documentation is now in infra-manual https://review.openstack.org/139821 | 12:32 |
| *** luisjariz has quit IRC | 12:32 | |
| *** iain-king has quit IRC | 12:33 | |
| *** mrunge has joined #openstack-infra | 12:35 | |
| *** jgallard has joined #openstack-infra | 12:36 | |
| *** adalbas has joined #openstack-infra | 12:36 | |
| openstackgerrit | Merged stackforge/python-jenkins: fix typo, must be 'classifier' to appear on pypi.python.org https://review.openstack.org/139286 | 12:39 |
| *** jamielennox is now known as jamielennox|away | 12:41 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Remove tripleo control variables https://review.openstack.org/124334 | 12:41 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Use a job-template for the tripleo jobs https://review.openstack.org/124335 | 12:41 |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add a tripleo juno overcloud jobs https://review.openstack.org/130540 | 12:41 |
| *** koolhead_ has quit IRC | 12:42 | |
| *** Masahiro has joined #openstack-infra | 12:44 | |
| *** _nadya_ has quit IRC | 12:48 | |
| *** Masahiro has quit IRC | 12:49 | |
| *** emagana has joined #openstack-infra | 12:50 | |
| *** koolhead17 has joined #openstack-infra | 12:50 | |
| *** HeOS has quit IRC | 12:51 | |
| *** sandywalsh has joined #openstack-infra | 12:52 | |
| *** amotoki has quit IRC | 12:54 | |
| openstackgerrit | Merged stackforge/python-jenkins: Workflow documentation is now in infra-manual https://review.openstack.org/139821 | 12:54 |
| *** emagana has quit IRC | 12:55 | |
| *** cnesa has joined #openstack-infra | 12:55 | |
| *** vigneshvar has quit IRC | 12:58 | |
| *** HeOS has joined #openstack-infra | 12:59 | |
| Kiall | Is Zuul is the process of exploding? "Queue lengths: 1777 events, 35 results. " with virtually nothing running (20 check, 15 gate) | 12:59 |
| Kiall | and it suddenly cleared.. heh | 13:00 |
| *** salv-orlando has joined #openstack-infra | 13:01 | |
| *** chandankumar has quit IRC | 13:02 | |
| sdague | merge check is probably back | 13:02 |
| *** _nadya_ has joined #openstack-infra | 13:03 | |
| *** dprince has joined #openstack-infra | 13:04 | |
| *** salv-orlando has quit IRC | 13:05 | |
| *** mbacchi has joined #openstack-infra | 13:06 | |
| *** mfink has quit IRC | 13:07 | |
| *** alexpilotti has joined #openstack-infra | 13:08 | |
| *** bknudson has quit IRC | 13:08 | |
| *** hashar has joined #openstack-infra | 13:10 | |
| *** koolhead17 has quit IRC | 13:13 | |
| *** koolhead17 has joined #openstack-infra | 13:13 | |
| *** koolhead17 has quit IRC | 13:18 | |
| *** obondarev has joined #openstack-infra | 13:19 | |
| *** dkehnm has joined #openstack-infra | 13:21 | |
| *** ihrachyshka_ has joined #openstack-infra | 13:22 | |
| *** salv-orlando has joined #openstack-infra | 13:22 | |
| *** ihrachyshka has quit IRC | 13:23 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/git-review: Fix encoding header https://review.openstack.org/139995 | 13:25 |
| openstackgerrit | Julien Danjou proposed openstack-infra/git-review: Allow to specify which commit to send https://review.openstack.org/139701 | 13:25 |
| *** koolhead17 has joined #openstack-infra | 13:29 | |
| odyssey4me | Do dvsm jobs need to be registered or something? I'm getting a 'NOT_REGISTERED' result in a new dvsm job. | 13:31 |
| *** sweston_ has joined #openstack-infra | 13:32 | |
| odyssey4me | The commit that brought it into being is this one: https://review.openstack.org/#/c/139676/ | 13:32 |
| sdague | odyssey4me: what's the job that failed? | 13:32 |
| *** doug-fish has joined #openstack-infra | 13:33 | |
| odyssey4me | sdague: check-os-ansible-deployment-dvsm-aio-build | 13:33 |
| sdague | odyssey4me: please provide a patch where it failed | 13:33 |
| odyssey4me | sure: https://review.openstack.org/#/c/139629/ | 13:33 |
| *** baoli has joined #openstack-infra | 13:33 | |
| odyssey4me | There are plenty of others - it's failing every time it runs. | 13:34 |
| *** yfried_ is now known as yfried|afk | 13:36 | |
| *** yfried|afk is now known as yfried_ | 13:36 | |
| *** kgiusti has joined #openstack-infra | 13:36 | |
| sdague | so probably someone better with zuul needs to step in. I suspect there is something more subtle about the structure there. | 13:37 |
| *** radez_g0n3 is now known as radez | 13:39 | |
| odyssey4me | thanks sdague - they should be on in an hour or two | 13:41 |
| *** koolhead17 has quit IRC | 13:43 | |
| *** mpaolino has joined #openstack-infra | 13:43 | |
| *** koolhead17 has joined #openstack-infra | 13:44 | |
| *** emagana has joined #openstack-infra | 13:44 | |
| *** sweston_ has quit IRC | 13:46 | |
| *** mwagner_lap has quit IRC | 13:47 | |
| *** emagana has quit IRC | 13:48 | |
| *** koolhead17 has quit IRC | 13:48 | |
| dstufft | fungi: btw we talked about the SQLAlchemy requirement a while back, I went ahead and submitted a CR to do what I beleive the intent is in aw ay that can be consistently interpreted -> https://review.openstack.org/#/c/139880/ | 13:50 |
| *** mfink has joined #openstack-infra | 13:50 | |
| *** mase_x200 has quit IRC | 13:50 | |
| jd__ | any plan to release a new pbr version btw? | 13:54 |
| *** sabeen1 has joined #openstack-infra | 13:55 | |
| sdague | dstufft: maybe it's time to just uplift to 0.9.7 and call it a day | 13:57 |
| dstufft | sdague: that's doable too! | 13:58 |
| dstufft | sdague: I didn't want to make that choice for Openstack so I just translated | 13:58 |
| sdague | dstufft: why don't you propose that. We aren't really testing with 0.8.x at the moment anyway, and lifting that on master before the first milestone seems sane | 13:58 |
| dstufft | sdague: should I make it a new CR or just update the current? | 13:59 |
| sdague | let's just update the current one | 13:59 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static) https://review.openstack.org/140004 | 13:59 |
| sdague | I'll leave a comment on the current patch to explain the direction change | 13:59 |
| *** cnesa has quit IRC | 14:00 | |
| *** cnesa has joined #openstack-infra | 14:01 | |
| *** dkranz has joined #openstack-infra | 14:01 | |
| *** talluri has quit IRC | 14:02 | |
| *** _nadya_ has quit IRC | 14:03 | |
| *** ala_ has quit IRC | 14:03 | |
| openstackgerrit | Donald Stufft proposed openstack/requirements: Depend on any 0.9.x SQLAlchemy that is >= 0.9.7 https://review.openstack.org/139880 | 14:04 |
| sdague | dstufft: +2 ... thanks | 14:05 |
| dstufft | sdague: okies, updated, thanks | 14:06 |
| *** ala_ has joined #openstack-infra | 14:06 | |
| *** pblaho has quit IRC | 14:06 | |
| fungi | odyssey4me: i replied to you over the weekend. https://review.openstack.org/139803 should fix it | 14:06 |
| SergeyLukjanov | sdague, super fast reviewing :) https://review.openstack.org/#/c/139880/3 | 14:06 |
| jcoufal | anybody from core devstack, who can help to get this patch merged? https://review.openstack.org/#/c/138337/ (already reviewed, needs just approvals) | 14:06 |
| jcoufal | it's blocking horizon patches to get in | 14:06 |
| sdague | jcoufal: +A | 14:07 |
| odyssey4me | lol, silly me - thanks fungi - I missed your reply in my scrollback | 14:07 |
| jcoufal | sdague: thanks | 14:07 |
| *** pblaho has joined #openstack-infra | 14:08 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static) https://review.openstack.org/140004 | 14:08 |
| sdague | jcoufal: we should probably remove that cells job from horizon anyway | 14:08 |
| sdague | it doesn't test horizon | 14:08 |
| *** acruz_ has quit IRC | 14:08 | |
| jcoufal | sdague: sounds reasonable to me | 14:08 |
| *** krtaylor has quit IRC | 14:09 | |
| fungi | odyssey4me: yeah, we do tend to be a chatty bunch in here, even over weekends and holidays | 14:10 |
| openstackgerrit | Merged openstack-infra/project-config: Correct misnamed check-{name}-dsvm-aio-build jobs https://review.openstack.org/139803 | 14:12 |
| *** otter768 has joined #openstack-infra | 14:13 | |
| *** _nadya_ has joined #openstack-infra | 14:14 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Remove dsvm-cells tests from places they don't do anything https://review.openstack.org/140011 | 14:14 |
| sdague | jcoufal: ^^^ | 14:14 |
| *** ddieterly has joined #openstack-infra | 14:14 | |
| jcoufal | sdague: nice | 14:15 |
| *** dkliban_afk is now known as dkliban | 14:16 | |
| *** mikedillion has joined #openstack-infra | 14:17 | |
| *** amitgandhinz has joined #openstack-infra | 14:18 | |
| *** otter768 has quit IRC | 14:18 | |
| *** salv-orlando has quit IRC | 14:21 | |
| *** IvanBerezovskiy1 has left #openstack-infra | 14:21 | |
| odyssey4me | fungi odd, the patch has merged, but the patch I put into the repo afterwards still ran with the old name - does zuul/jenkins need some time to refresh? | 14:21 |
| *** ryanpetrello has joined #openstack-infra | 14:22 | |
| *** koolhead17 has joined #openstack-infra | 14:23 | |
| *** mjturek has joined #openstack-infra | 14:23 | |
| fungi | odyssey4me: yes, the merge at 14:12 above would get picked up by the 14:15 puppet apply pass which according to the zuul status page caused it to reload its configuration at 14:22:44 | 14:24 |
| *** cpowell has joined #openstack-infra | 14:24 | |
| fungi | odyssey4me: looks like you tried about one minute early ;) | 14:24 |
| *** yfried_ is now known as yfried|afk | 14:24 | |
| odyssey4me | fungi ah, thanks - will retry now | 14:24 |
| *** aysyd has joined #openstack-infra | 14:24 | |
| *** ayoung has joined #openstack-infra | 14:26 | |
| *** ayoung has quit IRC | 14:26 | |
| *** weshay has joined #openstack-infra | 14:27 | |
| *** yfried|afk is now known as yfried_ | 14:30 | |
| *** krtaylor has joined #openstack-infra | 14:30 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: cloner: support for ref-updated parameters https://review.openstack.org/137675 | 14:30 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: cloner: set zuul_branch on ref-update events https://review.openstack.org/138433 | 14:30 |
| *** baoli has quit IRC | 14:31 | |
| openstackgerrit | Merged openstack-infra/git-review: Fix encoding header https://review.openstack.org/139995 | 14:31 |
| *** baoli has joined #openstack-infra | 14:32 | |
| *** Masahiro has joined #openstack-infra | 14:33 | |
| *** ayoung has joined #openstack-infra | 14:33 | |
| *** mriedem has joined #openstack-infra | 14:35 | |
| *** amuller_ has joined #openstack-infra | 14:35 | |
| *** amuller has quit IRC | 14:35 | |
| *** mestery has joined #openstack-infra | 14:36 | |
| *** mestery has quit IRC | 14:37 | |
| *** mestery has joined #openstack-infra | 14:37 | |
| *** Masahiro has quit IRC | 14:38 | |
| *** emagana has joined #openstack-infra | 14:38 | |
| *** mestery has quit IRC | 14:38 | |
| *** craigbr has joined #openstack-infra | 14:40 | |
| *** jaypipes has joined #openstack-infra | 14:40 | |
| *** jerryz has quit IRC | 14:40 | |
| *** obondarev_ has joined #openstack-infra | 14:43 | |
| *** tonytan4ever has joined #openstack-infra | 14:43 | |
| *** obondarev has quit IRC | 14:43 | |
| *** mpaolino has quit IRC | 14:43 | |
| *** emagana has quit IRC | 14:43 | |
| *** mpaolino has joined #openstack-infra | 14:44 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static) https://review.openstack.org/140004 | 14:44 |
| *** yamamoto has quit IRC | 14:48 | |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| *** obondarev_ has quit IRC | 14:51 | |
| hashar | fungi: hello! git://git.openstack.org seems to have some troubles | 14:51 |
| hashar | http://logs.openstack.org/33/138433/4/check/gate-zuul-python27/aca9204/console.html | 14:51 |
| *** armax has joined #openstack-infra | 14:51 | |
| *** eharney has quit IRC | 14:52 | |
| fungi | hashar: i'll look at the graphs for the git servers, but it could just as easily be a network problem in rackspace dfw | 14:52 |
| *** yamamoto has quit IRC | 14:52 | |
| *** signed8bit has joined #openstack-infra | 14:52 | |
| *** marcusvrn has quit IRC | 14:52 | |
| *** dkehnm_ has joined #openstack-infra | 14:55 | |
| fungi | several of the git servers look fairly busy, but none look overloaded | 14:55 |
| *** dkehnm has quit IRC | 14:56 | |
| *** dkehnm_ is now known as dkehnm | 14:56 | |
| hashar | fungi: seems it is solved now. | 14:56 |
| hashar | according to a logstash search for 'fatal:' :) | 14:56 |
| *** armax has quit IRC | 14:56 | |
| pleia2 | good morning | 14:57 |
| fungi | yeah, the graphs show there was a spike in activity for 30 minutes or so, but not high enough that it looks like it should have caused that | 14:57 |
| fungi | probably just the spike from our image updates | 14:57 |
| *** esker has joined #openstack-infra | 14:58 | |
| *** obondarev has joined #openstack-infra | 14:58 | |
| *** ddieterly has quit IRC | 14:58 | |
| *** mwagner_lap has joined #openstack-infra | 15:00 | |
| *** esker has quit IRC | 15:00 | |
| *** pradk has joined #openstack-infra | 15:00 | |
| *** bknudson has joined #openstack-infra | 15:00 | |
| *** julim has joined #openstack-infra | 15:02 | |
| *** obondarev has quit IRC | 15:03 | |
| openstackgerrit | Merged openstack/requirements: Drop greenio dependency https://review.openstack.org/138748 | 15:05 |
| *** amuller_ has quit IRC | 15:05 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add conf to replace nova V2 endpoint with V2.1 API https://review.openstack.org/138949 | 15:05 |
| *** esker has joined #openstack-infra | 15:06 | |
| *** rushiagr is now known as rushiagr_away | 15:06 | |
| *** jgjl has joined #openstack-infra | 15:06 | |
| *** salv-orlando has joined #openstack-infra | 15:08 | |
| *** obondarev has joined #openstack-infra | 15:08 | |
| *** pblaho_ has joined #openstack-infra | 15:10 | |
| *** jgjl has quit IRC | 15:10 | |
| *** armax has joined #openstack-infra | 15:11 | |
| *** salv-orlando has quit IRC | 15:12 | |
| *** armax has quit IRC | 15:13 | |
| *** pblaho has quit IRC | 15:14 | |
| openstackgerrit | Terry Howe proposed openstack/requirements: Add in python-openstacksdk to requirements.txt https://review.openstack.org/136486 | 15:15 |
| *** dmsimard_away is now known as dmsimard | 15:15 | |
| *** hdd has joined #openstack-infra | 15:17 | |
| *** weshay has quit IRC | 15:21 | |
| *** weshay has joined #openstack-infra | 15:21 | |
| *** zz_dimtruck is now known as dimtruck | 15:22 | |
| *** sputnik13 has joined #openstack-infra | 15:22 | |
| *** nelsnelson has joined #openstack-infra | 15:23 | |
| *** sputnik13 has quit IRC | 15:23 | |
| *** obondarev has quit IRC | 15:24 | |
| *** ddieterly has joined #openstack-infra | 15:25 | |
| *** marcusvrn has joined #openstack-infra | 15:25 | |
| *** mpaolino has quit IRC | 15:26 | |
| *** e0ne has quit IRC | 15:28 | |
| *** mestery has joined #openstack-infra | 15:29 | |
| *** sarob has joined #openstack-infra | 15:29 | |
| *** baoli has quit IRC | 15:29 | |
| *** chandankumar has joined #openstack-infra | 15:29 | |
| mordred | jd__: is there something in pbr master we need to get out there? | 15:29 |
| mordred | jd__: I'll take a look once I've chugged through morning email and whatnot | 15:29 |
| *** chandankumar has quit IRC | 15:30 | |
| jd__ | mordred: I know I want to use the doc generation stuff patch I merged | 15:31 |
| *** teran has quit IRC | 15:32 | |
| *** andreaf has quit IRC | 15:32 | |
| *** emagana has joined #openstack-infra | 15:32 | |
| *** amuller_ has joined #openstack-infra | 15:32 | |
| *** andreaf has joined #openstack-infra | 15:33 | |
| *** ihrachyshka_ has quit IRC | 15:33 | |
| *** teran has joined #openstack-infra | 15:33 | |
| *** julim has quit IRC | 15:33 | |
| *** sputnik13 has joined #openstack-infra | 15:33 | |
| mordred | ah. gotcha. | 15:34 |
| *** dkehnm has quit IRC | 15:34 | |
| *** mestery has quit IRC | 15:35 | |
| *** sarob has quit IRC | 15:35 | |
| *** julim has joined #openstack-infra | 15:35 | |
| mordred | jd__: oh jeez, we've really waited a while between releases haven't we? | 15:35 |
| *** dannywilson has joined #openstack-infra | 15:36 | |
| *** dprince_ has joined #openstack-infra | 15:36 | |
| *** sdake has joined #openstack-infra | 15:36 | |
| mordred | fungi, dhellmann: you guys know where we are on the lifeless semver patches? | 15:36 |
| jd__ | yeah not really release early release often anymore :) | 15:36 |
| mordred | jd__: :) | 15:36 |
| *** mestery has joined #openstack-infra | 15:36 | |
| *** baoli has joined #openstack-infra | 15:36 | |
| dhellmann | mordred: I think master was broken, so we didn't release them. I don't know if he has fixes up yet. | 15:36 |
| *** dprince has quit IRC | 15:36 | |
| mordred | dhellmann: yah. I was afraid of that :( | 15:37 |
| fungi | no clue--haven't looked back into it recently | 15:37 |
| *** emagana has quit IRC | 15:37 | |
| dhellmann | mordred: it's a shame we can't use https://pypi.python.org/pypi/semver | 15:37 |
| mordred | dhellmann: yah. | 15:38 |
| *** _shaps_ has joined #openstack-infra | 15:38 | |
| *** wenlock has joined #openstack-infra | 15:38 | |
| *** pc_m has joined #openstack-infra | 15:38 | |
| mordred | dhellmann: the packagers would hate pbr even more than they already do if we vendored a library in though, and transitive deps during setup_requires brings on cthulu levels of pain | 15:38 |
| dhellmann | mordred: yeah | 15:39 |
| *** e0ne has joined #openstack-infra | 15:39 | |
| *** k4n0 has quit IRC | 15:40 | |
| *** pc_m_ has joined #openstack-infra | 15:40 | |
| *** mikedillion has quit IRC | 15:40 | |
| *** weshay has quit IRC | 15:41 | |
| *** eharney has joined #openstack-infra | 15:41 | |
| *** weshay has joined #openstack-infra | 15:42 | |
| *** obondarev has joined #openstack-infra | 15:42 | |
| *** ala_ has quit IRC | 15:43 | |
| *** pc_m has quit IRC | 15:44 | |
| *** ala_ has joined #openstack-infra | 15:45 | |
| *** zz_jgrimm is now known as jgrimm | 15:45 | |
| *** mpaolino has joined #openstack-infra | 15:46 | |
| ttx | fungi: about the stable teams refactoring, how would you like to proceed and when ? If you're stuck with Neutronites today, maybe not now ? | 15:46 |
| *** armax has joined #openstack-infra | 15:47 | |
| *** sdake has quit IRC | 15:47 | |
| fungi | ttx: i'll probably be free by 18:00 utc if you want to take a stab at it then. i expect we'll do something similar to what we did for the third-party testing project-specific acls/groups | 15:47 |
| *** sdake has joined #openstack-infra | 15:48 | |
| *** sdake has quit IRC | 15:48 | |
| *** sdake has joined #openstack-infra | 15:48 | |
| ttx | fungi: hmm, how about we do that on Wednesday ? | 15:48 |
| openstackgerrit | Maish Saidel-Keesing proposed openstack-infra/project-config: Add rss.opml file to repository https://review.openstack.org/140041 | 15:49 |
| fungi | ttx: sure, sounds good | 15:49 |
| ttx | fungi: ok will ping you then | 15:50 |
| *** amotoki has joined #openstack-infra | 15:51 | |
| *** HACKING-FACEBOOK has quit IRC | 15:52 | |
| *** yamamoto has joined #openstack-infra | 15:53 | |
| *** rushiagr_away is now known as rushiagr | 15:55 | |
| *** ihrachyshka_ has joined #openstack-infra | 15:57 | |
| *** yamamoto has quit IRC | 15:58 | |
| *** mestery has quit IRC | 15:59 | |
| *** jaypipes has quit IRC | 15:59 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (static) https://review.openstack.org/140004 | 16:00 |
| krotscheck | Storyboard meeting in #openstack-meeting-3 | 16:01 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of Ask website https://review.openstack.org/140043 | 16:01 |
| mrmartin | fungi: ^^^^^^^^ | 16:01 |
| *** ihrachyshka_ is now known as ihrachyshka | 16:02 | |
| fungi | mrmartin: exciting--thank you!!! | 16:02 |
| *** armax has quit IRC | 16:02 | |
| mrmartin | fungi: more than exciting und wunderschön | 16:02 |
| *** harlowja_at_home has joined #openstack-infra | 16:03 | |
| *** amuller_ is now known as amuller | 16:04 | |
| mrmartin | anyway, we need to find a timeslot and talk about the details, it works in my vagrant, and can migrate the actual ask.openstack.org. but for final deployment I suggest to setup the instance first, migrate the actual site for testing, and not change the public ip until it is passing some acceptance test. | 16:04 |
| mrmartin | do you have the certs for this site? | 16:04 |
| *** sarob has joined #openstack-infra | 16:05 | |
| pleia2 | woo ask.o.o :) | 16:05 |
| fungi | mrmartin: that sounds like a safe plan. let me see if i have access to the key | 16:05 |
| fungi | mrmartin: hrm, i can't ssh into that instance | 16:05 |
| mrmartin | I guess I'll write an email to infra, there is a lot to do | 16:05 |
| *** dkranz has quit IRC | 16:05 | |
| mrmartin | ok, I can :) we could exchange the keys and backups somehow | 16:05 |
| fungi | mrmartin: yes, let's have an ml thread to highlight this patch | 16:06 |
| mrmartin | do we have some example in infra scripts for postgresql db backup? | 16:06 |
| fungi | mrmartin: and since you have ssh access into it, i can just have you add me so i can collect the stuff we'll need to copy to the new server | 16:06 |
| *** david-lyle_afk is now known as david-lyle | 16:06 | |
| fungi | mrmartin: not sure--we likely want to duplicate something for the database backup cron job we use for mysql | 16:07 |
| *** amitgandhinz has quit IRC | 16:07 | |
| clarkb | I take mysql still doesnt work? though migrating pgsql -> mysql probably not worth the trouble | 16:07 |
| mrmartin | ok, basically in middle term, we can migrate this back to mysql, I wanted to avoid that to decrease the number of possible issues now. | 16:07 |
| clarkb | can we? askbot did not work with mysql when I tried it | 16:08 |
| mrmartin | askbot works well with mysql, but the actual deployment runs on pgsql. | 16:08 |
| clarkb | and ++ to minimizing the diff | 16:08 |
| mrmartin | I guess the reason it was running pgsql that it used some full text search capability there, but it is replaced now with solr. | 16:08 |
| *** arxcruz has quit IRC | 16:09 | |
| clarkb | well that and mysql just didnt work | 16:09 |
| clarkb | the migrations were broken among other things iirc | 16:09 |
| fungi | sounds like askbot has likely evolved in the year or so since that was last attempted | 16:09 |
| mrmartin | worked well in my vagrant anyway. | 16:09 |
| clarkb | thats good | 16:09 |
| *** esker has quit IRC | 16:10 | |
| mrmartin | I spent a lot of time discovering misterious python bugs :) The best one was, not to use WSGIPythonOptimize parameter in apache with mod_wsgi, because asserts won't throw an AssertException. And the site basically worked until a point :) | 16:11 |
| fungi | hsh | 16:12 |
| fungi | er, hah | 16:12 |
| *** rushiagr is now known as rushiagr_away | 16:12 | |
| *** salv-orlando has joined #openstack-infra | 16:12 | |
| *** mestery has joined #openstack-infra | 16:12 | |
| *** dannywilson has quit IRC | 16:13 | |
| *** yfried_ has quit IRC | 16:13 | |
| mrmartin | anyway: http://logs.openstack.org/43/140043/1/check/gate-infra-puppet-apply-precise/d59b573/console.html Error: Could not install module 'puppetlabs-stdlib' (v4.3.2) | 16:14 |
| mrmartin | fungi: had you met with something similar before? | 16:14 |
| *** otter768 has joined #openstack-infra | 16:14 | |
| mrmartin | and this one: http://logs.openstack.org/43/140043/1/check/gate-system-config-pep8/eee449a/console.html complains about askbot python code quality | 16:16 |
| *** markmcclain has joined #openstack-infra | 16:16 | |
| *** amuller has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack/requirements: Depend on any 0.9.x SQLAlchemy that is >= 0.9.7 https://review.openstack.org/139880 | 16:16 |
| *** armax has joined #openstack-infra | 16:17 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: cloner: support for ref-updated parameters https://review.openstack.org/137675 | 16:17 |
| *** sputnik13 has quit IRC | 16:17 | |
| *** ala_ has quit IRC | 16:18 | |
| *** armax has quit IRC | 16:18 | |
| fungi | mrmartin: i think this is where the puppet-askbot module errors are coming in http://logs.openstack.org/43/140043/1/check/gate-infra-puppet-apply-precise/d59b573/console.html#_2014-12-08_16_08_22_707 | 16:19 |
| *** otter768 has quit IRC | 16:19 | |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config: oslo.db: make SQLAlchemy jobs voting https://review.openstack.org/140057 | 16:19 |
| mrmartin | Give me a sec, I'm uploading a quick patch, for some typo problems | 16:19 |
| *** aysyd has quit IRC | 16:19 | |
| mrmartin | anyway I needed to add a new puppet module, concat | 16:19 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of Ask website https://review.openstack.org/140043 | 16:20 |
| *** dkranz has joined #openstack-infra | 16:20 | |
| fungi | mrmartin: also, as for the pep8 issues inside modules/askbot/files i wonder if we shouldn't have puppet retrieving those files somehow rather than embedding them into the module's revision control | 16:20 |
| *** aysyd has joined #openstack-infra | 16:21 | |
| mrmartin | can we copy a file with a File resource ? | 16:21 |
| clarkb | yup use a path for source | 16:21 |
| clarkb | or you can set up a symlink too | 16:22 |
| *** Masahiro has joined #openstack-infra | 16:22 | |
| fungi | basically point it to the copy on the file system installed from the askbot source | 16:22 |
| *** achanda has joined #openstack-infra | 16:22 | |
| *** mikedillion has joined #openstack-infra | 16:22 | |
| mrmartin | yep, I can copy from askbot's setup_templates directory | 16:23 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: hipchat bug fix https://review.openstack.org/133997 | 16:23 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Support stracktracing on Geard embedded server https://review.openstack.org/131559 | 16:24 |
| mrmartin | ok, I'll try to solve those problem today, and at least reach a point where askbot puppet can pass the gating with a +1 | 16:24 |
| fungi | mestery: dougwig: getting ready to help coordinate your git repo updates? | 16:24 |
| dougwig | anteaya: https://github.com/dougwig/x-neutron-lbaas (sub vpnaas and fwaas for the other two) | 16:24 |
| mestery | fungi: Yes sir | 16:24 |
| *** jaypipes has joined #openstack-infra | 16:25 | |
| fungi | mrmartin: it may also be worth discussing whether this should start out as separate openstack-infra/puppet-askbot and openstack-infra/puppet-solr git repositories | 16:25 |
| fungi | mestery: dougwig: awesome--just making sure you weren't otherwise occupied with sprint setup problems | 16:26 |
| clarkb | fungi +1 | 16:26 |
| mrmartin | I agree | 16:26 |
| dougwig | warning: advanced services will be broken in neutron for a few days (see the ML). additionally, the new neutron-lbaas, neutron-fwaas, and neutron-vpnaas repos will not be usable for a few days. | 16:26 |
| clarkb | maybe a solr puppet module exists too? | 16:26 |
| *** Masahiro has quit IRC | 16:26 | |
| clarkb | askbot didnt at least not way back when | 16:26 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Doc: ref links for "approval" requirement https://review.openstack.org/130202 | 16:26 |
| *** emagana has joined #openstack-infra | 16:26 | |
| mrmartin | clarkb, yep, this solr have a multi-core support. :) | 16:27 |
| anteaya | fungi: I'm sitting beside them, right now we are working on communication plan and -2'ing patches in flight proposed to the dir that will be split | 16:27 |
| fungi | anteaya: great--also thrilled you're there! | 16:28 |
| *** armax has joined #openstack-infra | 16:28 | |
| mestery | fungi: Thanks, so far we're good, but we know where to find you when we have issues. And anteaya is also here to assist us :) | 16:28 |
| hashar | clarkb: hello! I pushed a doc change for Zuul to document yappi. Could use your review there since you added the related code (doc: https://review.openstack.org/#/c/97708/2/doc/source/zuul.rst,unified ) :] | 16:28 |
| *** salv-orlando has quit IRC | 16:29 | |
| anteaya | fungi: thanks | 16:29 |
| mrmartin | ok, I'm going to eat something and play with the children, sitting here since morning. I'll see back later, and care the askbot patch. | 16:29 |
| clarkb | hashar thanks will look | 16:30 |
| fungi | mrmartin: thanks again for working on that | 16:30 |
| dougwig | anteaya: neutron/db, neutron/services, neutron/tests/unit/services | 16:30 |
| dougwig | anteaya: info on the split: http://lists.openstack.org/pipermail/openstack-dev/2014-December/052439.html | 16:31 |
| *** emagana has quit IRC | 16:31 | |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: add support for pyton and tox matrix project axis https://review.openstack.org/134025 | 16:34 |
| *** mestery has quit IRC | 16:35 | |
| clarkb | hashar: just one tiny misspelling otherwise it looks good to me | 16:36 |
| *** ParsectiX has joined #openstack-infra | 16:36 | |
| hashar | clarkb: let me know, will fix right now :] | 16:37 |
| clarkb | hashar: yup left a comment | 16:37 |
| *** achanda has quit IRC | 16:37 | |
| fungi | other than that typo, lgtm too | 16:37 |
| *** achanda has joined #openstack-infra | 16:38 | |
| hashar | yappi dumpes ...., reset | 16:38 |
| hashar | ahhrhez | 16:38 |
| *** jpich has quit IRC | 16:39 | |
| hashar | clarkb: I meant: maybe 'reset' needs an extra s as well? | 16:39 |
| hashar | it dumpes the informations collected, resetS <--- ? | 16:39 |
| hashar | I am terribly with the third person S | 16:39 |
| clarkb | hashar: if you change it to resets make sure you s/states/state/ | 16:40 |
| *** packet has joined #openstack-infra | 16:40 | |
| clarkb | so that it becomes 'resets all yappi state' | 16:40 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 16:40 |
| *** mestery has joined #openstack-infra | 16:40 | |
| *** tnovacik has quit IRC | 16:40 | |
| clarkb | I think reset all yappi states is correct though | 16:40 |
| clarkb | fungi: ^ | 16:40 |
| hashar | clarkb: then I guess it stopS profiling? | 16:40 |
| *** mpaolino has quit IRC | 16:40 | |
| clarkb | hashar: ya | 16:40 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler https://review.openstack.org/97708 | 16:41 |
| hashar | clarkb: https://review.openstack.org/#/c/97708/2..3/doc/source/zuul.rst :-] | 16:41 |
| fungi | clarkb: agreed, resets->state or reset->states depending on whether there is a singular state of yappi or it maintains multiple individual states | 16:42 |
| clarkb | I am going to make Administrators own the old third party groups now and mark them not visible to the world. Then remove all group members sometime this week | 16:42 |
| fungi | hashar: additional functions | 16:42 |
| clarkb | anteaya: ^ do you want me to capture the current state anywhere before I start doing that? | 16:42 |
| *** achanda has quit IRC | 16:42 | |
| fungi | hashar: and thread stats | 16:43 |
| fungi | hashar: the information collected | 16:43 |
| hashar | fungi: well there are several threads aren't they? | 16:43 |
| fungi | i'll add inline comments | 16:43 |
| fungi | hashar: in that case the threads' states (possessive plural trheads) | 16:43 |
| *** hdd has quit IRC | 16:44 | |
| fungi | er, threads' stats | 16:44 |
| hashar | I need english lessons Oo, | 16:44 |
| *** jgrimm has quit IRC | 16:44 | |
| clarkb | anteaya: I will hold off on that until I hear if you need anymore info | 16:44 |
| clarkb | hashar: its ok I need them too | 16:44 |
| *** maurosr has quit IRC | 16:44 | |
| fungi | hashar: wouldn't help. people who grow up speaking and writing it make at least this many errors :/ | 16:44 |
| *** achanda has joined #openstack-infra | 16:46 | |
| fungi | hashar: okay, i think i have the correct pluralization and possessive punctuation in my latest inline comments | 16:46 |
| openstackgerrit | Roman Podoliaka proposed openstack/requirements: Bump the minimum required version of retrying https://review.openstack.org/140074 | 16:47 |
| *** nfedotov has quit IRC | 16:48 | |
| hashar | fungi: that is nice. Amending thank you very much | 16:48 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler https://review.openstack.org/97708 | 16:49 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Document usage of yappi profiler https://review.openstack.org/97708 | 16:49 |
| hashar | and rebased. Should be fine now | 16:49 |
| *** ParsectiX has quit IRC | 16:51 | |
| *** amotoki has quit IRC | 16:53 | |
| fungi | hashar: perfect--thanks! | 16:53 |
| *** pblaho_ has quit IRC | 16:55 | |
| *** salv-orlando has joined #openstack-infra | 16:55 | |
| *** markus_z has quit IRC | 16:56 | |
| *** ParsectiX has joined #openstack-infra | 16:56 | |
| *** mpaolino has joined #openstack-infra | 16:57 | |
| *** rushiagr_away is now known as rushiagr | 16:57 | |
| *** Hal_ has quit IRC | 16:58 | |
| hashar | one less patch in queue | 16:59 |
| openstackgerrit | Merged openstack-infra/zuul: Document usage of yappi profiler https://review.openstack.org/97708 | 16:59 |
| anteaya | clarkb: capturing current state can't hurt | 17:01 |
| anteaya | and then go ahead | 17:02 |
| anteaya | and yay | 17:02 |
| *** harlowja_at_home has quit IRC | 17:02 | |
| *** Hal_ has joined #openstack-infra | 17:03 | |
| *** ParsectiX has quit IRC | 17:03 | |
| *** hashar has quit IRC | 17:03 | |
| openstackgerrit | Merged openstack-infra/nodepool: Make the external links actual hyperlinks https://review.openstack.org/129309 | 17:04 |
| *** krtaylor has quit IRC | 17:04 | |
| clarkb | anteaya: ok I will list group membership for the two groups. Have a preference on where I stash that? | 17:05 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: add support for pyton and tox matrix project axis https://review.openstack.org/134025 | 17:05 |
| *** krtaylor has joined #openstack-infra | 17:05 | |
| *** ParsectiX has joined #openstack-infra | 17:05 | |
| *** zz_sabari is now known as sabari_ | 17:06 | |
| jeblair | clarkb, fungi: want to do the nodepool swap soonish? | 17:07 |
| jeblair | also good morning | 17:07 |
| anteaya | clarkb: paste is nice | 17:07 |
| *** esker has joined #openstack-infra | 17:07 | |
| anteaya | morning jeblair | 17:07 |
| fungi | jeblair: good morning, and sure | 17:07 |
| *** jmeridth_ has left #openstack-infra | 17:08 | |
| clarkb | jeblair: yup definitely | 17:09 |
| jeblair | clarkb: what needs prepping? | 17:09 |
| clarkb | jeblair: was thinking after we updated neutron repos, but those things can happen concurrently so not a big deal to do it at the same time | 17:09 |
| clarkb | jeblair: nodepool-dev should have all firewall rules set up properly. We should maybe kick off image builds on that host manually so that we have up to date image | 17:09 |
| fungi | there are also several of us if we need to divide attention | 17:09 |
| clarkb | jeblair: and also manually update gear on nodepool-dev | 17:10 |
| *** sputnik13 has joined #openstack-infra | 17:10 | |
| clarkb | jeblair: update gear on zuul.o.o and once that is done we can restart zuul and flip nodepool over | 17:10 |
| *** sabari_ is now known as zz_sabari_ | 17:10 | |
| *** nelsnelson has quit IRC | 17:11 | |
| jeblair | clarkb: we have images on np-dev, right? | 17:11 |
| *** afazekas has quit IRC | 17:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Add fwaas and vpnaas repos to #openstack-neutron https://review.openstack.org/139152 | 17:11 |
| fungi | they're likely outdated, but checking now | 17:11 |
| clarkb | jeblair: we do, but they would be older from the last time the daemon was running | 17:11 |
| fungi | yeah, about 1.5 weeks stale now | 17:11 |
| jeblair | if we wait for that, then we're talking a couple of hours, right? | 17:12 |
| fungi | and we have had some slave script updates go in which would make them break quite a few jobs i think | 17:12 |
| clarkb | jeblair: yes so maybe don't wait and have it update images concurrently | 17:12 |
| clarkb | fungi: oh ya | 17:12 |
| clarkb | fungi: so we do need new images first | 17:12 |
| *** ParsectiX has quit IRC | 17:12 | |
| fungi | i can manually build images now if desired | 17:12 |
| *** avozza is now known as zz_avozza | 17:12 | |
| clarkb | ya I think we should start with that | 17:12 |
| jeblair | fungi: in that case, i think we should probably delete the stale ones too | 17:12 |
| *** mmaglana has joined #openstack-infra | 17:12 | |
| clarkb | ++ to deleting stale images too | 17:12 |
| fungi | jeblair: clarkb: on it now | 17:13 |
| jeblair | okay, i'll go ahead and install gear+proposed change on nodepool-dev and zuul | 17:13 |
| clarkb | anteaya: http://paste.openstack.org/show/147317/ http://paste.openstack.org/show/147318/ http://paste.openstack.org/show/147319/ | 17:14 |
| *** nelsnelson has joined #openstack-infra | 17:14 | |
| *** armax has quit IRC | 17:15 | |
| jeblair | clarkb, fungi: gear is installed both places | 17:15 |
| *** dkranz has quit IRC | 17:16 | |
| *** nelsnelson has quit IRC | 17:17 | |
| *** nelsnelson has joined #openstack-infra | 17:17 | |
| *** baoli has quit IRC | 17:18 | |
| clarkb | then for the flip over, stop nodepool on old host, delete all ready nodes there, start nodepool on new host, delete all nodes on old host in a couple hours/after jobs are done running on them | 17:18 |
| clarkb | we probably want to do the deletion of ready nodes while zuul is off so that they don't transition state | 17:19 |
| *** openstackgerrit has quit IRC | 17:19 | |
| *** openstackgerrit has joined #openstack-infra | 17:19 | |
| *** tylerdurden has quit IRC | 17:20 | |
| *** baoli has joined #openstack-infra | 17:22 | |
| jeblair | clarkb: we need to restart zuul though... | 17:23 |
| *** shashankhegde has joined #openstack-infra | 17:23 | |
| clarkb | jeblair: right, I think we turn it off, do deletion then start zuul | 17:24 |
| jeblair | clarkb: so i think it's stop zuul, stop old nodepool, delete all old nodes, start new nodepool, start zuul | 17:24 |
| clarkb | that works | 17:24 |
| *** doug-fish has quit IRC | 17:24 | |
| fungi | though deleting that many nodes will take probably 20 minutes or more | 17:24 |
| *** TimO_ has joined #openstack-infra | 17:25 | |
| *** armax has joined #openstack-infra | 17:25 | |
| clarkb | ya so the optimization is we can just delete ready nodes first | 17:26 |
| fungi | i suppose we could minimize missed events by putting zuul in graceful shutdown, stopping nodepool, and then removing ready/building nodes in parallel with the running jobs wrapping up | 17:26 |
| *** MaxV has quit IRC | 17:26 | |
| *** marun has joined #openstack-infra | 17:26 | |
| *** armax has quit IRC | 17:27 | |
| fungi | and possibly then deleting straggler used/delete state nodes in parallel with starting nodepoold on nodepool-dev | 17:27 |
| clarkb | oh I see why zuul restart is important means we don't ahve to wait on the other nodes. So ya delete ready, start new nodepool, start zuul, delete all other old nodepool nodes would potentially be quicker | 17:27 |
| jeblair | yeah, i think that would work. but then we're committed to whatever timetable zuul has | 17:27 |
| jeblair | that was for fungi ^ | 17:27 |
| *** nelsnelson is now known as nels|afk | 17:28 | |
| TimO_ | As of Friday, my ssh gerrit stream events stopped (ssh -p 29418 tkonski@review.openstack.org gerrit stream events) for my OS Jenkins testing | 17:28 |
| fungi | true. weighing missing more events in the stream vs letting jobs pile up for longer | 17:28 |
| *** tnovacik has joined #openstack-infra | 17:28 | |
| *** tonytan4ever has quit IRC | 17:29 | |
| anteaya | TimO_: you are testing operating systems? | 17:29 |
| *** dkranz has joined #openstack-infra | 17:29 | |
| fungi | TimO_: when you run that you see no events right now? | 17:29 |
| TimO_ | yes as of a few min. ago...I tried it. I will attempt again right now | 17:30 |
| clarkb | we restarted gerrit over the weekend which would kill existing connections | 17:30 |
| fungi | TimO_: also it's stream-events (there's a hyphen) | 17:31 |
| fungi | TimO_: and i'm getting events on the stream | 17:31 |
| TimO_ | my typo...i always put it in (just not this time...typing it in here :-) thks... | 17:31 |
| fungi | all old images on nodepool-dev are now deleted. working on rebuilding them now | 17:32 |
| zaro | morning | 17:33 |
| *** nels|afk has quit IRC | 17:33 | |
| TimO_ | will try again via jenkins automation | 17:33 |
| *** wuhg has quit IRC | 17:33 | |
| *** zz_avozza is now known as avozza | 17:35 | |
| anteaya | clarkb: thank you | 17:36 |
| clarkb | zaro: good morning | 17:36 |
| anteaya | morning zaro | 17:36 |
| *** Hal_ has quit IRC | 17:36 | |
| anteaya | fungi: I hope you are not waiting for us, we are a ways away | 17:36 |
| clarkb | I have put the old third party groups into "inactive" mode but not removed any members just yet | 17:36 |
| anteaya | clarkb: awesome | 17:37 |
| fungi | anteaya: nope--working on nodepool stuff | 17:37 |
| anteaya | fungi: awesome | 17:37 |
| *** jgallard has quit IRC | 17:37 | |
| anteaya | fungi: I will give you a heads up when I feel we are close enough to schedule some of your time | 17:37 |
| fungi | anteaya: thanks! | 17:37 |
| TimO_ | $ ssh -p 29418 tkonski@review.openstack.org gerrit stream-events <-- is hanging now, no events | 17:38 |
| *** yamamoto has joined #openstack-infra | 17:39 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for new patchset-created exclude fields https://review.openstack.org/140095 | 17:40 |
| clarkb | TimO_: you do need to waitfor events to happen though our gerrit should be pretty active | 17:41 |
| clarkb | TimO_: I would give it at least a few minutes | 17:41 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: fix typo, must be 'classifier' to appear on pypi.python.org https://review.openstack.org/139289 | 17:42 |
| clarkb | ok I am going to step out for a short bit to eat some breakfast | 17:43 |
| *** pcrews has joined #openstack-infra | 17:44 | |
| *** shashankhegde has quit IRC | 17:46 | |
| openstackgerrit | Qin Zhao proposed openstack/requirements: Move retrying dependency to version 1.2.3 https://review.openstack.org/140098 | 17:46 |
| *** e0ne has quit IRC | 17:46 | |
| fungi | all snapshot-based images updating on nodepool-dev now... working on dib-based next | 17:47 |
| *** SumitNaiksatam has joined #openstack-infra | 17:47 | |
| *** yamamoto has quit IRC | 17:49 | |
| *** doug-fish has joined #openstack-infra | 17:52 | |
| *** stevemar has joined #openstack-infra | 17:52 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Add the new oslo.context https://review.openstack.org/140100 | 17:53 |
| fungi | nodepool-dev diskimages updating now too | 17:54 |
| *** dannywilson has joined #openstack-infra | 17:56 | |
| *** dannywilson has quit IRC | 17:56 | |
| clarkb | dib is less critical as we dont run any non test dib jobs on them but may as well | 17:59 |
| *** nuritv has quit IRC | 17:59 | |
| lifeless | mordred: Its a tiny patch, I've just been holidaying hard | 17:59 |
| *** timrc is now known as timrc-afk | 18:00 | |
| mordred | lifeless: yah, I know you're holidaying | 18:00 |
| mordred | lifeless: as well you should | 18:00 |
| *** Ryan_Lane has joined #openstack-infra | 18:01 | |
| fungi | clarkb: 2014-12-08 17:56:21,837 INFO nodepool.image.build.devstack-precise-dib: fatal: destination path '/root/.cache/image-create/source-repositories/devstack_9383d75c34a24bf0cd2ce50da1f4b74274dfb57b.tmp' already exists and is not an empty directory. | 18:01 |
| *** achanda has quit IRC | 18:01 | |
| fungi | clarkb: devstack-trusty-dib died at the same spot | 18:01 |
| jeblair | yolanda: ^ | 18:01 |
| *** derekh has quit IRC | 18:02 | |
| fungi | clarkb: but devstack-centos7-dib is still running fine | 18:02 |
| *** ParsectiX has joined #openstack-infra | 18:02 | |
| *** achanda has joined #openstack-infra | 18:02 | |
| fungi | makes me think we have a race on some assumption that there won't be parallel builds using the same tempspace? | 18:02 |
| fungi | er, cachespace | 18:02 |
| fungi | my theory is that devstack-centos7-dib created that and so devstack-precise-dib and devstack-trusty-dib found it and failed trying to cache the same | 18:03 |
| fungi | ooh, we've had an image go ready in nodepool-dev now | 18:04 |
| *** patrickeast has joined #openstack-infra | 18:04 | |
| fungi | (77 more snapshot images to go!) | 18:04 |
| clarkb | fungi: ya that may be a race in dib caching | 18:04 |
| *** cpowell has quit IRC | 18:05 | |
| fungi | 3 images ready | 18:06 |
| *** achanda has quit IRC | 18:06 | |
| fungi | falling like dominoes now | 18:06 |
| *** armax has joined #openstack-infra | 18:06 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:06 | |
| *** amcrn has joined #openstack-infra | 18:07 | |
| clarkb | fungi: if you restart trusty/precise-dib image builds now do they fail again? | 18:07 |
| fungi | clarkb: not so far any way | 18:07 |
| clarkb | if they don't then I think it is likely that dib caching is not thread safe | 18:07 |
| fungi | the error message alone makes me think this is likely | 18:07 |
| *** melwitt has joined #openstack-infra | 18:07 | |
| clarkb | we can probably make it so that all jobs run without updating the cache then have a separate thing keep the cache up to date | 18:07 |
| clarkb | mordred: ^ | 18:08 |
| *** mattfarina has joined #openstack-infra | 18:08 | |
| fungi | perhaps concatenating the pid or some random nonce in the cache tempfile string would solve it | 18:08 |
| *** ivar-lazzaro has quit IRC | 18:08 | |
| *** MaxV has joined #openstack-infra | 18:08 | |
| clarkb | fungi: perhaps but then we will have X copies of the thing when we want 1 | 18:08 |
| fungi | as long as the cache updates are atomic per file (which i expect they are or it wouldn't be updating to a tempfile) | 18:08 |
| *** ivar-lazzaro has joined #openstack-infra | 18:09 | |
| * mordred reads | 18:09 | |
| *** Parsecti_ has joined #openstack-infra | 18:10 | |
| clarkb | ianw: when you wake up https://review.openstack.org/#/c/138250/ has some comments from me. Would like to see that move forward so let me know what you think | 18:10 |
| *** ParsectiX has quit IRC | 18:10 | |
| mordred | SpamapS: ^^ | 18:10 |
| mordred | clarkb: I'll dig in to it | 18:10 |
| *** Masahiro has joined #openstack-infra | 18:11 | |
| fungi | also, when nodepool diskimage-build fails, it ends with a message of "image blah is ready" which is sorta misleading | 18:11 |
| *** jistr has quit IRC | 18:11 | |
| SpamapS | I believe we only ever do one dib build per VM right now, so it's possible the cache has grown some non-atomic bits | 18:11 |
| fungi | as it comes right after the message about "image blah deleted" | 18:11 |
| SpamapS | We'd certainly want it to stay coherent through parallel builds. | 18:12 |
| *** jp_at_hp has quit IRC | 18:12 | |
| SpamapS | source-repositories is certainly the most complex part of the cache. | 18:12 |
| *** emagana has joined #openstack-infra | 18:13 | |
| *** cpowell has joined #openstack-infra | 18:13 | |
| SpamapS | ah yeah that looks entirely non parallelizable. | 18:14 |
| SpamapS | https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/source-repositories/extra-data.d/98-source-repositories#n87 | 18:14 |
| *** Masahiro has quit IRC | 18:15 | |
| *** otter768 has joined #openstack-infra | 18:15 | |
| SpamapS | We just go ahead and skip the git command if the dir exists | 18:15 |
| *** jlibosva has quit IRC | 18:16 | |
| SpamapS | so then the second encounter leads to assuming this empty dir is the git dir. | 18:16 |
| clarkb | oh is this fixed but not in mordred fork? | 18:16 |
| SpamapS | no | 18:16 |
| SpamapS | this is broken period | 18:16 |
| SpamapS | https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/source-repositories/extra-data.d/98-source-repositories#n112 | 18:16 |
| *** _nadya_ has quit IRC | 18:16 | |
| SpamapS | further down | 18:16 |
| *** tonytan4ever has joined #openstack-infra | 18:17 | |
| SpamapS | so we probably need to use an flock to make the second one wait | 18:17 |
| *** mikedillion has quit IRC | 18:19 | |
| *** otter768 has quit IRC | 18:20 | |
| *** harlowja_away is now known as harlowja | 18:20 | |
| *** mikedillion has joined #openstack-infra | 18:20 | |
| fungi | opened https://storyboard.openstack.org/#!/story/2000067 for the misleading failure message "image foo is built" | 18:20 |
| clarkb | fungi: thank you, I will take a look at that now | 18:21 |
| clarkb | krotscheck: is there a reason I typically have to refresh opening a bug like ^ to see the contents? | 18:21 |
| TimO_ | My ssh -p 29418 timed out with this error: "ssh_exchange_identification: read: Connection timed out" | 18:22 |
| TimO_ | any ideas here? thks... | 18:22 |
| *** mikedillion has quit IRC | 18:22 | |
| fungi | i also think we should add a convenience redirect for https://storyboard.openstack.org/([0-9]+)$ to https://storyboard.openstack.org/#!/story/$1 | 18:22 |
| *** nfedotov has joined #openstack-infra | 18:22 | |
| * SpamapS almost done w/ flock fix | 18:22 | |
| *** shashankhegde has joined #openstack-infra | 18:22 | |
| *** e0ne has joined #openstack-infra | 18:23 | |
| fungi | TimO_: perhaps your firewall administrator went rogue and blocked egress for that? | 18:23 |
| fungi | SpamapS: i have a similar parallelism-related issue i think on a subsequent retry | 18:23 |
| *** Parsecti_ has quit IRC | 18:23 | |
| *** skolekonov has quit IRC | 18:23 | |
| fungi | SpamapS: looking over the log now to see if i can figure out where it went off the rails | 18:23 |
| fungi | SpamapS: "Did not get /opt/dib_cache/source-repositories/repositories_flock" | 18:24 |
| *** tonytan4ever has quit IRC | 18:25 | |
| SpamapS | oh there may be an flock at a higher level | 18:25 |
| TimO_ | possible! thks will check out here | 18:26 |
| SpamapS | doh forgot to look at that | 18:26 |
| fungi | SpamapS: yeah, i had two dib runs die on that one this most recent pass, with a third still running | 18:26 |
| SpamapS | oh well thats dumb | 18:26 |
| fungi | i'm going to retry them both for a third time | 18:26 |
| SpamapS | they take exclusive on all runs | 18:26 |
| SpamapS | looks like I need to have a chat about locking w/ our review team. :) | 18:27 |
| SpamapS | fungi: ok so the problem is, dib tries to serialize _all_ dib runs basically | 18:28 |
| *** mriedem has quit IRC | 18:28 | |
| fungi | SpamapS: wonderful! and we don't... | 18:28 |
| fungi | so, er, yeah failcity | 18:28 |
| jeblair | fungi, mordred: we may want to have manage-projects make one of these: https://github.com/golang/go/pull/9220 | 18:28 |
| jeblair | fungi, mordred: https://github.com/golang/go/pulls | 18:29 |
| SpamapS | what should be happening is "if you're populating a cache, exclusive, otherwise, shared until populating is done" | 18:29 |
| SpamapS | but because it's a massive coarse grained lock.. | 18:29 |
| SpamapS | it's just a massive serializer | 18:29 |
| fungi | jeblair: oh, neat. a single locked pull request saying we don't take pull requests is a great idea | 18:29 |
| SpamapS | fungi: ok so basically all clones have to finish in 10 minutes or the flock fails.. seems like what might be happening | 18:30 |
| jeblair | will file story | 18:30 |
| *** amitgandhinz has joined #openstack-infra | 18:30 | |
| jeblair | krotscheck: i thought storyboard copied the title into the initial task? that does not seem to be happening | 18:31 |
| jeblair | (i seem to have to type it twice) | 18:31 |
| fungi | i noticed the same when i opened the cosmetic logging story for nodepool just now | 18:31 |
| krotscheck | jeblair: Huhn. It used to. | 18:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Fix error handling when dib image builds fail https://review.openstack.org/140106 | 18:31 |
| clarkb | fungi: ^ | 18:32 |
| *** shashankhegde has quit IRC | 18:32 | |
| jeblair | https://storyboard.openstack.org/#!/story/2000068 filed | 18:32 |
| fungi | clarkb: Story: #2000067 in the commit message? i'll go ahead and update the status | 18:33 |
| clarkb | fungi: ya I need to figure out what the proper string is or that | 18:33 |
| clarkb | is it still closes-bug? | 18:33 |
| *** mrunge has quit IRC | 18:33 | |
| *** Ryan_Lane has quit IRC | 18:33 | |
| *** andreykurilin_ has joined #openstack-infra | 18:33 | |
| fungi | oh, maybe. at one point we were doing Story: for the stuff in storyboard, but maybe we normalize to make it the same as for lp since we're importing anyway? | 18:33 |
| fungi | there are likely downsides to either approach | 18:34 |
| jeblair | it's premature to fix that | 18:34 |
| jeblair | we need to determine how to address a task first | 18:34 |
| clarkb | git grep -i story doesn't show much in jeepyb so must still be closes-bug | 18:34 |
| jeblair | until then, it's basically just informative | 18:34 |
| jeblair | there's no automation around it anyway | 18:35 |
| fungi | right | 18:35 |
| clarkb | jeblair: fungi: in that case do people have a preference for what I use in the commit? | 18:35 |
| jeblair | i would stick with "Story:" for now | 18:35 |
| clarkb | kk | 18:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Fix error handling when dib image builds fail https://review.openstack.org/140106 | 18:35 |
| *** Ryan_Lane has joined #openstack-infra | 18:36 | |
| *** tonytan4ever has joined #openstack-infra | 18:36 | |
| armax | anteaya: ping | 18:37 |
| anteaya | hi there armax | 18:38 |
| *** sputnik13 has quit IRC | 18:38 | |
| *** avozza is now known as zz_avozza | 18:39 | |
| anteaya | if a job has has a provision in zuul/layout.yaml to not test stable branches, do we also need to specify icehouse and juno? | 18:39 |
| fungi | clarkb: in act, i just noticed the issue you're addressing in 140106 is significantly worse than i thought. they're being logged as built because nodepool is adding them to the database in a ready state according to nodepool dib-image-list | 18:39 |
| anteaya | for instance, http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n774 should I request that juno is specified? | 18:39 |
| clarkb | fungi: yup | 18:39 |
| *** pelix has quit IRC | 18:39 | |
| fungi | clarkb: yikes. glad we weren't relying on this yet :/ | 18:39 |
| *** yfried_ has joined #openstack-infra | 18:40 | |
| *** eharney has quit IRC | 18:40 | |
| *** eharney has joined #openstack-infra | 18:40 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: First task tracks story title. https://review.openstack.org/140108 | 18:42 |
| krotscheck | jeblair: ^^ | 18:42 |
| SpamapS | fungi: can you open a bug report against dib with the exact error you're getting? | 18:42 |
| SpamapS | fungi: it will help with the review process. :) | 18:42 |
| fungi | SpamapS: sure | 18:42 |
| SpamapS | fungi: ty | 18:42 |
| *** e0ne has quit IRC | 18:43 | |
| fungi | clarkb: jeblair: all but 6 snapshot images are completed now, the stragglers are retries (mostly in rax) because of instance boot/ssh connectivity failures | 18:43 |
| jeblair | krotscheck: opened based on something fungi said a moment ago: https://storyboard.openstack.org/#!/story/2000069 | 18:43 |
| *** SumitNaiksatam has quit IRC | 18:43 | |
| jeblair | krotscheck: and thx | 18:43 |
| *** sputnik13 has joined #openstack-infra | 18:44 | |
| *** shashankhegde has joined #openstack-infra | 18:44 | |
| clarkb | is there a plan for addressing tasks yet? maybe something similar to gerrit patchsets? | 18:45 |
| clarkb | eg 123456,3 | 18:45 |
| jeblair | clarkb: the thing we run into is that they can be removed and reordered | 18:46 |
| clarkb | though that may imply order which isn't the same as reality in the task list | 18:46 |
| clarkb | ya that | 18:46 |
| clarkb | UUID all the things? | 18:46 |
| anteaya | mtreinish: can you look at this? this editing job runs non-voting on tempest: https://review.openstack.org/#/c/139822/ | 18:46 |
| jeblair | otoh, 123456,0f24beb72b340628a564ce8d755341eca70a4c40 is not very friendly :) | 18:46 |
| jeblair | but maybe would actually be okay... | 18:47 |
| clarkb | jeblair: and if the uuid is actually unique we can drop the story too | 18:47 |
| clarkb | jeblair: and directly address the task | 18:47 |
| *** sdake has quit IRC | 18:47 | |
| anteaya | mtreinish: edited job | 18:47 |
| *** sdake has joined #openstack-infra | 18:47 | |
| *** sdake has quit IRC | 18:47 | |
| *** sdake has joined #openstack-infra | 18:47 | |
| *** teran has quit IRC | 18:47 | |
| jeblair | maybe you refer to stories by friendly number, and tasks by scary uuid, but in a commit message maybe that's fine, and yeah, it could just be the uuid which implies the story so is sufficient for a link. | 18:47 |
| SpamapS | ok so this is kind of interesting | 18:48 |
| fungi | SpamapS: bug 1400450 | 18:49 |
| uvirtbot | Launchpad bug 1400450 in diskimage-builder "source-repositories caching is not concurrency-safe" [Undecided,New] https://launchpad.net/bugs/1400450 | 18:49 |
| *** e0ne has joined #openstack-infra | 18:49 | |
| fungi | clarkb: jeblair: perhaps allow task uuids to be abbreviated to arbitrary length like git does with shas? | 18:50 |
| SpamapS | we have a cache per cloned URL, but it manipulates HEAD ... so we have to hold the exclusive lock through the whole process. | 18:50 |
| *** yamamoto has joined #openstack-infra | 18:50 | |
| *** sputnik13 has quit IRC | 18:50 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for new patchset-created exclude fields https://review.openstack.org/140095 | 18:50 |
| jeblair | krotscheck: have we had a conversation about when i type "jeepyb<enter>" into the search box and end up with "jee" as the search string? because i apparently type faster than a computer can read. (?) | 18:50 |
| *** sputnik13 has joined #openstack-infra | 18:51 | |
| SpamapS | until we do the second local clone to the REPO_DEST | 18:51 |
| krotscheck | jeblair: WE have not. | 18:51 |
| *** MaxV has quit IRC | 18:51 | |
| jeblair | clarkb, fungi, anteaya: added [LHF] to title of this story https://storyboard.openstack.org/#!/story/2000068 to indicate 'low hanging fruit' since we don't have tags | 18:51 |
| jeblair | krotscheck: is that a known thing? file story or no? | 18:52 |
| krotscheck | jeblair: That’s a usability discussion. | 18:52 |
| fungi | jeblair: like typing on an acoustically coupled 110-baud modem and watching the echo catch up with your keystrokes? | 18:52 |
| *** vigneshvar has joined #openstack-infra | 18:52 | |
| jeblair | krotscheck: are you saying that it is not clearly a bug? | 18:52 |
| krotscheck | Yep. it is not clearly a bug, because it’s a combination of your typing speed, the speed at which the criteria are resolved, and the fact that you still have to hit the down arrow to select one of the returned criteria to be able to navigate to it. | 18:53 |
| krotscheck | or use it. | 18:53 |
| anteaya | jeblair: thank you | 18:54 |
| anteaya | and a nice [LHF] story it is | 18:54 |
| *** rediskin has left #openstack-infra | 18:54 | |
| jeblair | krotscheck: i don't have to hit the down arrow. if i type "jeepyb<enter>" very slowly it searches for jeepyb. if i type "jeepyb<enter>" at the speed at which a first year typing student might type it, i get "jee". | 18:55 |
| *** yamamoto has quit IRC | 18:55 | |
| jeblair | krotscheck: i think there's only one variable there. | 18:55 |
| krotscheck | Oh, different issue. | 18:55 |
| krotscheck | Your bug seemed to indicate the latter. | 18:55 |
| krotscheck | Sorry | 18:55 |
| anteaya | fungi: status update on force push, we have -2'd any patch we fill might be close to merging and creating havoc durng the split, dougwig has fresh repos created and has shas for all three repos, mestery has reviewed the neutron patch to remove the split out code from neutron/ we are waiting on carl_baldwin for his additional review on the netron code removal patch before we move aheahd | 18:56 |
| jeblair | ok, i'll file a story then | 18:56 |
| openstackgerrit | Steven Weston proposed openstack-infra/system-config: Adds puppet system-config module split script https://review.openstack.org/137991 | 18:56 |
| dougwig | repo info: | 18:56 |
| dougwig | https://www.irccloud.com/pastebin/jgjW89ci | 18:57 |
| anteaya | fungi: -dev and -ops mls have been notified of the split and being told to expect new repos won't work until next week at the earliest | 18:57 |
| sweston | asselin: ^ there you go | 18:57 |
| anteaya | anything we have left out? | 18:57 |
| krotscheck | jeblair: How many milliseconds do you have between keystrokes? | 18:57 |
| * krotscheck can’t believe he just asked that question | 18:57 | |
| * jeblair can't believe he doesn't know off the top of his head ;) | 18:58 | |
| asselin | sweston, got it | 18:58 |
| *** jedimike has quit IRC | 18:58 | |
| sweston | asselin: k :-) | 18:58 |
| *** dannywilson has joined #openstack-infra | 18:58 | |
| * jeblair is writing a program to measuer the speed he types 'jeepyb<enter>' | 18:58 | |
| SpamapS | fungi: ok so the one about .tmp existing and not being empty is actually very weird and should not happen given the flocking in use. | 18:59 |
| *** emagana has quit IRC | 19:00 | |
| SpamapS | fungi: the other one makes all fetching of all remote things a serial operation.. and if any of them takes longer than 10 minutes.. all waiting builds fail. | 19:00 |
| dougwig | https://www.irccloud.com/pastebin/iulDoyFE | 19:00 |
| fungi | SpamapS: yeah, either there's a race engaging the flock or that's happening outside the flock? | 19:00 |
| *** davideagnello has joined #openstack-infra | 19:00 | |
| *** emagana has joined #openstack-infra | 19:00 | |
| SpamapS | fungi: https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/source-repositories/extra-data.d/98-source-repositories#n243 | 19:01 |
| *** armax has quit IRC | 19:01 | |
| SpamapS | fungi: basically nothing in that script happens outside the flock except making the cache base if it doesn't exist, and some manifest stuff. | 19:01 |
| SpamapS | I wonder... | 19:01 |
| fungi | anteaya: dougwig: thanks for the update | 19:01 |
| SpamapS | maybe something is already using fd 9 | 19:01 |
| SpamapS | and closes it later on | 19:02 |
| clarkb | fungi: anteaya dougwig not sure what the context of those shas is | 19:02 |
| *** dannywil_ has joined #openstack-infra | 19:02 | |
| *** dannywilson has quit IRC | 19:02 | |
| fungi | clarkb: i'm assuming they're related to whatever carl_baldwin (not in channel?) is supposed to be checking | 19:03 |
| *** emagana_ has joined #openstack-infra | 19:03 | |
| fungi | SpamapS: could there be two flocks fighting over one file descriptor? | 19:03 |
| jeblair | krotscheck: i think i was in the range of 180-220 ms per keystroke at the time. i am now, for some reason, much faster at typing 'jeepyb' (down to 120!) | 19:04 |
| anteaya | sorry the shas are the directories that are sitting in github which fungi will soon force push to the newly created (empty) neutron repos splitting out advanced services | 19:04 |
| krotscheck | jeblair: :-P | 19:04 |
| clarkb | anteaya: I am not sure why the shas are relevant htough | 19:04 |
| clarkb | anteaya: we push everything into the repos | 19:04 |
| *** emagana has quit IRC | 19:04 | |
| anteaya | right | 19:04 |
| *** doude_ has quit IRC | 19:04 | |
| anteaya | being overly cautious | 19:05 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of Ask website https://review.openstack.org/140043 | 19:05 |
| ashp | hey I just wanted to swing by and say a huge thank you to the amazingly helpful people of this channel | 19:05 |
| ashp | my migration was (eventually) a success and everyone is happy | 19:05 |
| krotscheck | jeblair: So the issue I think is that the typebox doesn’t kill outstanding criteria searches as you’re typing, which quickly exhausts the browser’s query pool. So the solution will require adding both a ms delay to the typeahead resolution (to reduce the number of queries) and to kill pending queries. | 19:05 |
| fungi | ashp: you got it all working? congrats! | 19:05 |
| krotscheck | I can do one very quickly. | 19:05 |
| clarkb | ashp: woot | 19:05 |
| krotscheck | The other will take tiem. | 19:06 |
| ashp | Now we can actually have serious conversations about what to do next and where to go from here :) | 19:06 |
| ashp | Now we're not stuck on gerrit 2.4 forever | 19:06 |
| fungi | waiting for one last snapshot update to finish, and then they're all done building | 19:06 |
| clarkb | fungi: yay | 19:06 |
| *** timrc-afk is now known as timrc | 19:06 | |
| mordred | jeblair: yes on manage-project pull request issue thing | 19:07 |
| jeblair | krotscheck: https://storyboard.openstack.org/#!/story/2000070 | 19:07 |
| *** dtantsur is now known as dtantsur|afk | 19:07 | |
| SpamapS | fungi: seems unlikely.. bash will wait for processes that it forks and execs, and I'd assume the sub-shell opening 9 will be waited on as well. | 19:07 |
| SpamapS | fungi: what _MIGHT_ be possible is flock gets killed or dies. | 19:08 |
| *** sputnik13 has quit IRC | 19:08 | |
| SpamapS | actually no | 19:09 |
| SpamapS | it isn't left running, the process has the flock | 19:09 |
| *** sputnik13 has joined #openstack-infra | 19:09 | |
| * SpamapS will continue to investigate a bit later | 19:10 | |
| fungi | clarkb: also dib images are done building and in the process of being uploaded | 19:10 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Added millisecond delay to reduce the number of queries. https://review.openstack.org/140120 | 19:10 |
| krotscheck | jeblair: ^^ | 19:10 |
| anteaya | fungi: carl_baldwin is +2 on https://review.openstack.org/#/c/139901/, we are ready for you to do the force push, mestery is standing by to +A 139901 once we have confirmed the last commits on all three repos | 19:10 |
| anteaya | fungi: do you have what you need to force push | 19:11 |
| clarkb | fungi: you going to do it then? remember to do what jeepyb does if we need to grab all the branches and tags and things | 19:11 |
| *** cnesa has quit IRC | 19:11 | |
| *** dprince_ has quit IRC | 19:11 | |
| fungi | clarkb: yep | 19:11 |
| fungi | anteaya: this should be sufficient yes. looking over it now | 19:11 |
| clarkb | I think swapping it out directly on disk will require a reindex which is less desireable | 19:11 |
| mestery | thanks fungi and anteaya | 19:12 |
| anteaya | fungi: great, let me know if you need anything more | 19:12 |
| fungi | clarkb: i'm going to push --force in as a project bootstrapper | 19:12 |
| dougwig | no branches or tags to worry about. | 19:12 |
| clarkb | fungi: sounds good | 19:12 |
| clarkb | dougwig: there should be though | 19:12 |
| *** SumitNaiksatam has joined #openstack-infra | 19:12 | |
| clarkb | dougwig: you should have stable/* and release tags | 19:12 |
| mordred | clarkb, jeblair: I want to put fully fixing rax-dib on hold for a week. I'll keep poking - but they're rolling out the config-drive fix for us | 19:12 |
| *** sputnik13 has quit IRC | 19:12 | |
| fungi | looking now to see if i need to care about additional tags and non-master branches | 19:12 |
| mordred | and I'd prefer to base what we're doing on that, rather than on nova-agent | 19:12 |
| *** sputnik13 has joined #openstack-infra | 19:12 | |
| clarkb | mordred: thats fine, we have a lot of other dib things to fix first | 19:12 |
| mordred | so it seems like finalizing the current code is less useful | 19:12 |
| anteaya | fungi clarkb do say if we have more work do to here | 19:13 |
| clarkb | anteaya: well I expect branches and tags | 19:13 |
| anteaya | don't force push until you are happy | 19:13 |
| mordred | yah. I can hack on the parallel thing | 19:13 |
| *** mwagner_lap has quit IRC | 19:13 | |
| clarkb | because you are taking a repo with branches and tags and splitting some files out of that repo. Splitting should preserve those branches and tags | 19:13 |
| *** pc_m_ has quit IRC | 19:13 | |
| mestery | clarkb: In our case, we're not interested in preserving the branches/tags in the new repos, they will still be there in the old ones. | 19:14 |
| *** pc_m has joined #openstack-infra | 19:14 | |
| fungi | clarkb: looks like no tags, and no branches besides master | 19:14 |
| fungi | just double-checked all three | 19:14 |
| clarkb | mestery: why wouldn't you be interested? that is histor... | 19:14 |
| *** cnesa has joined #openstack-infra | 19:14 | |
| fungi | clarkb: well, the branches are likely irrelevant if this is only being integrated into kilo neutron | 19:15 |
| mestery | fungi: ++ | 19:15 |
| clarkb | fungi: its relevant as history | 19:15 |
| clarkb | fungi: git diff master stable/foo | 19:15 |
| clarkb | for example | 19:15 |
| fungi | clarkb: relevant until we delete those branches? | 19:15 |
| clarkb | fungi: they will be relaced with tags so still relevant | 19:15 |
| fungi | clarkb: though i guess we do tag the eventual branch state | 19:15 |
| fungi | yeah | 19:15 |
| fungi | clarkb: does git filter-branch handle that though? | 19:16 |
| fungi | i've not done a filter-branch on multiple branches and preserved tags before | 19:16 |
| clarkb | fungi: I am pretty sure it can | 19:16 |
| dougwig | fyi, this is what created those repos: https://github.com/dougwig/split-repo/blob/master/service-split.sh | 19:17 |
| mtreinish | anteaya: makes sense to me, although I think that's still just a neutron job at this point | 19:17 |
| clarkb | fungi: -- --all passed to filter branch apparently does branches and tags | 19:18 |
| fungi | okay, all snapshot images and diskimages on nodepool-dev are refreshed successfully. we should be set to press forward om that front | 19:19 |
| clarkb | it proably increases time to run but I know one of the things that frustrates folks like sdague is when we make large moves like this and don't preserve enough history to do repo operations against important points in time | 19:19 |
| *** mwagner_lap has joined #openstack-infra | 19:19 | |
| dougwig | just so we're all on the same page, neutron itself is *not* getting its history pruned. | 19:20 |
| clarkb | dougwig: understood | 19:20 |
| clarkb | dougwig: but if I have nuetron-foo cloned and not neutron and am on an airplane I won't have any of that info | 19:20 |
| clarkb | dougwig: and doing diffs across repos is harder than within a repo | 19:20 |
| fungi | dougwig: yeah, but clarkb's right that it's useful to see what state of these files were covered in what branches/tags prior to split without having to dig that out of the neutron repo | 19:21 |
| dougwig | can someone tell me exactly how to tweak this? | 19:23 |
| dougwig | https://www.irccloud.com/pastebin/wYFbPlWV | 19:23 |
| *** marun has quit IRC | 19:23 | |
| anteaya | mtreinish: great thank you | 19:23 |
| clarkb | dougwig: I believe you remove HEAD and replace it with '-- --all' | 19:23 |
| dougwig | ok, let me run that right now. | 19:24 |
| *** zul has quit IRC | 19:25 | |
| dougwig | running. used to take 21 minutes. i'll let you know. | 19:25 |
| *** sarob has quit IRC | 19:25 | |
| *** Ryan_Lane has quit IRC | 19:25 | |
| anteaya | thank you | 19:25 |
| *** esker has quit IRC | 19:26 | |
| *** dprince has joined #openstack-infra | 19:27 | |
| clarkb | jeblair: fungi: I am ready to start working on nodepool switch whenever you guys are | 19:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Log ipv6 addresses for nodes https://review.openstack.org/132307 | 19:28 |
| jeblair | clarkb: ready | 19:30 |
| fungi | ready | 19:30 |
| clarkb | jeblair: do we want an etherpad? | 19:30 |
| jeblair | clarkb: nah, we only have a few steps | 19:30 |
| *** vigneshvar has quit IRC | 19:31 | |
| clarkb | and we are going with: stop zuul, stop old nodepool, delete all old nodes, start new nodepool, start zuul ? | 19:31 |
| jeblair | if we go with fungi's low-impact way, we need to wait at least 1 hour, possibly more, and would need to watch it the whole time | 19:32 |
| fungi | do we want to stop old nodepool first and get ready/building node deletion underway? | 19:32 |
| *** liusheng has quit IRC | 19:32 | |
| *** liusheng has joined #openstack-infra | 19:32 | |
| clarkb | fungi: ya I Think first delete pass should be ready/building | 19:32 |
| fungi | and then delete used nodes on the old nodepool in parallel with new nodepool running? | 19:32 |
| clarkb | that at least gives us the option to start new nodepool earlier | 19:32 |
| clarkb | jeblair: ya I am not a fan of being on zuuls schedule | 19:33 |
| jeblair | and also stop zuul later | 19:33 |
| *** zz_jgrimm has joined #openstack-infra | 19:33 | |
| *** maurosr has joined #openstack-infra | 19:33 | |
| fungi | a similar alternative might be to quiesce jenkins masters instead? | 19:33 |
| jeblair | and even though we throw out those results, zuul continues queueing so we don't miss events | 19:33 |
| clarkb | I can stop old nodepool now and kick off the ready/building deletes | 19:33 |
| *** andreykurilin_ has quit IRC | 19:33 | |
| jeblair | fungi: if we do that, new nodepool will not attach new nodes to them | 19:33 |
| fungi | fair point | 19:33 |
| *** fandi has joined #openstack-infra | 19:34 | |
| *** amcrn has quit IRC | 19:34 | |
| fungi | though we could do that just long enough to delete the used nodes, then start them again right after a zuul restart | 19:34 |
| fungi | probably at least as messy in the end | 19:34 |
| *** ociuhandu has quit IRC | 19:35 | |
| *** hdd has joined #openstack-infra | 19:35 | |
| jeblair | so i think current plan is: stop old nodepool, delete ready/building, stop zuul, start new nodepool, delete all old nodes, start zuul | 19:35 |
| fungi | that works and minimizes zuul downtime without too much complexity | 19:35 |
| clarkb | jeblair: sounds good. I think we don't have to wait for al old nodes to be deleted before we start zuul. we will just hit quota issues in the worst case | 19:35 |
| clarkb | but I like that plan | 19:35 |
| clarkb | should I stop old nodepool and start deleting things? | 19:36 |
| fungi | i say go for it | 19:36 |
| jeblair | just a sec | 19:36 |
| clarkb | k | 19:36 |
| *** cpowell has quit IRC | 19:36 | |
| * nibalizer having deja vu :P | 19:36 | |
| jeblair | stop old nodepool, delete ready/building, stop zuul, start new nodepool, wait for representative set of nodes, start zuul, delete all old nodes | 19:36 |
| jeblair | clarkb, fungi: revised ^ ? | 19:36 |
| fungi | jeblair: boh, right. otherwise the dreaded not_registered | 19:37 |
| clarkb | huge +2 :) | 19:37 |
| openstackgerrit | Merged openstack-infra/project-config: Make Neutron DVR job run on the full tempest suite https://review.openstack.org/139822 | 19:37 |
| jeblair | okay, sounds like we're good; clarkb: ready for you to proceed | 19:38 |
| jeblair | i'm standing by on zuul | 19:38 |
| clarkb | stopping old nodepool now | 19:38 |
| *** shakamunyi has joined #openstack-infra | 19:38 | |
| *** zul has joined #openstack-infra | 19:38 | |
| jeblair | also, we're currently at approx 1/2 capacity, so not the worst time to do this :) | 19:39 |
| mrmartin | fungi, clarkb: so the askbot.o.o almost passing the gate now, have a single problem only: I needed to add puppetlabs-concat, and it have a dependency on puppetlabs-stdlibs, but we are pinning stdlibs to 4.3.2 and puppetlabs-concat deploys a 4.4.0 as a dependency (works well anyway with 4.3.2) too. Any idea how to enforce this concat module to install after 4.3.2 stdlibs? :) http://logs.openstack.org/43/140043/3/check/gate-infra-pup | 19:39 |
| mrmartin | pet-apply-precise/e9cd353/console.html | 19:39 |
| clarkb | its stopped I am going to start deletions now | 19:39 |
| clarkb | starting with building so that the ready nodes have a chance to change state | 19:39 |
| fungi | mrmartin: i thought we had dealt with this once already... nibalizer: ^ ? | 19:39 |
| *** krtaylor has quit IRC | 19:40 | |
| mrmartin | this is the full url for wrong puppet module order: http://logs.openstack.org/43/140043/3/check/gate-infra-puppet-apply-precise/e9cd353/console.html | 19:40 |
| *** baoli has quit IRC | 19:40 | |
| *** amcrn has joined #openstack-infra | 19:40 | |
| jeblair | SlickNik: gate-trove-functional-dsvm-mysql seems like a very slow job | 19:40 |
| *** zz_jgrimm is now known as jgrimm | 19:40 | |
| *** cpowell has joined #openstack-infra | 19:41 | |
| pleia2 | oof, it's very clear to me now that this zanata "ansible to puppet translation" really is "write puppet stuff, use RedHat's ansible playbooks as a reference" - so much Red Hat and JBoss (we're using Widlfly) specific stuff in here | 19:42 |
| *** weshay has quit IRC | 19:42 | |
| pleia2 | fortunately there's an open source wildfly puppet module I'm testing now, so that helps | 19:42 |
| *** vigneshvar has joined #openstack-infra | 19:42 | |
| *** baoli has joined #openstack-infra | 19:43 | |
| *** cpowell_ has joined #openstack-infra | 19:43 | |
| *** KurtMartin has joined #openstack-infra | 19:43 | |
| nibalizer | fungi: mrmartin hrm | 19:43 |
| anteaya | pleia2: :( | 19:44 |
| *** jcoufal has quit IRC | 19:44 | |
| nibalizer | so i dont think concat enforces specific stdlib versions | 19:44 |
| *** cdent has quit IRC | 19:44 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/devstack-gate: Introduce support for running IPv6 networking in Neutron https://review.openstack.org/140128 | 19:44 |
| nibalizer | "name": "puppetlabs/stdlib", | 19:45 |
| nibalizer | "version_requirement": ">= 3.2.0 < 5.0.0" | 19:45 |
| mrmartin | nibalizer: seems to be that real dependency of modules not enforced here, and add to MODULES array keep the modules in ascending order by their names. | 19:45 |
| *** Rockyg has joined #openstack-infra | 19:45 | |
| pleia2 | anteaya: it's ok, I kind of expected this, these were wrestled from internal RedHat, so they are bound to be a bit messy :) | 19:45 |
| *** emagana_ has quit IRC | 19:45 | |
| pleia2 | only really written for internal consumption | 19:45 |
| mrmartin | nibalizer, you are right it is not, but puppetlabs-concat < puppetlabs-stdlibs, and puppetlabs-concat tries to deploy his dependency | 19:45 |
| mrmartin | so it will collide with a specific stdlibs version. | 19:46 |
| *** emagana has joined #openstack-infra | 19:46 | |
| *** cpowell has quit IRC | 19:46 | |
| *** kmartin has quit IRC | 19:47 | |
| nibalizer | uh | 19:47 |
| nibalizer | so concat and stdlib are both being pulled in by other modules | 19:48 |
| *** _nadya_ has joined #openstack-infra | 19:48 | |
| nibalizer | i think | 19:48 |
| nibalizer | probably postgresql is pulling in concat | 19:48 |
| anteaya | pleia2: glad this isn't a surprise for you then | 19:48 |
| clarkb | deletions seem to have slowed down | 19:48 |
| mrmartin | nibilizer: I specified concat there, do you think it is pulled by other modules also? https://review.openstack.org/#/c/140043/2/modules.env | 19:49 |
| mrmartin | oh nice, I checking that. | 19:49 |
| *** sputnik13 has quit IRC | 19:49 | |
| nibalizer | can you link the exact error that you're seeing in the console.html from above? | 19:49 |
| clarkb | looks like we may be waiting on clouds to tell us they have deleted things | 19:50 |
| mrmartin | 2014-12-08 19:11:12.011 | [1;31mError: Could not install module 'puppetlabs-stdlib' (v4.3.2) | 19:50 |
| mrmartin | 2014-12-08 19:11:12.011 | Module 'puppetlabs-stdlib' (v4.4.0) is already installed | 19:50 |
| *** emagana has quit IRC | 19:50 | |
| mrmartin | this is the dependency line of pgsql: {"name":"puppetlabs/concat","version_requirement":">= 1.1.0 <2.0.0"} | 19:50 |
| *** Ryan_Lane has joined #openstack-infra | 19:50 | |
| dougwig | anteaya, fungi: fresh repos: | 19:51 |
| dougwig | https://www.irccloud.com/pastebin/TIPg0nEw | 19:51 |
| fungi | dougwig: thanks--checking over them now | 19:51 |
| mrmartin | nibalizer: so if I remove the concat definition from modules.env, it must work, because pgsql pulls in the puppetlabs-concat | 19:52 |
| *** sputnik13 has joined #openstack-infra | 19:52 | |
| *** sputnik13 has quit IRC | 19:52 | |
| nibalizer | ya | 19:52 |
| nibalizer | well 'it must work' is somethig thats hard to agree to | 19:52 |
| *** julim has quit IRC | 19:53 | |
| nibalizer | but if your asking if concat is a transitivie dependency that you can depend on then yes, it will be there | 19:53 |
| *** tkelsey has joined #openstack-infra | 19:53 | |
| fungi | dougwig: the url for x-neutron-lbaas 404s for me, and the other two seems to still have no branches or tags. did you push all up to github or just your worktree's current active branch? | 19:53 |
| dougwig | yep, messed up the lbaas push. | 19:54 |
| mrmartin | nibalizer: this will work until we are not removing the puppetlabs-postgresql module :) | 19:54 |
| *** radez is now known as radez_g0n3 | 19:54 | |
| dougwig | sec, i think i pushed wrong. standby. | 19:54 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of Ask website https://review.openstack.org/140043 | 19:55 |
| *** armax has joined #openstack-infra | 19:55 | |
| clarkb | jeblair: fungi according to nodepool list the ready nodes are gone but my delete command has not returned | 19:55 |
| clarkb | jeblair: still waiting on ~20 building nodes to delete | 19:55 |
| *** krtaylor has joined #openstack-infra | 19:55 | |
| mrmartin | so the conclusion here, it is really a good idea to move the askbot and solr modules into puppetforge, where they can precisely define their module dependencies. | 19:55 |
| jeblair | clarkb: think it's run down enough to make it worth continuing? | 19:56 |
| *** radez_g0n3 is now known as radez | 19:57 | |
| clarkb | jeblair: ya I think the ready nodes were the most problematic and with them gone we should be safe to stop zuul and start new nodepool | 19:57 |
| *** weshay has joined #openstack-infra | 19:57 | |
| jeblair | who's standing buy to start new nodepool? | 19:58 |
| jeblair | standing by | 19:58 |
| clarkb | I can start it | 19:58 |
| jeblair | ack. i will stop zuul now... | 19:58 |
| jeblair | er, hold a min | 19:58 |
| clarkb | k | 19:58 |
| nibalizer | mrmartin: +1 | 19:59 |
| *** cpowell_ has quit IRC | 19:59 | |
| jeblair | change at top may be about to report... checking on it | 19:59 |
| *** cpowell has joined #openstack-infra | 19:59 | |
| *** Masahiro has joined #openstack-infra | 19:59 | |
| *** amitgandhinz has quit IRC | 19:59 | |
| jeblair | maybe not. proceeding. | 20:00 |
| *** amitgandhinz has joined #openstack-infra | 20:00 | |
| dougwig | i think i need to pull all the remote branches local before running the split, else they don't get pulled along. standby. | 20:00 |
| jeblair | clarkb: zuul is stopped, ready for you to proceed | 20:00 |
| clarkb | jeblair: starting new nodepool now | 20:00 |
| clarkb | jeblair: do you have out of band geard running? | 20:00 |
| jeblair | clarkb: is now | 20:00 |
| clarkb | new nodepool is starting | 20:01 |
| clarkb | it just did allocation calculations /me checks the building list | 20:01 |
| *** baoli has quit IRC | 20:01 | |
| clarkb | looks like it is building all the things | 20:02 |
| mtreinish | mordred: if you get a sec can you take a look at the mysql-proxy patch, if the approach doesn't make sense I can respin it to use something else. | 20:04 |
| *** Masahiro has quit IRC | 20:04 | |
| fungi | dougwig: yes, you need a full clone for this to work properly | 20:04 |
| dougwig | i did a checkout on each branch to get it local. is there more that needs to happen? | 20:04 |
| fungi | dougwig: it's easier to do this with a bare clone, but your script looks like it wants to be able to inspect files | 20:05 |
| *** ZZelle_ has joined #openstack-infra | 20:05 | |
| *** emagana has joined #openstack-infra | 20:05 | |
| dougwig | i stole that from oslo-incubator. if there's a better way, i can do that. | 20:06 |
| clarkb | we haev devstack-trusty, bare-centos6, devstack-centos7, and bare-f20 nodes ready | 20:06 |
| clarkb | waiting on bare-trusty, bare-precise and devstack-precise | 20:06 |
| clarkb | looks like we have those and py3k-precise now | 20:06 |
| clarkb | anyone else want to double check that list of ready nodes before we start zuul? | 20:06 |
| fungi | clarkb: suggestions on how to do a full clone which is also non-bare? | 20:06 |
| fungi | clarkb: yeah, checking now | 20:07 |
| clarkb | fungi: dougwig: for branch in git branch -a | grep origin | do some sed | checkout -b branchname origin/branchname ? | 20:07 |
| *** mpaolino has quit IRC | 20:08 | |
| dougwig | ok, that's what i did. | 20:08 |
| fungi | clarkb: also checkout the list of tags? | 20:08 |
| jeblair | i stopped oob geard since i don't think we need it anymore; waiting for fungi's check before starting zuul | 20:08 |
| dougwig | those came along already. | 20:08 |
| dougwig | split is running again. | 20:08 |
| clarkb | fungi: ya tags should magically get pulled in by the branches | 20:08 |
| clarkb | jeblair: ya we shouldn't need it anymore | 20:08 |
| fungi | clarkb: jeblair: looks like we now have bare-centos6, bare-f20, bare-precise, bare-trusty, devstack-centos7, devstack-f20, devstack-f20-virt-preview, devstack-precise, devstack-trusty, devstack-trusty-2-node, py3k-precise | 20:10 |
| clarkb | ok building and ready node deletions are completely done on old nodepool. I am starting deletion of the other nodes on old nodepool now | 20:10 |
| fungi | i think that's everything we care about, unless we want to wait until tripleo jobs won't not_registered too | 20:10 |
| fungi | we're also waiting on devstack-precise-dib and devstack-trusty-dib but those aren't running anything critical | 20:11 |
| clarkb | dib should be fine, we may consider waiting for tripleo | 20:11 |
| *** Longgeek has quit IRC | 20:11 | |
| *** dizquierdo has joined #openstack-infra | 20:11 | |
| jeblair | tripleo are building? | 20:11 |
| *** rushiagr is now known as rushiagr_away | 20:11 | |
| *** koolhead17 has quit IRC | 20:11 | |
| fungi | yes | 20:11 |
| *** markmcclain has quit IRC | 20:12 | |
| fungi | no idea how successfully. they may be quota-constrained there | 20:12 |
| fungi | checking logs now | 20:12 |
| clarkb | I am deleting those nodes on the old side by hand now in hopes that it helps buils | 20:13 |
| *** gyee has joined #openstack-infra | 20:13 | |
| *** subscope_ has joined #openstack-infra | 20:14 | |
| jeblair | hrm, i think we should proceed | 20:14 |
| clarkb | jeblair: fine with me | 20:14 |
| jeblair | fungi: ? | 20:14 |
| fungi | yeah, go for it | 20:15 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: deprecate postbuildscript onsuccess and onfailure parameter names https://review.openstack.org/139257 | 20:15 |
| fungi | they can catch up if they need to | 20:15 |
| jeblair | starting zuul | 20:15 |
| clarkb | we are under 300 nodes on old nodepool side now | 20:15 |
| *** ldnunes has quit IRC | 20:16 | |
| jeblair | re-enqueuing | 20:16 |
| fungi | tripleo-f20 just went ready | 20:16 |
| *** otter768 has joined #openstack-infra | 20:16 | |
| fungi | still waiting for tripleo-precise | 20:16 |
| fungi | i see used nodes already | 20:17 |
| fungi | good sign | 20:17 |
| *** mriedem has joined #openstack-infra | 20:17 | |
| *** markmcclain has joined #openstack-infra | 20:18 | |
| clarkb | we are hitting some quota limits but that should go away as old nodepool removes nodes | 20:18 |
| clarkb | which it is doing at a reasonable rate | 20:18 |
| jeblair | enqueue is complete; merger queue is empty | 20:18 |
| jeblair | (which i think is a good thing) | 20:19 |
| *** hdd has quit IRC | 20:20 | |
| *** cnesa has quit IRC | 20:20 | |
| clarkb | under 250 old nodes now | 20:21 |
| *** tonytan4ever has quit IRC | 20:21 | |
| *** otter768 has quit IRC | 20:21 | |
| SpamapS | fungi: going back to the dib problems... is it possible the filesystem that the flock's are running on are on AFS? | 20:22 |
| SpamapS | fungi: if so.... I think I know what's up | 20:22 |
| fungi | SpamapS: nope! (though that would be pretty cool) | 20:22 |
| clarkb | SpamapS: no its an ext4 ephemeral volume | 20:22 |
| SpamapS | Ok, and also is it possible it's on top of a union filesystem? | 20:23 |
| SpamapS | chroot.. etc.? | 20:23 |
| clarkb | SpamapS: rax gives the volume to us ot could be hosted on all osrts of weirdness :) | 20:23 |
| clarkb | s/ot/it/ | 20:23 |
| SpamapS | Basically, as long as fd#9 remains open the flocks should see eachother. | 20:23 |
| SpamapS | clarkb: no not at this level | 20:24 |
| SpamapS | I mean, on the box is it an aufs or something similar. | 20:24 |
| SpamapS | sounds like no | 20:24 |
| clarkb | no its just ext4 | 20:24 |
| *** nfedotov has quit IRC | 20:24 | |
| clarkb | no lvm either | 20:24 |
| *** radez is now known as radez_g0n3 | 20:24 | |
| SpamapS | nothing below the filesystem layer matters.. flock, however, is an in-box process-namespaced thing. | 20:25 |
| SpamapS | so another possibility.. are these running inside containers? | 20:25 |
| clarkb | nope | 20:25 |
| clarkb | dib doesn't like running in a container | 20:25 |
| SpamapS | There are patches up to fix that. | 20:25 |
| SpamapS | Not sure why they haven't landed it's been months actually. :-/ | 20:26 |
| clarkb | SpamapS: ya I am still skeptical because you have to let it do things like mount | 20:26 |
| clarkb | which for dib is valuable beacuse you can build as if you were in a different OS and you can likely deal with isolation in oither ways | 20:26 |
| clarkb | but it always seems to me like a failure if we have to unisolate isolated things | 20:26 |
| clarkb | under 200 ndoes now | 20:26 |
| *** alexpilotti has quit IRC | 20:27 | |
| SpamapS | mount is something that can be done by users given the appopriate capabilities. | 20:27 |
| SpamapS | but that is neither here nor there. | 20:27 |
| clarkb | SpamapS: yes and it completely undoes any isolation you might have :) | 20:27 |
| SpamapS | the mounting is entirely so we can build the end product efficiently and produce an actual filesystem. | 20:27 |
| SpamapS | If you have a way to do that without a VM, I'm all ears. :) | 20:28 |
| *** radez_g0n3 is now known as radez | 20:28 | |
| clarkb | SpamapS: I think dib should be mounting under the container then "booting" the container in the mount | 20:28 |
| *** salv-orlando has quit IRC | 20:28 | |
| clarkb | SpamapS: and it should do that regardless of the mount because my local machine spends most of its time CPing stuff into the mounted location | 20:28 |
| clarkb | er regardless of containers or not | 20:28 |
| SpamapS | thats like, 'rewrite the whole thing to always use containers'. A fine idea, but not an easy implementation to swallow. :) | 20:29 |
| clarkb | technically its a mv but since it crosses FS boundaries it is slow like a cp | 20:29 |
| clarkb | SpamapS: no its not | 20:29 |
| *** timrc is now known as timrc-afk | 20:29 | |
| clarkb | dib should be doing this anyways regardless of containers. I just haven't gotte naround to fixing it yet | 20:29 |
| SpamapS | oh I know what you're saying | 20:29 |
| SpamapS | yeah we tried that. it's slow. | 20:29 |
| SpamapS | Running against tmpfs is about 2x faster. | 20:30 |
| SpamapS | dpkg is to blame. | 20:30 |
| clarkb | oh we don't tmpfs so I don't care | 20:30 |
| clarkb | so my interest is making not tmpfs not suck | 20:30 |
| SpamapS | and $(all the other things that use sync()) | 20:30 |
| mordred | SpamapS: did you see my "make dpkg not use sync()" patch? | 20:30 |
| clarkb | SpamapS: something like if not tmpfs then do what clarkb says | 20:30 |
| SpamapS | clarkb: we have these other users that do use it.. so.. please don't regress their experience. | 20:31 |
| SpamapS | mordred: yeah I think that would probably get 80% of the speedup tmpfs did. | 20:31 |
| SpamapS | which might be enough to ditch the tmpfs and cp method. | 20:31 |
| *** shashankhegde has quit IRC | 20:31 | |
| SpamapS | we also tried eatmydata btw.. but that broke a bunch of stuff | 20:32 |
| fungi | i have just wasted far too much of my day arguing by e-mail with a "security researcher" (it must be true because he put it in his .signature!) about how ~all in our openstack.org spf records isn't a massive security vulnerability | 20:32 |
| *** dkranz has quit IRC | 20:32 | |
| *** e0ne has quit IRC | 20:32 | |
| *** tonytan4ever has joined #openstack-infra | 20:32 | |
| mordred | SpamapS: there is a non-eatmydata dpkg option | 20:32 |
| fungi | he even went to the trouble to provide a php (php?!?) proof of concept exploit and sent a spoofed e-mail to me. oh the horror | 20:33 |
| SpamapS | mordred: yes I'm quite aware, the ubuntu installers use it. | 20:33 |
| mordred | kk | 20:33 |
| SpamapS | (both the fast one for applying updates and the d-i based one) | 20:33 |
| fungi | also he kept repeatedly misspelling smtp "smpt" and insisted for several iterations that it wasn't a typo before i finally convinced him | 20:33 |
| SpamapS | Anyway, back to concurrency issues... I'm reminded that Helion's CI does lots of concurrent builds and doesn't hit this. I wonder if there is just some poor error handling going on that is letting a failed lock get through as a working lock. | 20:34 |
| *** weshay has quit IRC | 20:34 | |
| mordred | SpamapS: https://review.openstack.org/#/c/136660/ in case you want to review ... (clarkb you too) | 20:35 |
| clarkb | just about at 100 nodes to go | 20:35 |
| jeblair | fungi: i. wow. | 20:36 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: add a turbo hipster worker for testing https://review.openstack.org/107331 | 20:37 |
| fungi | jeblair: apparently he's going to post this all over some "security" webboard and we should expect lots of people taking advantage of our customers by spoofing e-mail | 20:37 |
| fungi | i'm so afraid | 20:37 |
| clarkb | jeblair: so we did hit a TimeOut error listing functions according to the log at 12:08 which is when you killed the standalone geard | 20:37 |
| clarkb | jeblair: other than that log looks clean so far | 20:37 |
| fungi | SpamapS: clarkb: i really still think having virtual machine dib workers backending nodepool's image updates is inevitable, particularly for some unusual image types we may eventually end up wanting | 20:38 |
| clarkb | fungi: yup | 20:38 |
| clarkb | fungi: I just think we can make dib do the container thing better too :) | 20:38 |
| clarkb | but it requires changing a pretty fundamental piece of how dib works | 20:39 |
| fungi | we simply won't be able to guarantee we can vbuild all the things we want from one single starting system | 20:39 |
| clarkb | or at least that was my impression when I first looked at it and decided to leave that for later | 20:39 |
| fungi | s/vbuild/build/ | 20:39 |
| *** dizquierdo has quit IRC | 20:39 | |
| * mordred also has some container-based patches he wants to make for dib | 20:39 | |
| fungi | i think the centos 7 tar on ubuntu precise issue is just the tip of that iceberg | 20:39 |
| mordred | and also wants to be able to use dib to create containers | 20:39 |
| jeblair | merger queue is still empty | 20:40 |
| clarkb | er wait no TimeoutErrors are more recent than that. I need more caffeine | 20:41 |
| clarkb | we haev had two | 20:41 |
| SpamapS | clarkb: it's not that fundamental. There's one output from the setup process.. "Where to write things you want in the image". And then that is the input to the "make an image" process. | 20:41 |
| clarkb | 2014-12-08 20:37:16,852 and 2014-12-08 20:25:29,699 | 20:41 |
| clarkb | SpamapS: right and it should just start as chroot/container on dest of image | 20:41 |
| clarkb | SpamapS: at least as an option | 20:42 |
| *** MaxV has joined #openstack-infra | 20:42 | |
| jeblair | clarkb: no timeouterrors in zuul | 20:42 |
| clarkb | jeblair: I think you may be right about nodepool starving itself at times | 20:43 |
| mordred | sdague, morganfainberg: /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) | 20:43 |
| mordred | SecurityWarning | 20:43 |
| morganfainberg | mordred, pong, looking | 20:43 |
| clarkb | jeblair: which should be fine if it generally doesn't and the server keeps running without losing track | 20:43 |
| mordred | sdague, morganfainberg: I get this with most recent client libs installed - seems to be a requests thing - is it a thing we're aware of/care about? | 20:43 |
| mordred | morganfainberg: OR -is that telling me that the target cert I'm connecting to is yuck? | 20:44 |
| morganfainberg | not sure. | 20:44 |
| sdague | so... I thought we weren't using their vendored urllib3 any more | 20:44 |
| morganfainberg | sdague, i.. lost track of that conversation | 20:44 |
| morganfainberg | and i *hate* that vendored lib thing | 20:44 |
| sdague | I think it's that your cert is crap | 20:44 |
| mordred | jroll: it's possible that there is a rax cert issue ^^ OR that there is simply a bonghits client lib on our side. no idea which | 20:44 |
| morganfainberg | but i think that is a cert issue | 20:44 |
| fungi | mordred: morganfainberg: that sounds backwards... it's complaining that there's no subjectAltName and that it has to fall back on commonName? | 20:44 |
| *** _nadya_ has quit IRC | 20:45 | |
| morganfainberg | fungi, yeah that *sounds* like a bad cert. | 20:45 |
| morganfainberg | and it's complaining that it's trying to fallback? | 20:45 |
| mordred | morganfainberg: I think maybe falling back is the thign that's deprecated? | 20:45 |
| morganfainberg | mordred, ++ | 20:45 |
| morganfainberg | that sounds right | 20:46 |
| clarkb | in general it appeas to be listing demand and allocating resources so Iam not worried yet | 20:46 |
| *** baoli has joined #openstack-infra | 20:46 | |
| fungi | morganfainberg: but, i mean, shouldn't using only commonName be perfectly safe? why would that be warning-fodder | 20:46 |
| fungi | ? | 20:46 |
| clarkb | jeblair: and the launch all the nodes at start may contribute | 20:46 |
| mordred | jroll, JayF: so, as you guys are my current funnels into rackspace - there may be a cert issue there that might want to be raised with someone | 20:46 |
| morganfainberg | fungi, yeaaaah i don't know the details on RFC2818 | 20:46 |
| mordred | jroll, JayF: its not a problem for us - just being friendly | 20:46 |
| *** baoli has quit IRC | 20:47 | |
| dougwig | fungi, anteaya - new repos, with branches and tags: | 20:47 |
| dougwig | https://www.irccloud.com/pastebin/apwmsjHE | 20:47 |
| *** weshay has joined #openstack-infra | 20:47 | |
| fungi | morganfainberg: it strikes me as being like an mta that warns that it had to fall back on an a rr because the destination had no corresponding mx rr | 20:47 |
| *** baoli has joined #openstack-infra | 20:47 | |
| jeblair | clarkb, fungi: i'm starting full packet dumps on zuul and nodepool (filtering for each other, so not all traffic) to correlate with the next timeouterror | 20:47 |
| fungi | like, you should just do that. it's a legitimate configuration | 20:47 |
| *** shashankhegde has joined #openstack-infra | 20:47 | |
| fungi | dougwig: thanks--looking at latest iteration now | 20:47 |
| morganfainberg | fungi, sure. but if $SECURITY_PEOPLE have determined that that behavior in certs can be abused [maybe thats the issue], that fallback is not sane? | 20:48 |
| morganfainberg | fungi, i think this is something i want to know more about now. | 20:48 |
| *** Sukhdev has joined #openstack-infra | 20:49 | |
| fungi | jeblair: sounds good... nodepool on the precise host exhibited the same issue a couple times right after we started it, last time we switched back, but then didn't continue to do that with any regularity | 20:49 |
| fungi | morganfainberg: agreed. i'm looking for explanations as to why a lack of subjectAltName would be a risk | 20:50 |
| jeblair | fungi: yeah, i think thread starvation as clarkb suggested is not inconceivable | 20:50 |
| clarkb | we are down to 15 nodes on old nodepool. they are being stubborn | 20:50 |
| fungi | in which case not a regression in behavior, just something else which needs fixing | 20:50 |
| jeblair | fungi: this should help tell us if the problem is server or client side if it happens again. and if it doesn't happen again, it helps support that idea :) | 20:50 |
| morganfainberg | fungi, ok... odd: If a subjectAltName extension of type dNSName is present, that MUST | 20:50 |
| morganfainberg | be used as the identity. Otherwise, the (most specific) Common Name | 20:50 |
| morganfainberg | field in the Subject field of the certificate MUST be used. Although | 20:50 |
| morganfainberg | the use of the Common Name is existing practice, it is deprecated and | 20:50 |
| morganfainberg | Certification Authorities are encouraged to use the dNSName instead. | 20:50 |
| fungi | totally | 20:50 |
| jroll | mordred: lovely, thenks | 20:50 |
| * jroll looks into this | 20:50 | |
| mordred | morganfainberg, jroll: for context, I get that warning several times now on every operation against rackspce cloud using python-*client | 20:50 |
| morganfainberg | wow, that was not expected to be multiple lines... | 20:50 |
| morganfainberg | ssorry | 20:50 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add oslo.context https://review.openstack.org/140141 | 20:51 |
| jroll | mordred: do you have a gist or something that's more comprehensible than scrollback? | 20:51 |
| mordred | jroll: I will make you one | 20:51 |
| jroll | thanks | 20:51 |
| morganfainberg | oh interesting, this might be something-something TLS with wildcards | 20:51 |
| fungi | morganfainberg: interesting--so that's quoted from 2818? | 20:51 |
| morganfainberg | fungi, yeah | 20:51 |
| morganfainberg | and chromium deprecated/removed it | 20:52 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add oslo.context https://review.openstack.org/140141 | 20:52 |
| *** jamespage_ has joined #openstack-infra | 20:52 | |
| *** marun has joined #openstack-infra | 20:52 | |
| morganfainberg | so urllib3 is | 20:52 |
| morganfainberg | following in suit | 20:52 |
| mordred | jroll: http://paste.openstack.org/show/147441/ | 20:52 |
| fungi | morganfainberg: right, subjectAltName is there to handle name-based virtual hosting, which is why it seemed odd it would be viewed as a preferred field already | 20:52 |
| mordred | jroll: there's the command and output to reproduce | 20:52 |
| morganfainberg | fungi, yep | 20:52 |
| morganfainberg | fungi, well. apparantly 10 yrs ago | 20:53 |
| jroll | mordred: thanks | 20:53 |
| * fungi suddenly feels very old | 20:53 | |
| morganfainberg | "preferred" according to the urllib3 bug | 20:53 |
| morganfainberg | fungi, eyah... | 20:53 |
| clarkb | jeblair: we just had a timeout | 20:53 |
| jroll | mordred: is this new today? new version of urllib3 or? | 20:53 |
| mordred | jroll: looks like changes in tls/ssl related to RFC2818 are producing deprecating warnings with the certs on some elements of public RAX api endpoints | 20:53 |
| clarkb | jeblair 2014-12-08 20:53:14,979 ERROR nodepool.GearmanClient: Exception while listing function | 20:53 |
| morganfainberg | fungi, so - uh - yeah i think it's a fair warning now | 20:53 |
| jroll | mordred: word, ty | 20:53 |
| mordred | jroll: this is just most recent python-glanceclient and dependencies - so it might be new urllib3 and/or new requests lib | 20:53 |
| fungi | dougwig: all three lgtm now--i see the branches and tags i'd expect... did you need anyone else to bless it before i cram it into gerrit? | 20:54 |
| mordred | jroll: but it happens with python-novaclient too - so I think it's anything using requests, tbh | 20:54 |
| dougwig | no, good to go. | 20:54 |
| *** Rockyg has quit IRC | 20:54 | |
| morganfainberg | mordred, yeah this is a cert issue | 20:54 |
| dougwig | i'm sitting next to mestery and anteaya | 20:54 |
| morganfainberg | mordred, with a new-ish requests with a new urllib3 | 20:54 |
| *** mrmartin has quit IRC | 20:54 | |
| fungi | morganfainberg: at any rate, i'm going to be paying much closer attention to the subjectAltName in my certs now ;) | 20:54 |
| *** dprince has quit IRC | 20:55 | |
| morganfainberg | fungi, ack | 20:55 |
| fungi | i used to mostly ignore it because i was doing per-ip-address exclusively | 20:55 |
| *** yamamoto has joined #openstack-infra | 20:56 | |
| clarkb | jeblair: and again. there were a lot of ssh timeouts too | 20:56 |
| clarkb | it does seem to recover quickly from those timeouts though. not too long later we get successful listings | 20:57 |
| jeblair | clarkb: yeah, which also supports either thread starvation or network hiccup | 20:57 |
| *** MaxV has quit IRC | 20:57 | |
| fungi | morganfainberg: yep, on closer inspection, all my certs have my X509v3 Subject Alternative Name DNS matching the Subject CN, and othername marked unsupported | 20:57 |
| *** e0ne has joined #openstack-infra | 20:59 | |
| *** jamespage_ has quit IRC | 21:00 | |
| *** MaxV has joined #openstack-infra | 21:01 | |
| *** e0ne has quit IRC | 21:02 | |
| *** sputnik13 has joined #openstack-infra | 21:04 | |
| *** annegent_ has joined #openstack-infra | 21:04 | |
| *** sputnik13 has quit IRC | 21:06 | |
| *** jamespage_ has joined #openstack-infra | 21:06 | |
| *** amcrn has quit IRC | 21:06 | |
| *** e0ne has joined #openstack-infra | 21:06 | |
| *** craigbr has quit IRC | 21:07 | |
| clarkb | the ready and used node numbers are climbing | 21:07 |
| lbragstad | hi all, quick question on https://github.com/stackforge/saml-generator this project. There doesn't seem to be anyone listed as the core maintainers (https://review.openstack.org/#/admin/groups/436,members). Outside of the normal core proposal process, is there a way to 'boot strap' a core group? | 21:07 |
| clarkb | we appear to be chewing through the backlog and I don't see oscilations yet | 21:07 |
| clarkb | lbragstad: the people that create the project request an initial group member. That initial member can then fill out the remaining members | 21:08 |
| *** rkukura has joined #openstack-infra | 21:08 | |
| *** kgiusti has left #openstack-infra | 21:09 | |
| *** sputnik13 has joined #openstack-infra | 21:09 | |
| *** annegent_ has quit IRC | 21:11 | |
| lbragstad | clarkb: ok, that makes sense, I don't believe the original proposer is still active on the project (https://github.com/openstack-infra/project-config/blame/master/gerrit/acls/stackforge/saml-generator.config). Is there another way to do it? | 21:12 |
| clarkb | looks like the project came out of rackerlabs? maybe someone from rax wants to have ownership? | 21:13 |
| clarkb | lbragstad: I would start by contacting the original proposer | 21:13 |
| lbragstad | I have a co-worker that wants to pick it up? https://review.openstack.org/#/c/124080/ | 21:13 |
| *** salv-orlando has joined #openstack-infra | 21:14 | |
| *** sputnik13 has quit IRC | 21:14 | |
| lbragstad | clarkb: I'll do a little more digging and see what i can come up with. Thank you for the help! | 21:14 |
| clarkb | I guess my only concern is handing it off to someone to have the original interested party go "huh? /me is still here" | 21:14 |
| *** armax has quit IRC | 21:14 | |
| *** mriedem1 has joined #openstack-infra | 21:15 | |
| *** fifieldt_ has joined #openstack-infra | 21:16 | |
| lbragstad | clarkb: yeah, I use to work with the original maintainer but he moved to a new project a couple months ago | 21:16 |
| lbragstad | I'll have the two of them work it out | 21:17 |
| *** mriedem has quit IRC | 21:17 | |
| mattoliverau | Morning | 21:18 |
| clarkb | mattoliverau: good morning | 21:18 |
| *** julim has joined #openstack-infra | 21:19 | |
| mordred | look. australia has awokened | 21:19 |
| mordred | mattoliverau: congrats. you are now all of oz | 21:19 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Converted application to use utcnow() https://review.openstack.org/140148 | 21:19 |
| *** fifieldt has quit IRC | 21:19 | |
| mattoliverau | I always suspected it :p | 21:19 |
| * krotscheck is rather embarrassed that ^^ was a thing. | 21:19 | |
| *** salv-orlando has quit IRC | 21:20 | |
| grue_pm | mordred no he's not | 21:20 |
| jeblair | krotscheck: did you mean 'prescribe' instead of 'proscribe' in that commit msg? | 21:21 |
| * grue_pm turns out the lights and glares | 21:21 | |
| mordred | grue_pm: fair | 21:21 |
| krotscheck | oh | 21:21 |
| krotscheck | hah | 21:21 |
| krotscheck | sec. | 21:21 |
| *** marun has quit IRC | 21:21 | |
| *** baoli has quit IRC | 21:21 | |
| mordred | krotscheck: or maybe circumscribe ... or circumnavigate! | 21:21 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Converted application to use utcnow() https://review.openstack.org/140148 | 21:21 |
| *** andreykurilin_ has joined #openstack-infra | 21:22 | |
| clarkb | woot we are down to 5 building nodes and have a ton used and a few ready I think nodepool is mostly beyond the shock of being moved | 21:22 |
| krotscheck | mordred: circumstantiated? | 21:22 |
| clarkb | jeblair: fungi ^ from about now on is where we should look for that oscillation to happen | 21:22 |
| *** MaxV has quit IRC | 21:23 | |
| mtreinish | mordred: heh, we need to circumnavigate more in commits... | 21:24 |
| *** TimO_ has quit IRC | 21:26 | |
| *** tonytan4ever has quit IRC | 21:26 | |
| *** tonytan_brb has joined #openstack-infra | 21:26 | |
| fungi | this is all very circumspect | 21:26 |
| *** MaxV has joined #openstack-infra | 21:27 | |
| mordred | clarkb, fungi, jeblair: I believe I have end-to-end dib-rax working - although as I mentioned I'd really liek to land it using cloud-init instead | 21:28 |
| mordred | but I have now created an image, uploaded it, created a vm based on it and ssh'd in to that vm | 21:28 |
| *** e0ne has quit IRC | 21:29 | |
| fungi | mestery: dougwig: anteaya: i have the pushes going in now, though it's pretty slow-going (lots of history for it to deal with). i'll give you a heads-up when it completes | 21:30 |
| mestery | fungi: Thanks man! | 21:30 |
| dougwig | fungi: sweet, thank you | 21:30 |
| jhesketh | Morning | 21:31 |
| clarkb | hrm not so sure the Beginning node launch calculation and Finished node launch calculations are happening as quickly as I expected | 21:32 |
| clarkb | jeblair: fungi ^ | 21:32 |
| *** gyee has quit IRC | 21:32 | |
| jeblair | clarkb: will look | 21:33 |
| *** e0ne has joined #openstack-infra | 21:34 | |
| jeblair | clarkb: last begin->end took 4 seconds, is that what you mean? | 21:34 |
| jeblair | clarkb: or do you mean end->begin ? | 21:34 |
| clarkb | jeblair: end -> begin | 21:34 |
| jeblair | ah k | 21:34 |
| clarkb | jeblair: beacuse when we timeout we end then don't start up again quickly | 21:34 |
| fungi | mestery: dougwig: anteaya: http://git.openstack.org/cgit/openstack/neutron-fwaas is showing the branches and tags replicated there now | 21:35 |
| dims | fungi: sdague: clarkb: can one of you please take a peek at https://review.openstack.org/#/c/137372/ when you get a chance? (Update to Zuul Status page, has +2 from Sergey) | 21:36 |
| *** tkelsey has quit IRC | 21:36 | |
| jeblair | clarkb: the main loop has a delay of 10 seconds, so i would expect timeout -> start to be ~ 10 secs; but we haven't seen a timeout recently, right? | 21:36 |
| clarkb | jeblair: we did at 21:33 UTC | 21:36 |
| clarkb | jeblair: 2014-12-08 21:33:14,836 ERROR nodepool.GearmanClient: Exception while listing functions | 21:37 |
| mestery | fungi: Looking sharp! | 21:37 |
| *** radez is now known as radez_g0n3 | 21:37 | |
| jeblair | clarkb: gah, was tailing log on wrong host :( | 21:37 |
| clarkb | jeblair: oh :) | 21:37 |
| jeblair | clarkb: fortunately, was tcpdumping on correct one | 21:38 |
| *** MaxV has quit IRC | 21:38 | |
| clarkb | I am not sure its super happy right now but yay to tcpdumps | 21:38 |
| *** tonytan_brb is now known as tonytan4ever | 21:38 | |
| jeblair | clarkb: what do you think is unhappy? | 21:38 |
| *** armax has joined #openstack-infra | 21:38 | |
| clarkb | jeblair: the number of times we calculate demand seems to be low | 21:39 |
| clarkb | jeblair: especiall when you factor in timeouts | 21:39 |
| *** marun has joined #openstack-infra | 21:39 | |
| fungi | mestery: dougwig: anteaya: http://git.openstack.org/cgit/openstack/neutron-lbaas looks correct to me now too | 21:40 |
| dougwig | ty | 21:41 |
| *** nelsnelson has joined #openstack-infra | 21:41 | |
| clarkb | zuul geard seems super responsive to me, so not sure why nodepool is so unhappy unless it is related to starvation within that one process | 21:41 |
| *** aysyd has quit IRC | 21:41 | |
| jeblair | i'm going to analyze the tcp dumps now | 21:42 |
| fungi | clarkb: yeah, wondering whether maybe nodepool is waiting on itself, not on teh gearman server? | 21:42 |
| clarkb | jeblair: fungi: if you change the node graph from zuul status page to -4 hours you can see it oscilate | 21:42 |
| clarkb | the oscilations are definitely not as far apart as they were though so I think we sort of made it better | 21:42 |
| *** nelsnelson has quit IRC | 21:43 | |
| fungi | mestery: dougwig: anteaya: http://git.openstack.org/cgit/openstack/neutron-vpnaas just finished now too. let me know if anything looks awry | 21:43 |
| dougwig | fungi: checking | 21:43 |
| mestery | fungi: This is fantastic, thanks again for the help! Thanks to anteaya for the guidance in person as well. :) | 21:44 |
| *** nelsnelson has joined #openstack-infra | 21:44 | |
| *** nelsnelson has quit IRC | 21:45 | |
| *** nelsnelson has joined #openstack-infra | 21:45 | |
| *** sarob has joined #openstack-infra | 21:46 | |
| *** nelsnelson has quit IRC | 21:48 | |
| *** subscope_ has quit IRC | 21:48 | |
| clarkb | top does say that nodepoold is basically using an entire CPU at all times | 21:48 |
| *** Masahiro has joined #openstack-infra | 21:48 | |
| clarkb | so maybe the way we tackle this is to make nodepoold more multi CPU aware | 21:48 |
| *** cpowell has quit IRC | 21:49 | |
| clarkb | looking at graphs more. I think it may be related to the number of building nodes | 21:49 |
| *** yamamoto has quit IRC | 21:49 | |
| *** mpaolino has joined #openstack-infra | 21:49 | |
| clarkb | building nodes will steadily drop then spike and so on | 21:49 |
| clarkb | hrm same with deleting | 21:50 |
| *** teran has joined #openstack-infra | 21:50 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/reviewstats: Update neutron core https://review.openstack.org/140159 | 21:50 |
| *** Masahiro has quit IRC | 21:53 | |
| *** nelsnelson has joined #openstack-infra | 21:53 | |
| *** armax has quit IRC | 21:54 | |
| *** MaxV has joined #openstack-infra | 21:54 | |
| *** markmcclain has quit IRC | 21:54 | |
| *** rkukura has quit IRC | 21:55 | |
| mestery | fungi: Relocating, but had this for you: | 21:55 |
| mestery | fungi: Can we force merge this one (https://review.openstack.org/#/c/139901/), it's waiting for a tempest test to merge to disable services tests to pass the gate. | 21:55 |
| mestery | fungi: This is from anteaya and her advice too. | 21:56 |
| mestery | fungi: OK, moving, think that over and I'll be back | 21:56 |
| dougwig | hold up 2 mins | 21:56 |
| *** jgrimm is now known as zz_jgrimm | 21:57 | |
| fungi | mestery: dougwig: anteaya: merging that will definitely mean that no neutron changes are able to pass those jobs and will be completely broken until the corresponding test changes merge | 21:57 |
| *** obondarev has quit IRC | 21:58 | |
| clarkb | CPU use is low right now and nodepool is calculating launch nodes | 21:58 |
| clarkb | so I think it must be that | 21:58 |
| *** zz_jgrimm is now known as jgrimm | 21:58 | |
| *** marun has quit IRC | 21:58 | |
| fungi | mestery: dougwig: anteaya: if you wait until https://review.openstack.org/140091 lands you should be able to self-test 139901 | 21:59 |
| *** Sukhdev has quit IRC | 22:00 | |
| fungi | also, looks like we passed the 140k changes mark today | 22:00 |
| *** boris-42 has joined #openstack-infra | 22:00 | |
| mriedem1 | what happened to the nice nodepool graph on zuul.o.o? | 22:00 |
| mriedem1 | or can someone point me at docs for what the current limit for nodepool is? trying to put together some education slides. | 22:01 |
| clarkb | mriedem1: it should still be there | 22:01 |
| fungi | mriedem1: you want status.o.o | 22:01 |
| *** shashankhegde has quit IRC | 22:01 | |
| mriedem1 | fungi: ah yea | 22:01 |
| mriedem1 | so looks like 800 is about the max for nodepool? | 22:02 |
| clarkb | mriedem1: I think ~900 | 22:02 |
| fungi | mriedem1: just add up all the max-servers values in http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb if you want a more exact number, but that's fairly close yeah | 22:02 |
| *** teran_ has joined #openstack-infra | 22:03 | |
| clarkb | ya we definitely timeout as nodepoold related load climbs | 22:03 |
| mriedem1 | fungi: ok, cool, that helps me | 22:03 |
| fungi | mriedem1: clarkb: i get 884 | 22:04 |
| *** pradk has quit IRC | 22:04 | |
| clarkb | also that number is somewhat artificial due to hpcloud node sizestuff | 22:04 |
| fungi | right | 22:05 |
| *** dkranz has joined #openstack-infra | 22:05 | |
| *** jamespage_ has quit IRC | 22:05 | |
| mriedem1 | ok, ~900 it is though, i'm just doing a lunch and learn for january, very high level intro on openstack | 22:05 |
| *** teran has quit IRC | 22:05 | |
| *** Sukhdev has joined #openstack-infra | 22:05 | |
| mordred | [ 3.799143] input: PC Speaker as /devices/platform/pcspkr/input/input6 | 22:06 |
| mordred | really? | 22:06 |
| fungi | could be higher come january ;) | 22:06 |
| fungi | mordred: did you find an emulated pc speaker on a vm? | 22:06 |
| mordred | fungi: yup | 22:06 |
| mordred | fungi: connected as an input device | 22:06 |
| fungi | mordred: floppy drives too? | 22:06 |
| fungi | oh, as an input! | 22:06 |
| fungi | wow | 22:06 |
| *** marun has joined #openstack-infra | 22:06 | |
| jeblair | clarkb, fungi: network traffic for 21:33:14,836 looks good. entire request/response happened betwen 21:31:44 and 21:31:47 | 22:07 |
| mordred | also, there's a QEMU USB Tablet too | 22:07 |
| jeblair | clarkb, fungi: network traffic for 21:33:14,836 TimeoutError looks good. entire request/response happened betwen 21:31:44 and 21:31:47 | 22:07 |
| * mordred boggles | 22:07 | |
| fungi | jeblair: yeah, this does sound like it's all on the nodepool end then | 22:07 |
| clarkb | yup I am almost positive it is | 22:08 |
| *** armax has joined #openstack-infra | 22:08 | |
| clarkb | I have roughly correlated timeouts to periods of high load and successes to periods of low load | 22:08 |
| fungi | jeblair: so it sent the request, had the full response within 3 seconds, then spun on its heels for several minutes | 22:08 |
| *** mbacchi has quit IRC | 22:08 | |
| dtroyer | mordred: in case you haven't seen this yet: http://www.qemu-advent-calendar.org/ | 22:08 |
| *** dkliban is now known as dkliban_Afk | 22:08 | |
| *** dkliban_Afk is now known as dkliban_afk | 22:08 | |
| *** shashankhegde has joined #openstack-infra | 22:08 | |
| clarkb | fungi: ya | 22:08 |
| clarkb | I wonder if either python or our kernel is doing something ufnny to us when scheduling | 22:09 |
| *** markmcclain has joined #openstack-infra | 22:10 | |
| jeblair | clarkb, fungi: the timeout error is generated by a thread condition, and probably ultimately is generated by the kernel, so it makes sense that it could fire before the thread we are interested in has gotten around to reading from the socket and processing the data. | 22:10 |
| *** oomichi has joined #openstack-infra | 22:11 | |
| clarkb | I think the server side changes have made nodepool timeouts less terrible though | 22:11 |
| anteaya | fungi: thank you | 22:11 |
| clarkb | We could try using not pvhvm image for this node | 22:12 |
| *** obondarev has joined #openstack-infra | 22:12 | |
| jeblair | clarkb: yeah, i think at this point we've cleared server-side problems; if new geard continues to hold up, we're in a much better place to work from | 22:12 |
| *** mestery has quit IRC | 22:12 | |
| jeblair | (much easier to change nodepool/gear client than zuul/gear server) | 22:12 |
| *** teran has joined #openstack-infra | 22:13 | |
| mordred | dtroyer: magic | 22:14 |
| dougwig | fungi: concur on waiting on 140091 | 22:15 |
| *** reed has joined #openstack-infra | 22:16 | |
| *** teran_ has quit IRC | 22:17 | |
| *** otter768 has joined #openstack-infra | 22:17 | |
| fungi | dougwig: it passed check | 22:17 |
| dougwig | fungi: i just pinged QA. | 22:18 |
| dougwig | that plus a quick db unit fix and i think the neutron patch will go in clean. | 22:18 |
| fungi | excellent. the gate's not busy, so shouldn't have to wait too terribly long | 22:18 |
| oomichi | anteaya: hi | 22:18 |
| *** rkukura has joined #openstack-infra | 22:19 | |
| anteaya | oomichi: hello | 22:19 |
| oomichi | anteaya: re: https://review.openstack.org/#/c/138951 | 22:19 |
| anteaya | oomichi: what are you asking from me? | 22:19 |
| clarkb | jeblair: fungi: other than making nodepool more multi cpu aware I could build another host on a non pvhvm image | 22:19 |
| clarkb | but I think those are probably our two options | 22:19 |
| oomichi | anteaya: we can get +1s from nova-drivers and it seems enough to go. | 22:20 |
| anteaya | I'm at netron sprnt and kind of focused | 22:20 |
| *** MaxV has quit IRC | 22:20 | |
| fungi | clarkb: what's the expectation, that pvhvm is making context switches more expensive or something? | 22:20 |
| jeblair | clarkb: since we think this is a python threading thing, we should be able to reproduce on test machines using nodepool's fake mode | 22:20 |
| oomichi | anteaya: ok, will see you later. | 22:21 |
| clarkb | fungi: possibly or potentially tickling the newer kernel in unexpected ways | 22:21 |
| jeblair | clarkb: overall load average is about the same as on the old host | 22:21 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for running Neutron with IPv6 Networking https://review.openstack.org/140163 | 22:21 |
| *** otter768 has quit IRC | 22:21 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 22:23 |
| *** tonytan4ever has quit IRC | 22:23 | |
| openstackgerrit | Michael Pratt proposed openstack-infra/git-review: Add reviewers on upload https://review.openstack.org/139165 | 22:24 |
| *** salv-orlando has joined #openstack-infra | 22:24 | |
| fungi | dougwig: what prompted 140164? | 22:25 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: add a turbo hipster worker for testing https://review.openstack.org/107331 | 22:25 |
| *** mfink has quit IRC | 22:25 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 22:25 |
| dougwig | fungi: mtreinish didn't like the other patch, even though it was a temp fix. so i was trying an alternate, which i don't think will actually work. | 22:25 |
| jeblair | clarkb, fungi: http://paste.openstack.org/show/147485/ | 22:25 |
| mtreinish | dougwig: if you give me a couple min, I'll add onto that patch which will also skip the tempest tests... | 22:26 |
| jeblair | clarkb, fungi: on the new one, i think hour 20 has some time with no load, and hour 22 is only half over | 22:26 |
| dougwig | mtreinish: sweet, was just researching the right way. will definitely be happy to wait. thank you! | 22:26 |
| mtreinish | dougwig: I just need to spin up devstack to generate the complete ext list for master | 22:27 |
| *** salv-orlando has quit IRC | 22:27 | |
| *** mestery has joined #openstack-infra | 22:28 | |
| *** rkukura has quit IRC | 22:29 | |
| *** andreykurilin_ has quit IRC | 22:30 | |
| *** mfink has joined #openstack-infra | 22:30 | |
| *** andreykurilin_ has joined #openstack-infra | 22:31 | |
| *** mpaolino has quit IRC | 22:31 | |
| *** rkukura has joined #openstack-infra | 22:31 | |
| ZZelle_ | fungi, hi | 22:32 |
| *** ChuckC has quit IRC | 22:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 22:32 |
| jeblair | clarkb, fungi: i'm going to spin up a test node and see if i can produce the bad behavior with fakes | 22:32 |
| clarkb | jeblair: ok let me know if I can help | 22:33 |
| *** ociuhandu has joined #openstack-infra | 22:34 | |
| *** kumartin has joined #openstack-infra | 22:35 | |
| *** mestery has quit IRC | 22:35 | |
| jgriffith | anteaya: ping | 22:36 |
| jgriffith | anteaya: I *really* hate to bug you with this.... but :( | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add debootstrap to the needed packages for dib https://review.openstack.org/124077 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Make apt skip grabbing translations https://review.openstack.org/136837 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add elements for Infra servers https://review.openstack.org/136597 | 22:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add rinse to nodepool https://review.openstack.org/136598 | 22:36 |
| clarkb | jeblair: I did use the pvhvm trusty image from rax when I built this node. So you likely want to start with that image | 22:36 |
| *** shashankhegde has quit IRC | 22:36 | |
| jgriffith | anteaya: do we have anything in mind for the overwhelming email spam from all the Cinder 3'rd paty CI systems? | 22:36 |
| jeblair | clarkb: yep, building that now. | 22:37 |
| jgriffith | anteaya: or should I just create a filter in my inbox | 22:37 |
| fungi | ZZelle_: hey there, what's up? | 22:37 |
| clarkb | jgriffith: I think most of us bitbucket gerrit email fwiw. Then use tools like gertty to better poll gerrit for things you are interested in | 22:38 |
| jgriffith | clarkb: yeah... I seem to have made the mistake of not doing that with Cinder reviews | 22:38 |
| jgriffith | clarkb: time to evolve :) | 22:38 |
| jgriffith | clarkb: fair answer, thanks | 22:38 |
| jeblair | jgriffith: that, but also we did make it a requirement to have "^.*CI" as a name, so if you do need to match on that, you should be able to | 22:39 |
| jeblair | jgriffith: "^.* CI$" even | 22:39 |
| ZZelle_ | fungi, if you have some times, could have a look to https://review.openstack.org/109851 ? which is a great feature for lazy people (=>me) | 22:39 |
| jgriffith | jeblair: yeah.. I've been meaning to try that. Ok, thanks folks | 22:40 |
| fungi | jgriffith: what i did was set up e-mail filters on the gerrit commenter full e-mail address with a pattern match on " CI <" | 22:40 |
| anteaya | jgriffith: clarkb's and jeblair's suggestions sound like the way forward | 22:40 |
| jeblair | actually "^.*? CI$". i love regexes. | 22:40 |
| anteaya | and fungi's | 22:40 |
| fungi | er, what jeblair just said | 22:40 |
| * fungi is juggling too many conversations | 22:40 | |
| fungi | jgriffith: but that was the main impetus originally for standardizing the account display names | 22:40 |
| *** ihrachyshka has quit IRC | 22:40 | |
| fungi | to make them easier to filter (in e-mail as well as the webui) | 22:41 |
| *** timrc-afk is now known as timrc | 22:41 | |
| *** mestery has joined #openstack-infra | 22:41 | |
| *** shashankhegde has joined #openstack-infra | 22:42 | |
| jgriffith | fungi: now it all becomes clear | 22:43 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting docs job for git-review https://review.openstack.org/139581 | 22:44 |
| *** teran_ has joined #openstack-infra | 22:44 | |
| *** rkukura has quit IRC | 22:44 | |
| fungi | ZZelle_: looking | 22:44 |
| *** mwagner_lap has quit IRC | 22:45 | |
| *** teran has quit IRC | 22:45 | |
| ZZelle_ | fungi, i added a question about using git ls-remote --get-url when available and disabling the feature otherwise (which seems more bulletproof) | 22:46 |
| jeblair | ./launch-node.py jeblairtest.openstack.org --flavor "8 GB Performance" --cert nodepool.openstack.org.pem --image "Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)" | 22:46 |
| jeblair | clarkb, fungi: ^ any idea why that would not work? | 22:46 |
| *** packet has quit IRC | 22:46 | |
| jeblair | clarkb, fungi: http://paste.openstack.org/show/147492/ | 22:46 |
| jeblair | oh, ci-puppetmaster? | 22:47 |
| *** alexpilotti has joined #openstack-infra | 22:47 | |
| jeblair | launch-node is old | 22:47 |
| fungi | jeblair: current launch-node.py is doing that, or you didn't update to tip of master first? | 22:48 |
| jeblair | fungi: i used what was in ~root which was not updated | 22:48 |
| fungi | aha, i usually run it non-root from my homedir | 22:48 |
| jeblair | perhaps i should just remove that checkout :) | 22:48 |
| fungi | hence hadn't run into that | 22:48 |
| *** mattfarina has quit IRC | 22:48 | |
| fungi | and i always update to tip of master plus whatever patches to the launch tooling i'm testing if needed | 22:49 |
| *** yamamoto has joined #openstack-infra | 22:50 | |
| *** eharney has quit IRC | 22:50 | |
| clarkb | ya I run it from my homedir too | 22:51 |
| *** rkukura has joined #openstack-infra | 22:53 | |
| clarkb | jeblair: so we are not running https://review.openstack.org/#/c/136910/1 on new nodepool best I can tell | 22:54 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 22:54 |
| *** yamamoto has quit IRC | 22:54 | |
| mtreinish | dougwig: ^^^ | 22:54 |
| dougwig | mtreinish: ty, looking | 22:54 |
| clarkb | jeblair: but I am not sure that that matters since the server should disconnect us | 22:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Users may now issue their own access tokens. https://review.openstack.org/140176 | 22:55 |
| krotscheck | zaro: ^^ | 22:55 |
| *** bknudson has quit IRC | 22:55 | |
| krotscheck | Actually, lemme rebase that on master. | 22:55 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Users may now issue their own access tokens. https://review.openstack.org/140176 | 22:55 |
| krotscheck | There we go | 22:55 |
| jeblair | clarkb: we are probably not running it; it's probably not urgent, but we should get that in so that if something goes wrong, we don't get stuck unable to query gearman | 22:55 |
| jeblair | clarkb: the server may not disconnect us | 22:56 |
| * grue_pm wonders why she consistently reads mtrenish as mctrenish | 22:56 | |
| jeblair | or "mac t" as we will all now call him | 22:57 |
| clarkb | jeblair: oh right as the server may not see that as an error having occurred | 22:57 |
| clarkb | jeblair: should I go ahead and approve that change to nodepool now? | 22:57 |
| jeblair | clarkb: prolly a good idea | 22:57 |
| clarkb | or do you want to monkeypatch it in first | 22:58 |
| anteaya | jeblair: + | 22:58 |
| *** rlandy has quit IRC | 22:58 | |
| *** vigneshvar has quit IRC | 22:58 | |
| clarkb | ok approving now | 22:59 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 22:59 |
| mtreinish | dougwig, anteaya: heh, we probably should make sure that tempest skips all the approriate things before we approve it. Because I didn't test it locally :) | 22:59 |
| mordred | grue_pm: that's the best new name for mtreinish ever - mtreinish, you have a new casual nick friday nick I think | 23:00 |
| *** weshay has quit IRC | 23:01 | |
| anteaya | mtreinish: oh | 23:01 |
| anteaya | good to know | 23:01 |
| openstackgerrit | Merged openstack-infra/reviewstats: Updating horizon core https://review.openstack.org/139694 | 23:02 |
| anteaya | mtreinish: hyphen or underscore to bring your t along with you, mac t? | 23:02 |
| openstackgerrit | Merged openstack-infra/nodepool: Reconnect to gearman on error https://review.openstack.org/136910 | 23:02 |
| *** ayoung has quit IRC | 23:03 | |
| mtreinish | mordred: heh, it will definitely cause less confusion than the last one I used, mriedem-dplgnr | 23:03 |
| openstackgerrit | Merged openstack-infra/reviewstats: Adjust wording for new Workflow URLs https://review.openstack.org/139719 | 23:03 |
| jeblair | clarkb: did you have a problem compiling zmq? | 23:03 |
| *** rkukura has quit IRC | 23:04 | |
| grue_pm | mordred: well it's in line with aussie nicknames - we shorten everything and add y where possible ;-p | 23:04 |
| anteaya | mtreinish: ha a ha | 23:04 |
| *** alex7376 is now known as asettle | 23:04 | |
| anteaya | grue_pm: mtreinish is half-aussie, the other aussie's picked up on it | 23:04 |
| anteaya | grue_pm: mtreinish he still tries to deny it though | 23:05 |
| clarkb | jeblair: ya, you need https://review.openstack.org/#/c/136418/ | 23:06 |
| clarkb | jeblair: sorry forgot about that until you asked. we should probabl merge that change | 23:06 |
| *** salv-orlando has joined #openstack-infra | 23:06 | |
| zaro | krotscheck: thanks. i'm currently reviewing the crontab one | 23:06 |
| mtreinish | anteaya: heh hardly, I'm pretty sure that I'm the first person in my family who's ever gone to AU... | 23:07 |
| jeblair | clarkb: thx | 23:07 |
| krotscheck | zaro: Mind you, that patch only makes it easy for users to issue tokens for themselves. Admins can’t yet issue arbitrary tokens for users. | 23:07 |
| zaro | nibalizer: can you pls take a look at https://review.openstack.org/107331 again? | 23:08 |
| *** Sukhdev has quit IRC | 23:08 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 23:08 |
| *** armax has quit IRC | 23:09 | |
| jeblair | clarkb: aprvd | 23:09 |
| *** kumartin has quit IRC | 23:09 | |
| *** alexpilotti has quit IRC | 23:11 | |
| *** armax has joined #openstack-infra | 23:11 | |
| *** doug-fish has left #openstack-infra | 23:11 | |
| anteaya | mtreinish: they all acknowledge you though, didn't for the rest of us :D | 23:11 |
| *** ryanpetrello has quit IRC | 23:12 | |
| jeblair | ImportError: No module named Crypto.PublicKey | 23:12 |
| jeblair | clarkb: ^ seen that? | 23:12 |
| jeblair | clarkb: i think pycrypto may have failed to install but pip didn't notice | 23:13 |
| *** Longgeek has joined #openstack-infra | 23:13 | |
| fungi | jeblair: oh, yes, it does that | 23:13 |
| mordred | because, you know, why fail on failure? | 23:14 |
| *** ChuckC has joined #openstack-infra | 23:14 | |
| fungi | pycrypto setup.py encounters some fatal condition and then returns 0 | 23:14 |
| openstackgerrit | Merged openstack-infra/system-config: Install build deps for nodepool installation https://review.openstack.org/136418 | 23:14 |
| clarkb | jeblair: ya, I think its easier if you just rebuild nodepool from scratch either b booting new node or installing all the things again after deleting them | 23:14 |
| clarkb | jeblair: that said I think I may see the bug | 23:14 |
| clarkb | jeblair: maybe | 23:14 |
| *** ryanpetrello has joined #openstack-infra | 23:14 | |
| fungi | and pip goes, oh! setup.py exited successfully, this must be installed now. i'll make note of that | 23:14 |
| jeblair | pip install pycripto; pip install ecdsa ... | 23:14 |
| jeblair | pycrypto even | 23:14 |
| mriedem1 | fungi: clarkb: just to confirm my understanding, this means the rax-dfw provider provides at most 184 VMs, right? http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n232 | 23:15 |
| *** teran has joined #openstack-infra | 23:15 | |
| mriedem1 | however many compute nodes it takes to service that is up to the provider i'm assuming | 23:15 |
| fungi | mriedem1: yep | 23:15 |
| *** jamielennox|away is now known as jamielennox | 23:15 | |
| mriedem1 | what does max-servers: 0 mean? | 23:15 |
| mriedem1 | unlimited? | 23:15 |
| fungi | mriedem1: we don't see compute nodes--that's all the provider's internal business | 23:15 |
| fungi | mriedem1: 0 means exactly 0 | 23:16 |
| *** hdd has joined #openstack-infra | 23:16 | |
| fungi | mriedem1: as in don't build any here | 23:16 |
| mriedem1 | ok, off limits | 23:16 |
| fungi | mriedem1: a convenient means of disabling a provider | 23:16 |
| mriedem1 | yup, cool | 23:16 |
| *** MaxV has joined #openstack-infra | 23:16 | |
| *** mestery has quit IRC | 23:16 | |
| *** marun has quit IRC | 23:16 | |
| clarkb | jeblair: actually maybe not /me is reading code | 23:17 |
| clarkb | becomes much more familiar with gear | 23:17 |
| mriedem1 | is there anything that describes the flavor requirements? i thought we run 8 GB RAM and 8 VCPU flavors, but wasn't sure about disk, or if that's universal | 23:17 |
| clarkb | mriedem1: its not universal, we get different disk from rax and hpcloud | 23:17 |
| mriedem1 | clarkb: but ram/vcpu is the same? | 23:17 |
| mriedem1 | or depends on the job? | 23:18 |
| *** mestery has joined #openstack-infra | 23:18 | |
| clarkb | mriedem1: ram/vcpu/disk are the same per provider | 23:18 |
| *** Longgeek has quit IRC | 23:18 | |
| *** rkukura has joined #openstack-infra | 23:18 | |
| mriedem1 | ah i see min-ram in here | 23:18 |
| mriedem1 | but ok | 23:18 |
| *** teran_ has quit IRC | 23:18 | |
| clarkb | mriedem1: and when min-ram is > 8GB we artificially limit to 8GB with kernel boot params | 23:18 |
| *** ryanpetrello has quit IRC | 23:20 | |
| dougwig | fungi: can we get a force merge on this and the two sequentially after it? i had botched the gitreview in the three repos, and don't want anyone to clone them that way: https://review.openstack.org/#/c/140156/ | 23:20 |
| fungi | (like in hpcloud where we need to boot 32gb flavors to get the same vcpu count as 8gb has in rax) | 23:20 |
| mriedem1 | fungi: i imagine managing those mappings is a real treat | 23:21 |
| mriedem1 | or relatively trivial? | 23:21 |
| dougwig | (140156, 140157, 140158) | 23:21 |
| fungi | mriedem1: you're welcome to part of that treat if you like! | 23:21 |
| mriedem1 | so it's not tasty | 23:22 |
| mtreinish | fungi: you can't go recruiting mriedem1 for infra yet, he's got to get his tempest review count up first... | 23:23 |
| mriedem1 | mtreinish: and that sql testr pet project of yours | 23:24 |
| *** marun has joined #openstack-infra | 23:24 | |
| mtreinish | mriedem1: all you need are 2-5 real reviews for that and I'm comfortable giving you +2 :) | 23:25 |
| mtreinish | or 1 real patch... | 23:25 |
| *** salv-orlando has quit IRC | 23:26 | |
| *** annegent_ has joined #openstack-infra | 23:26 | |
| fungi | dougwig: i'm convinced the gating bypass on those relatively trivial patches is warranted so people don't end up with potentially disasterous git remotes while you're iterating on the packaging fixes | 23:27 |
| mriedem1 | that's more than i'm available for right now :) | 23:27 |
| dougwig | fungi: thanks. | 23:27 |
| clarkb | jeblair: you know I wonder if its possibly related to calling conn.recv with a 1 byte buffer over and over and over again | 23:27 |
| *** salv-orlando has joined #openstack-infra | 23:27 | |
| jeblair | clarkb: talking about gear? | 23:27 |
| clarkb | jeblair: ya | 23:27 |
| nibalizer | zaro: sure | 23:27 |
| nibalizer | also idunno if you've been to ##openstack-robots yet | 23:28 |
| nibalizer | but i got that hubot up for you :P | 23:28 |
| clarkb | jeblair: since its possible that is a yield point and we could then be yielding a lot | 23:28 |
| nibalizer | if you give me an ssh key ill give you total access | 23:28 |
| jeblair | nibalizer: hubot? | 23:28 |
| zaro | nibalizer: i haven't but that's great. need to check it out | 23:28 |
| *** Sukhdev has joined #openstack-infra | 23:29 | |
| zaro | jeblair: nibalizer and i talked about using it for the purpose of replacing all of the other bots. | 23:30 |
| *** MaxV has quit IRC | 23:30 | |
| zaro | nibalizer: my weechat died and i lost that channel, cann you please send me another invite? | 23:31 |
| jeblair | zaro: excellent idea, which is why we were looking into errbot | 23:31 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 23:31 |
| mtreinish | dougwig: ^^^ closing " are important... | 23:32 |
| dougwig | mtreinish: ty, i hate when i do that. | 23:33 |
| fungi | looks like hubot is using mit-expat license | 23:33 |
| sc68cal | jeblair: Hey, I'm sitting next to Anita and working on IPv6 testing in DevStack, at the gate - could I bother you for a review? It's a devstack-gate repo review - https://review.openstack.org/#/c/140128/ | 23:33 |
| *** amitgandhinz has quit IRC | 23:33 | |
| nibalizer | zaro: its not password protected or anything | 23:33 |
| zaro | jeblair: ohh cool, python version of it. | 23:33 |
| nibalizer | you can just hop it | 23:33 |
| *** Sukhdev has quit IRC | 23:34 | |
| fungi | though hubot is coffeescript | 23:34 |
| morganfainberg | fungi, nodejs | 23:34 |
| morganfainberg | right? | 23:34 |
| morganfainberg | or is it coffee? | 23:34 |
| fungi | morganfainberg: yep | 23:34 |
| *** Sukhdev has joined #openstack-infra | 23:34 | |
| morganfainberg | or are those... inter...changable | 23:34 |
| *** craigbr has joined #openstack-infra | 23:34 | |
| * morganfainberg knows little about nodejs. | 23:34 | |
| jeblair | yeah, tbh i'm much more interested in using something like errbot because of the python. | 23:35 |
| fungi | morganfainberg: right, so the bot engine seems to be in javascript/node and then its extensibility is via coffeescript | 23:36 |
| jeblair | after all, our existing bots are in python | 23:36 |
| fungi | agreed on the "something in python" preference | 23:36 |
| *** Ryan_Lane has quit IRC | 23:36 | |
| fungi | javascript seems like an odd language choice for an irc bot | 23:36 |
| morganfainberg | hehe | 23:36 |
| *** Ryan_Lane has joined #openstack-infra | 23:37 | |
| *** Masahiro has joined #openstack-infra | 23:37 | |
| fungi | then again, probably no more maddening than eggdrop being written in tcl (though i guess it must have descended from an expect script so maybe not surprising) | 23:37 |
| grue_pm | fungi: as long as it doesnt have a SLAP database backend ;-p | 23:38 |
| *** shashankhegde has quit IRC | 23:39 | |
| jeblair | clarkb: i think the fundamental problem is that the thread is starving, and i think that is happening outside of the gearman calls too | 23:40 |
| jeblair | clarkb: after all, we've seen up to 10 minutes between cycles, and only 90 seconds of that, at most, could be gearman recv's | 23:40 |
| jeblair | clarkb: so it could be exacerbating the problem, but i don't think it's the only source of delay | 23:41 |
| clarkb | jeblair: yes, I think that is correct. But Ithink that it is possible that recv(1) is contributing to that | 23:41 |
| clarkb | jeblair: no it could be >90 seconds beacuse you have to read all the bytes off of the receive queue before you get to the next admin request | 23:41 |
| *** Masahiro has quit IRC | 23:41 | |
| *** jgrimm is now known as zz_jgrimm | 23:41 | |
| clarkb | the exception should be thrown within ~90 seconds though | 23:42 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/reviewstats: Update neutron core https://review.openstack.org/140159 | 23:42 |
| jeblair | clarkb: it's never going to spend more than 90 seconds during each cycle recving though | 23:43 |
| clarkb | jeblair: wouldn't it have to in order to remove the now useless bytes from the queue? | 23:44 |
| jeblair | clarkb: oh, actually, recving is a different thread | 23:44 |
| jeblair | lemme look at something | 23:45 |
| *** salv-orlando has quit IRC | 23:45 | |
| jeblair | clarkb: yeah, so it's the client poll loop that deals with recving, while it's the main nodepool thread that's waiting on a complete event to be set by that thread; and it's the main nodepool thread that gets the 90 second timeout | 23:46 |
| *** zz_avozza is now known as avozza | 23:47 | |
| jeblair | clarkb: so we don't really know how often the gearman io thread is being serviced | 23:47 |
| clarkb | yup, so what I think may be happening is the client poll loop thread isn't getting enough cpu time to completely read in the entire status packet | 23:47 |
| jeblair | it could be way behind since it's not subject to the timeout | 23:47 |
| clarkb | then the nodepool loop timesout | 23:47 |
| jeblair | ya | 23:47 |
| clarkb | and since recv() is a likely yield point calling it a bunch is giving up that threads control of the cpu | 23:48 |
| jeblair | clarkb: having said that, my test server is using 116% cpu and appears to be idle. | 23:49 |
| jeblair | i think there's something to this pathological scheduling theory | 23:49 |
| *** shashankhegde has joined #openstack-infra | 23:50 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 23:50 |
| jeblair | clarkb: it's only created 100 out of the 1000 servers it scheduled so far | 23:50 |
| jeblair | clarkb: note that the fake drivers spawn a thread that sleeps, so this is actually much worse from a # of python threads perspective (approx 2x while building) | 23:51 |
| clarkb | jeblair: but the fake status getting is working? | 23:52 |
| *** dmsimard is now known as dmsimard_away | 23:52 | |
| clarkb | jeblair: maybe make that status result large? | 23:52 |
| jeblair | clarkb: yeah, i have not made the status result large yet; i feel like there's already some weird behavior going on, and i want to directly compare it to precise | 23:52 |
| jeblair | clarkb: do you think we should roll back to precise or continue on trusty? | 23:53 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 23:53 |
| clarkb | jeblair: the way the test load has been today trusty has been ok | 23:54 |
| jeblair | yeah, though we could be a bit unhappy if it increases significantly | 23:54 |
| clarkb | jeblair: I guess it depends on whether or not we are worried about the load picking up and being caught in a bad spot. | 23:54 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 23:54 |
| clarkb | I am happy to rollback if we want to avoid future potential trouble | 23:55 |
| clarkb | should I start that process? | 23:55 |
| *** dimtruck is now known as zz_dimtruck | 23:56 | |
| jeblair | omg, python recv does a select each time | 23:57 |
| *** gyee has joined #openstack-infra | 23:58 | |
| clarkb | lol | 23:59 |
| jeblair | recv -> {release gil; select, recv; acquire gil} (broadly speaking) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!